Mail content feature matching method and system based on flow analysis system

Abstract

The invention relates to a mail content feature matching method and system based on a flow analysis system, which realize non-complete cache stream processing of a subject and a main body of a mail through TCP stream recombination, line-by-line MIME analysis and dynamic code identification, effectively solve the problem of delay of long mail body extraction in the prior art, and improve the extraction efficiency. And the accuracy of continuous matching of the inter-row keywords is ensured. Meanwhile, limited byte cache and Base64 grouping characteristics are utilized to reduce memory occupation, rule grouping transcoding, hit mark priority adjustment and alarm frequency suppression are combined, matching timeliness and system load efficiency are remarkably improved, a prevention and control strategy is taken into effect in time on the premise of ensuring characteristic correctness, missed transmission of malicious mails is avoided, and the service life of the system is prolonged. And an efficient and real-time network security guarantee is provided for a traffic analysis system.

Description

A Method and System for Matching Email Content Features Based on Traffic Analysis Technical Field

[0001] This invention belongs to the field of communication technology, and in particular relates to a method and system for matching email content features based on a traffic analysis system. Background Technology

[0002] In existing traffic analysis systems, email feature matching technology has evolved from simple rule-based keyword filtering to support MIME multi-part parsing and transmission encoding / decoding. Early systems relied on SMTP proxies or terminal emulators to capture the entire email body, generate an EML file, and then verify and decode it. Regular expressions or hash matching rules were used to detect threats such as spam and phishing links. Subsequently, with the widespread adoption of open-source intrusion detection systems (such as Suricata and Snort), this technology introduced Base64 / Quoted-Printable streaming decoding and multi-threaded content scanning, supporting real-time alerts and policy blocking, further improving detection accuracy. In recent years, commercial traffic analysis devices (such as Palo Alto Networks and Fortinet) have integrated machine learning models to automatically identify email header fields (such as the RFC 2047 encoding of Subject) and MIME headers (Content-Type, charset, Content-Transfer-Encoding), enabling Chinese character encoding (UTF-8, GB2312) conversion matching and attachment scanning, adapting to multilingual spam scenarios.

[0003] However, the current development still has significant shortcomings. The matching process relies on the extraction and integrity verification of the entire email body, leading to excessive cache usage and significant processing latency under long emails or high-concurrency traffic. It cannot achieve line-by-line real-time matching, and cross-line keywords (such as obfuscated features of line boundary segmentation) are easily missed. At the same time, rule matching requires waiting for the entire EML file to be decoded, resulting in delayed alarm triggering and difficulty in timely policy implementation, creating the risk of malicious emails being missed. In addition, full decoding and matching increase the computational load. In high-throughput scenarios with bypass mirror traffic, system resources are wasted significantly, making it difficult to dynamically adjust rule priorities or suppress duplicate alarms. Overall efficiency is low, failing to meet the timeliness and low-load requirements of enterprise-level network security prevention and control. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for matching email content features based on a traffic analysis system, so as to solve the problems of insufficient timeliness and waste of computing resources caused by the reliance on complete email body extraction for email feature matching in existing traffic analysis systems.

[0005] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a method for matching email content features based on a traffic analysis system, the method comprising:

[0006] The received mirrored email protocol traffic is reassembled using TCP streams to obtain an ordered email protocol data stream.

[0007] The system parses the email header and MIME header information line by line, identifies character encoding and transmission encoding, and performs feature rule matching on the decoded email subject.

[0008] The email body content is read line by line, and streaming decoding is performed according to the identified transmission encoding. At the same time, a cache of a preset length of bytes at the end of the previous line is maintained and combined with the current line to form a continuous data buffer to achieve continuous feature matching of cross-line keywords.

[0009] As a further improvement to one embodiment of the present invention, the method further includes, in that the line-by-line parsing of the email header and MIME header information includes,

[0010] Read the email header fields line by line, using \r\n as the delimiter, and extract the From, To, Date, and Subject fields;

[0011] For the Subject field, if the RFC 2047 encoding format is used, its encoding identifier is parsed to determine the character encoding and transmission encoding type, and feature rule matching is performed on the decoded plaintext.

[0012] As a further improvement to one embodiment of the present invention, the method further includes, the identification character encoding and transmission encoding including,

[0013] Identify whether the Content-Type is of type multipart;

[0014] If it is a multipart type, extract the boundary string and use the boundary string to separate each MIME part;

[0015] For each MIME part or a single non-multipart part, parse its Content-Type, charset, and Content-Transfer-Encoding fields to determine the character encoding and transfer encoding of the current part.

[0016] As a further improvement to one embodiment of the present invention, the method further includes, in which the streaming decoding based on the identified transport code includes,

[0017] Read the email body line by line and decode it in real time;

[0018] When the transmission encoding is Base64, the fixed block size of Base64 encoding, where every 4 characters correspond to 3 bytes, is used for line-by-line streaming decoding.

[0019] As a further improvement to one embodiment of the present invention, the method further includes maintaining a cache of a preset length of bytes at the end of the previous line, which is combined with the current line to form a continuous data buffer to achieve continuous feature matching of cross-line keywords.

[0020] Maintain a 64-byte buffer at the end of the previous line, and combine it with the current line to form a continuous data buffer for feature matching;

[0021] For pure English feature rules, match directly on the decoded plaintext or the original data;

[0022] For feature rules containing Chinese or a mixture of Chinese and English, user input rules are stored in UTF-8 encoding by default; during matching, the rule is converted to the character encoding of the current MIME part before matching, and the character encoding includes at least GB2312;

[0023] The feature rules are grouped according to Chinese character encoding variants, and only the rule group corresponding to the current recognition encoding is loaded for matching.

[0024] As a further improvement to one embodiment of the present invention, the method further includes, when a feature rule is matched, marking the corresponding traffic flow, and dynamically adjusting the priority of subsequent rules based on historical match records, specifically including,

[0025] Mark the traffic flows that hit the rules, and record the rule identifier and time.

[0026] Based on historical hit records, subsequent matching of the same traffic flow adopts a priority sorting mechanism, giving priority to matching rules with higher historical hit frequencies.

[0027] Frequency suppression is implemented for alarms with the same or similar rules for the same traffic flow to limit the number of alarm reports within a preset time period, and to suspend some matching operations for that traffic flow during the suppression period.

[0028] As a further improvement of one embodiment of the present invention, the method further includes releasing the cache and tag resources related to the traffic stream after the email protocol interaction ends.

[0029] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a method for matching email content features based on a traffic analysis system, wherein the system includes a traffic reconstruction module, an encoding recognition module, and a cross-line matching module;

[0030] The traffic reassembly module is used to reassemble the received mirrored mail protocol traffic into an ordered mail protocol data stream using TCP stream reassembly.

[0031] The encoding recognition module is used to parse the email header and MIME header information line by line, identify character encoding and transmission encoding, and perform feature rule matching on the decoded email subject.

[0032] The cross-line matching module is used to read the email body content line by line, perform streaming decoding according to the identified transmission encoding, and maintain a cache of a preset length of bytes at the end of the previous line. This cache is combined with the current line to form a continuous data buffer, so as to achieve continuous feature matching of cross-line keywords.

[0033] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the email content feature matching method based on the traffic analysis system as described above.

[0034] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps in the email content feature matching method based on a traffic analysis system as described above.

[0035] Compared with existing technologies, this invention provides an email content feature matching method and system based on a traffic analysis system. Through TCP stream reassembly, line-by-line MIME parsing, and dynamic encoding recognition, it achieves unrestricted, buffer-free streaming processing of email subjects and bodies, effectively solving the latency problem in long email body extraction in existing technologies and ensuring the accuracy of continuous cross-line keyword matching. Simultaneously, it reduces memory usage by utilizing limited-byte caching and Base64 grouping characteristics. Combined with rule-based grouping transcoding, hit flag priority adjustment, and alarm frequency suppression, it significantly improves matching timeliness and system load efficiency. This allows for timely implementation of prevention and control strategies while ensuring feature accuracy, preventing malicious emails from being missed and providing efficient, real-time network security protection for traffic analysis systems. Attached Figure Description

[0036] Figure 1 is an overall flowchart of the email content feature matching method based on the traffic analysis system described in this invention.

[0037] Figure 2 is a schematic diagram of the architecture of the email content feature matching system based on the traffic analysis system described in this invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0039] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0040] In Embodiment 1 of the present invention, the present invention provides an email content feature matching method based on a traffic analysis system, as shown in Figure 1. The method includes,

[0041] S1: Perform TCP stream reassembly on the received mirrored email protocol traffic to obtain an ordered email protocol data stream;

[0042] S2: Parse the email header and MIME header information line by line, identify character encoding and transmission encoding, and perform feature rule matching on the decoded email subject;

[0043] S3: Read the email body content line by line, perform streaming decoding based on the identified transmission encoding, and maintain a cache of the preset length bytes at the end of the previous line. Combine this cache with the current line to form a continuous data buffer, so as to achieve continuous feature matching of cross-line keywords.

[0044] In one specific embodiment of the present invention, the email header and MIME header information are parsed line by line, specifically as follows:

[0045] Read the email header fields line by line, using \r\n as the delimiter, and extract the From, To, Date, and Subject fields;

[0046] For the Subject field, if the RFC 2047 encoding format is used, its encoding identifier is parsed to determine the character encoding and transmission encoding type, and feature rule matching is performed on the decoded plaintext.

[0047] It should be noted that the process of parsing the email header and MIME header information line by line aims to obtain key metadata and initiate feature matching as early as possible through an efficient and orderly line-by-line reading method, without relying on the complete email cache, thereby achieving a balance between real-time performance and resource efficiency.

[0048] Furthermore, extracting the From, To, Date, and Subject fields allows for structured parsing of the email header immediately upon arrival of the email protocol data stream. The email header uses the standard RFC 5322 format, with \r\n as the field separator. By reading line by line, the core fields can be extracted from the first few lines of the email header without waiting for the entire email body to be transmitted.

[0049] Specifically, the From and To fields are extracted for subsequent sender / recipient reputation analysis or white / blacklist matching (an optional extended function of this invention); the Date field is extracted for timestamp verification and alarm time recording; and the Subject field is the core early matching entry point of this invention, because the email subject often carries the most important semantic features (such as the deceptive titles commonly used in phishing emails). Through this line-by-line reading mechanism, this step can send the Subject field into the feature matching process as early as possible, thereby significantly shortening the time window from the start of email transmission to the first feature hit.

[0050] Furthermore, for the Subject field, if the RFC 2047 encoding format is used, its encoding identifier is parsed to determine the character encoding and transmission encoding type. This addresses the common non-ASCII character encoding issues in email subjects and enables real-time feature detection of the subject content. According to the RFC 2047 specification, non-ASCII characters in the Subject field typically use the encoding format "=?charset?X?encoded-text?=", where:

[0051] charset represents the character set (such as UTF-8, GB2312, GBK, etc.);

[0052] X indicates the transmission encoding type (B represents Base64, Q represents Quoted-Printable).

[0053] This step first identifies the encoded marker, extracts the charset and encoding type (B / Q), and then performs the corresponding decoding operation (Base64 decoding or Quoted-Printable decoding) on ​​the encoded-text part to obtain plaintext that can be directly matched with strings.

[0054] After decoding, the plaintext is immediately sent to the feature rule matching engine for matching against a pre-defined threat feature database (such as sensitive keywords, phishing phrases, malicious domains, etc.). The early execution of this step is crucial:

[0055] Email subjects are usually short and located at the very beginning of the email header; prioritizing their matching can achieve "quick hit and quick alert".

[0056] Many malicious emails are characterized primarily or solely in their subject lines (such as "urgent invoice", "account anomaly", "claim bonus"). Therefore, matching the subject as early as possible can effectively improve the timeliness of threat detection in the overall system.

[0057] By completing decoding and matching during the header parsing stage, the latency and memory overhead caused by loading the entire email body and then performing secondary processing are avoided.

[0058] In one specific embodiment of the present invention, character encoding and transmission encoding are identified, specifically as follows:

[0059] Identify whether the Content-Type is of type multipart;

[0060] If it is a multipart type, extract the boundary string and use the boundary string to separate each MIME part;

[0061] For each MIME part or a single non-multipart part, parse its Content-Type, charset, and Content-Transfer-Encoding fields to determine the character encoding and transfer encoding of the current part.

[0062] It should be noted that the process of identifying character encoding and transmission encoding aims to accurately and promptly determine the character encoding and transmission encoding methods of each part of the email body after parsing the main fields of the email header. This provides a reliable decoding basis for subsequent streaming decoding and feature matching, and avoids matching errors or missed detections caused by encoding misjudgment.

[0063] Furthermore, identifying whether the Content-Type is multipart involves determining whether the current email uses a MIME multipart structure (common types such as multipart / mixed, multipart / alternative, and multipart / related). In actual email traffic, most emails containing multiple parts such as the body, HTML version, and attachments use the multipart type. This determination helps decide whether subsequent boundary separators need to be processed, thus avoiding unnecessary complex parsing operations on single-text emails and improving parsing efficiency.

[0064] Furthermore, after confirming that the email is of multipart type, the value of the boundary parameter in the Content-Type field is extracted (e.g., boundary="----=_Part_123456789.1234567890"), and this value is used as a delimiter to locate the start and end positions of each MIME part in the subsequent data stream. Extracting and applying the boundary string is a prerequisite for implementing part-by-part parsing. Its core purpose is to: divide the continuous email body data stream into multiple logically independent parts (e.g., text body, HTML version, attachments); ensure that the header information and body content of each MIME part can be accurately divided, avoiding parsing chaos or data confusion caused by boundary recognition errors; and provide clear processing boundaries for subsequent independent encoding, recognition, and decoding of each part.

[0065] Furthermore, for each MIME part or a single non-multipart part, its Content-Type, charset, and Content-Transfer-Encoding fields are parsed. This is to precisely determine the content type, charset, and content-transfer-encoding fields for each MIME part (or the case where the entire email body is a single non-multipart part) by reading its sub-header fields one by one.

[0066] Character encoding (charset): such as UTF-8, GB2312, GBK, ISO-8859-1, etc., used for subsequent encoding matching of Chinese or multilingual feature rules;

[0067] Content-Transfer-Encoding: such as base64, quoted-printable, 8-bit, 7-bit, etc., is used to determine whether and how streaming decoding is needed.

[0068] By parsing each part individually, the system ensures that even if different parts of the same email use different encoding methods (such as UTF-8+8bit for the body and base64 for the attachment), the system can still make the correct decoding decision for each part independently, thereby guaranteeing the accuracy of feature matching.

[0069] In one specific embodiment of the present invention, streaming decoding is performed based on the identified transmission code, specifically,

[0070] Read the email body line by line and decode it in real time;

[0071] When the transmission encoding is Base64, the fixed block size of Base64 encoding, where every 4 characters correspond to 3 bytes, is used for line-by-line streaming decoding.

[0072] It should be noted that the process of streaming decoding based on the identified transmission encoding aims to perform targeted decoding on the transmission encoding characteristics of the current line as the email body data stream arrives line by line. This converts the encoded transmission data into plaintext or near-plaintext in real time, which can be directly matched for features, without waiting for the complete email body to be received or caching the entire email content in memory. This significantly reduces system resource consumption and improves the timeliness of threat detection.

[0073] Furthermore, reading the email body line by line and decoding it in real time transforms the email body processing method from the traditional "full caching → overall decoding → post-matching" to a streaming mode of "processing as received". By reading the email body line by line (usually separated by \r\n), the system can initiate the decoding operation immediately as the data stream arrives linearly, without needing to pre-cache the complete email body (especially for long emails or emails with large attachments). The "real-time decoding" design directly serves the following goals: achieving the earliest possible feature matching trigger point; avoiding processing latency caused by waiting for the complete email body; and significantly reducing peak memory usage and overall system resource pressure in high-concurrency traffic scenarios.

[0074] Furthermore, when the transmission encoding is Base64, the fixed grouping characteristic of Base64 encoding, where every 4 characters correspond to 3 bytes, is used to perform line-by-line streaming decoding. This is specifically optimized for the most common binary / attachment encoding method (Base64) in email transmission. Its core purpose is to utilize the mathematical characteristics of Base64 encoding to achieve efficient, lossless, and continuous streaming decoding across lines.

[0075] The fixed grouping rule of Base64 encoding (encoding 3 bytes of raw data into 4 printable characters) determines its inherent "segmentability" and "partial decoding capability". This step utilizes this characteristic, specifically by combining the Base64 characters of the current line with the remaining characters at the end of the previous line (obtained through a 64-byte buffer) to form a complete 4-character group; decoding the complete group to obtain the corresponding 3 bytes of raw data; and temporarily storing any remaining characters less than 4 to be added to the next line, thus ensuring that the decoding process is continuous, seamless, and without losing any information.

[0076] This step enables seamless recovery of cross-line Base64 encoding, and can correctly decode and match even if the keyword or feature string is split by line boundaries; there is no need to store the entire Base64 encoded block in memory (especially for large attachments that may be several MB), only a small amount of cross-line residual cache (usually tens of bytes) needs to be maintained; the decoded plaintext stream can be output while processing line by line, which can be used immediately by the feature matching engine, greatly shortening the time window from data arrival to threat alarm.

[0077] In one specific embodiment of the present invention, a cache of a preset length of bytes at the end of the previous line is maintained and combined with the current line to form a continuous data buffer, so as to achieve continuous feature matching of cross-line keywords. Specifically,

[0078] Maintain a 64-byte buffer at the end of the previous line, and combine it with the current line to form a continuous data buffer for feature matching;

[0079] For pure English feature rules, match directly on the decoded plaintext or the original data;

[0080] For feature rules containing Chinese or a mixture of Chinese and English, user input rules are stored in UTF-8 encoding by default; during matching, the rule is converted to the character encoding of the current MIME part before matching, and the character encoding includes at least GB2312;

[0081] The feature rules are grouped according to Chinese character encoding variants, and only the rule group corresponding to the current recognition encoding is loaded for matching.

[0082] It should be noted that maintaining a cache of a preset length of bytes at the end of the previous line, combined with the current line to form a continuous data buffer, enables continuous feature matching of cross-line keywords. The aim is to eliminate the fragmentation effect of line boundaries (\r\n) on the integrity of feature strings during line-by-line reading and decoding by using a historical data cache of limited length. This ensures that even if keywords, sensitive phrases, or malicious features are artificially split between adjacent lines, they can still be accurately and completely identified and matched, while maintaining extremely low memory usage.

[0083] Furthermore, after processing the current line, the system saves the last 64 bytes of data in the cache. When the next line arrives, the 64 bytes in the cache are concatenated with the entire content of the current line to form a continuous buffer with a maximum length of "64 bytes + the length of the current line".

[0084] The choice of a 64-byte length is based on the following technical considerations: it is sufficient to cover the length of most feature strings (common threat keywords, URLs, phishing phrases, etc., are usually much shorter than this length); for Base64 encoded scenarios, it can guarantee coverage of at least one complete block (4 characters → 3 bytes) of residual data; and for most email line lengths, this cache occupies very little space (tens to hundreds of bytes), and will not put significant pressure on system memory. Through this continuous buffer, the system can achieve "seamless cross-line matching" of features at line boundaries, effectively countering the line splitting obfuscation techniques commonly used by spammers.

[0085] Furthermore, for rules based solely on English characters, matching is performed directly on the decoded plaintext or the original data. This approach provides the most efficient matching path for the most common English threat characteristics (such as English phishing phrases, malicious domains, C2 addresses, etc.). Since pure English characters are unaffected by character encoding and typically do not require additional transcoding, matching is performed directly on the plaintext that has already been decoded (or on some 8-bit / 7-bit original data), minimizing computational overhead and achieving the highest real-time performance.

[0086] Furthermore, user-input feature rules are uniformly stored in UTF-8 encoding (currently the mainstream and most compatible encoding). During actual matching, based on the charset (such as GB2312, GBK, UTF-8, etc.) of the current MIME part identified in the aforementioned steps, the rules are converted to the corresponding encoding in real time before being matched with the decoded email content. The reason for supporting at least GB2312 conversion is that GB2312 and its extended GBK are still used by a large number of Chinese emails (especially early or industry-specific emails), and the completeness of conversion support directly determines the system's compatibility and detection coverage with older email systems.

[0087] Furthermore, during initialization, the system pre-groups all Chinese / mixed rules according to their encoding type (such as UTF-8, GB2312, GBK, etc.). When actually processing a certain MIME part, only the rule group that matches the current charset is loaded into the matching engine, avoiding memory waste and decreased matching efficiency caused by loading all rules.

[0088] In one specific embodiment of the present invention

[0089] When a feature rule is matched, the corresponding traffic flow is marked, and the priority of subsequent rules is dynamically adjusted based on historical match records. Specifically, this includes...

[0090] Mark the traffic flows that hit the rules, and record the rule identifier and time.

[0091] Based on historical hit records, subsequent matching of the same traffic flow adopts a priority sorting mechanism, giving priority to matching rules with higher historical hit frequencies.

[0092] Frequency suppression is implemented for alarms with the same or similar rules for the same traffic flow to limit the number of alarm reports within a preset time period, and to suspend some matching operations for that traffic flow during the suppression period.

[0093] It should be noted that when a feature rule is hit, the corresponding traffic flow is marked, and the priority of subsequent rules is dynamically adjusted based on historical hit records. The aim is to enable the system to quickly identify the most likely related threat features for the same traffic flow through the real-time recording and intelligent feedback mechanism of hit events, while effectively controlling the consumption of system resources and maintenance personnel by repeated alarms. This significantly improves the overall processing efficiency and real-time response capability while ensuring detection accuracy.

[0094] Furthermore, when any feature rule is matched, the system immediately records the five-tuple identifier (source IP, source port, destination IP, destination port, protocol) or flow ID of the traffic flow along with the matched rule ID, match timestamp, and other information into the flow state table in memory or a lightweight cache structure. The direct uses of this marking and recording operation include: enabling rapid tracing of match events and alarm correlation display; accumulating statistical data on historical match behavior for the same flow; and serving as the starting point for subsequent priority adjustments and alarm suppression.

[0095] Furthermore, when maintaining the hit history of each traffic flow, the system counts the number of hits or frequency of each rule (using a simple counter or a weighted decay mechanism). When subsequent data packets arrive in that flow, the matching engine first sorts the rules from highest to lowest according to their historical hit frequency, and then performs the matching one by one. The core purpose of this mechanism is that the same traffic flow often has continuous threat patterns (such as a wave of spam or phishing attacks), and prioritizing the matching of high-frequency hit rules can significantly reduce the number of rule traversals; it can significantly reduce the average matching latency in high-concurrency traffic environments; and it can improve the system's response speed to recurring threats.

[0096] Furthermore, when the same traffic flow continuously hits the same or similar rules within a preset time window (e.g., configurable periods such as 5 seconds or 30 seconds), (similarity can be determined by rule ID or rule category), the system only reports one alarm and suppresses subsequent matching of the same / similar rules for that flow within this time period (the system can choose to skip the matching directly or simply record without alarming). The specific uses of this suppression include: effectively controlling the number of alarms to prevent maintenance personnel from being overwhelmed by a massive number of duplicate alarms; reducing unnecessary matching calculations within the suppression period, further reducing CPU load; and achieving rational handling of persistent threats without affecting the timeliness of the first hit.

[0097] In one specific embodiment of the present invention, after the email protocol interaction ends, the cache and tag resources related to the traffic stream are released.

[0098] It should be noted that after the email protocol interaction ends, all temporary resources associated with the traffic stream of the completed email transmission should be released in a timely and thorough manner to avoid memory leaks and continuous accumulation of resource consumption, thereby ensuring that the system can maintain a stable level of memory usage and processing performance under long-term, high-concurrency traffic analysis scenarios.

[0099] Furthermore, the system monitors the status flags of the underlying mail protocols (SMTP, POP3, or IMAP) to determine whether the current traffic stream has completed a full interaction. The criteria for this determination include, but are not limited to: receiving a "." terminator and subsequent QUIT command in the SMTP protocol; completing the DELE / QUIT sequence after the RETR command in the POP3 protocol; completing FETCH and closing the session (LOGOUT or connection termination) in the IMAP protocol; or completing the TCP four-way handshake without subsequent data packets. This mechanism ensures that resource release is triggered only when the mail transmission has truly ended, avoiding the accidental release of mail streams still in transit.

[0100] Furthermore, once the interaction is confirmed to be complete, the system immediately performs the following cleanup operations: releases the last 64 bytes of the previous line's cross-line buffer; clears the flow state table entries for this traffic stream (including hit flags, hit rule IDs, hit timestamps, historical hit counters, etc.); releases any temporary rule priority sorting structures or alarm suppression timers associated with this flow; and, if LRU or other cache management mechanisms are used, synchronously updates the cache eviction queue. The direct purpose of these release operations is to: prevent continuous memory growth, especially in high-load scenarios handling tens or even hundreds of thousands of concurrent email streams; avoid matching logic errors caused by unreleased flow states (such as incorrectly confusing new email streams with historical hit records of old streams); and ensure the cyclical utilization of system resources, enabling limited memory to continuously support the processing of newly arriving email traffic.

[0101] In a second embodiment of the present invention, the present invention provides a method for matching email content features based on a traffic analysis system, as shown in Figure 2. The system includes a traffic reconstruction module 1, an encoding recognition module 2, and a cross-line matching module 3.

[0102] The traffic reconstruction module 1 is used to reconstruct the received mirrored mail protocol traffic using TCP streams to obtain an ordered mail protocol data stream.

[0103] The encoding recognition module 2 is used to parse the email header and MIME header information line by line, recognize character encoding and transmission encoding, and perform feature rule matching on the decoded email subject.

[0104] The cross-line matching module 3 is used to read the email body content line by line, perform streaming decoding according to the identified transmission encoding, and maintain a cache of a preset length of bytes at the end of the previous line. This cache is combined with the current line to form a continuous data buffer, so as to achieve continuous feature matching of cross-line keywords.

[0105] In a third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the dual-sensing-based relay multi-hop adaptive routing method as described above.

[0106] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the dual-sensing-based relay multi-hop adaptive routing method as described above.

[0107] In summary, the email content feature matching method and system provided by this invention, based on a traffic analysis system, achieves unrestricted, buffer-free streaming processing of email subjects and bodies through TCP stream reassembly, line-by-line MIME parsing, and dynamic encoding recognition. This effectively solves the latency problem in long email body extraction in existing technologies, ensuring the accuracy of continuous cross-line keyword matching. Simultaneously, by utilizing limited-byte caching and Base64 grouping characteristics to reduce memory consumption, and combining rule-based grouping transcoding, hit flag priority adjustment, and alarm frequency suppression, it significantly improves matching timeliness and system load efficiency. This allows for timely implementation of prevention and control strategies while ensuring feature accuracy, preventing malicious emails from being missed, and providing efficient and real-time network security protection for traffic analysis systems.

[0108] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0109] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0110] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0111] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for matching email content features based on a traffic analysis system, characterized in that: This includes performing TCP stream reassembly on the received mirrored email protocol traffic to obtain an ordered email protocol data stream; The system parses the email header and MIME header information line by line, identifies character encoding and transmission encoding, and performs feature rule matching on the decoded email subject. It reads the email body content line by line, performs streaming decoding based on the identified transmission encoding, and maintains a cache of a preset length of bytes at the end of the previous line. This cache is combined with the current line to form a continuous data buffer, enabling continuous feature matching of cross-line keywords.

2. The email content feature matching method based on a traffic analysis system according to claim 1, characterized in that: The process of parsing the email header and MIME header information line by line includes reading the email header fields line by line using \r\n as the delimiter, extracting the From, To, Date, and Subject fields; for the Subject field, if the RFC 2047 encoding format is used, its encoding identifier is parsed to determine the character encoding and transmission encoding type, and feature rule matching is performed on the decoded plaintext.

3. The email content feature matching method based on a traffic analysis system according to claim 2, characterized in that: The identification of character encoding and transmission encoding includes identifying whether the Content-Type is a multipart type; if it is a multipart type, extracting the boundary string and using the boundary string to separate each MIME part; for each MIME part or a single non-multipart part, parsing its Content-Type, charset, and Content-Transfer-Encoding fields to determine the character encoding and transmission encoding of the current part.

4. The email content feature matching method based on a traffic analysis system according to claim 3, characterized in that: The streaming decoding based on the identified transmission encoding includes reading the email body content line by line and decoding it in real time; when the transmission encoding is Base64, streaming decoding is performed line by line by utilizing the fixed grouping characteristic of Base64 encoding where every 4 characters correspond to 3 bytes.

5. The email content feature matching method based on a traffic analysis system according to claim 4, characterized in that: The step of maintaining a cache of a preset length of bytes at the end of the previous line and combining it with the current line to form a continuous data buffer for continuous feature matching of cross-line keywords includes: maintaining a 64-byte cache at the end of the previous line and combining it with the current line to form a continuous data buffer for feature matching; for pure English feature rules, matching is performed directly on the decoded plaintext or original data; for feature rules containing Chinese or mixed Chinese and English, user input rules are stored in UTF-8 encoding by default; during matching, the rule is converted into the character encoding of the current MIME part before matching, and the character encoding includes at least GB2312; The feature rules are grouped according to Chinese character encoding variants, and only the rule group corresponding to the current recognition encoding is loaded for matching.

6. The email content feature matching method based on a traffic analysis system according to claim 1, characterized in that: It also includes marking the corresponding traffic flow when a feature rule is matched, and dynamically adjusting the priority of subsequent rules based on historical hit records. Specifically, this includes marking the traffic flow that matches the rule and recording the rule identifier and time; using a priority sorting mechanism for subsequent matching of the same traffic flow based on historical hit records, prioritizing the matching of rules with higher historical hit frequencies; and implementing frequency suppression for alarms of the same or similar rules for the same traffic flow to limit the number of alarm reports within a preset time period and suspend some matching operations for that traffic flow during the suppression period.

7. The email content feature matching method based on a traffic analysis system according to claim 6, characterized in that: After the email protocol interaction ends, release the cache and tag resources associated with the traffic stream.

8. A method for matching email content features based on a traffic analysis system, characterized in that: It includes a traffic reconstruction module, an encoding recognition module, and a cross-line matching module; the traffic reconstruction module is used to reconstruct the received mirrored email protocol traffic using TCP stream to obtain an ordered email protocol data stream. The encoding recognition module is used to parse the email header and MIME header information line by line, identify character encoding and transmission encoding, and perform feature rule matching on the decoded email subject; the cross-line matching module is used to read the email body content line by line, perform streaming decoding according to the identified transmission encoding, and maintain a cache of a preset length of bytes at the end of the previous line, which is combined with the current line to form a continuous data buffer to achieve continuous feature matching of cross-line keywords.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the email content feature matching method based on the traffic analysis system as described in any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps in the email content feature matching method based on the traffic analysis system as described in any one of claims 1-7.

Images