Malicious mail identification method and system based on semantic and behavior mutual authentication, medium and product

Abstract

The invention discloses a malicious email identification method and system with semantics and behaviors mutually certified, a medium and a product, and belongs to the field of email security detection.The method comprises the steps that text content and protocol structure information of a plurality of emails to be analyzed are obtained; according to each text content, generating a corresponding semantic embedding vector, and performing hierarchical residual clustering on each semantic embedding vector to obtain a semantic path corresponding to each mail; according to each piece of protocol structure information, generating a corresponding behavior embedding vector, and based on each behavior embedding vector, obtaining a reconstructed behavior embedding vector and a behavior path corresponding to each mail through generative reconstruction processing; and calculating a reconstruction error between each reconstruction behavior embedding vector and the corresponding behavior embedding vector, and performing identification according to the semantic path, the behavior path and the reconstruction error corresponding to each mail to obtain an identification result of the malicious mail. According to the invention, effective identification of highly disguised malicious mails can be realized.

Description

Technical Field

[0001] This invention belongs to the field of email security detection and relates to a method, system, medium and product for identifying malicious emails based on semantic and behavioral verification. Background Technology

[0002] Email, as a core medium for enterprise communication, handles massive amounts of business interactions daily. Its content includes text, attachments, and complex structured protocol information, making it a major entry point for cyberattacks. Modern email systems generally integrate security detection mechanisms to identify malicious activities such as phishing, ransomware, and commercial email fraud.

[0003] Among the current mainstream email threat detection methods, text semantic analysis models rely on content features to determine malice. When attackers use fluent and compliant business language, such methods are difficult to identify effectively. On the other hand, detection mechanisms based on protocol rules only verify whether the email header fields meet the preset standards. They lack the ability to distinguish delivery behaviors with abnormal combinations of protocol parameters but pass individual verifications, which allows some covert attacks to bypass detection. Summary of the Invention

[0004] This application provides a method, system, medium, and product for identifying malicious emails based on semantic and behavioral verification, which can effectively identify highly disguised malicious emails.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for identifying malicious emails based on semantic and behavioral verification, comprising: Obtain the text content and protocol structure information of several emails to be analyzed; wherein, the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint; Based on the text content, generate corresponding semantic embedding vectors, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email. Based on the protocol structure information, a corresponding behavior embedding vector is generated, and based on the behavior embedding vector, a reconstructed behavior embedding vector and the behavior path corresponding to each email are obtained through generative reconstruction processing. The reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector is calculated, and the malicious emails are identified based on the semantic path, behavior path and reconstruction error corresponding to each email.

[0006] Compared to existing technologies, the embodiments of this application have the following beneficial effects: They acquire text content and protocol structure information of several emails to be analyzed, providing a complete input foundation for dual-perspective analysis and ensuring that semantic and behavioral features cover all discriminable dimensions of the emails; they generate semantic embedding vectors based on the text content and perform hierarchical residual clustering on the semantic embedding vectors to obtain semantic paths, constructing a semantic index structure that can be drilled down layer by layer, enabling security operations to quickly locate fine-grained attack variants from macro-level business categories; and they establish a system by generating behavioral embedding vectors based on protocol structure information and obtaining reconstructed behavioral embedding vectors and behavioral paths through generative reconstruction processing based on the behavioral embedding vectors. The ability to discretize delivery behavior prototypes enables stable clustering of emails sent from the same infrastructure. By calculating the reconstruction error between the reconstructed behavior embedding vectors, the deviation of emails from the behavior manifold is quantified, thereby effectively identifying hidden threats with abnormal protocol combinations but compliant individual aspects. The identification results of malicious emails are obtained by identifying them based on semantic paths, behavior paths, and reconstruction errors, realizing a cross-validation mechanism of multi-dimensional features. The synergistic effect of the above features enables the system to penetrate semantic masquerading, detect behavioral anomalies, and cluster co-source attacks without relying on labels, thereby achieving effective identification of highly disguised malicious emails as a whole.

[0007] In some embodiments of the first aspect of this application, obtaining the text content and protocol structure information of several emails to be analyzed includes: Extract the text content of each email; wherein, the text content includes the optical character recognition text of the email subject, email body, and email attachments; Extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email to serve as the delivery protocol fingerprint for each email. Extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email to serve as the carrier structure fingerprint of each email.

[0008] Compared to existing technologies, the above embodiments have the following beneficial effects: By extracting the optical character recognition text of the email subject, body, and attachments as text content, semantic analysis is ensured to cover all readable information in the email, avoiding semantic blind spots caused by ignoring attachment content; by extracting the DKIM algorithm type, DKIM standardization method, DKIM version, and the relative order of email header fields from each email as delivery protocol fingerprints, cryptographic parameters and field temporal logic in the delivery protocol are captured, providing highly discriminative protocol features for behavioral modeling; by extracting the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email as carrier structure fingerprints, the complexity and randomness of the email carrier are quantified, enabling the system to perceive the essential structural differences between machine-generated emails and human-generated emails.

[0009] In some embodiments of the first aspect of this application, the step of generating corresponding semantic embedding vectors based on the text content and performing hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email includes: Input the text content of each email into a preset language model to generate corresponding semantic embedding vectors; The semantic embedding vectors are iterated until a preset maximum number of iterations is reached, resulting in a codebook for each iteration and a codeword index for each email in each iteration. In each iteration, all current semantic embedding vectors are clustered to obtain the codebook for this iteration and the codeword index for each email in the codebook for this iteration. The cluster center vector for each email in this iteration is determined based on the codeword index. The difference between the semantic embedding vector of each email and the cluster center vector for this iteration is calculated to obtain a residual vector, which serves as the semantic embedding vector for the next iteration. The codebook is a set of cluster center vectors. The semantic path of each email is obtained by sequentially combining all the codeword indices of each email.

[0010] Compared to existing technologies, the above embodiments have the following advantages: The email text content is input into a preset language model to generate semantic embedding vectors, utilizing large-scale semantic prior knowledge to achieve accurate vectorization of business language; Multiple rounds of iterative clustering are performed on the semantic embedding vectors, and the difference between the current vector and its cluster center vector is calculated in each round to obtain a residual vector as input for the next round, enabling fine-grained semantic differences to be separated step-by-step in the residual space; By sequentially combining the codeword indices of each email in each round to obtain a semantic path, a discrete encoding with hierarchical semantic attribution is formed, allowing security personnel to drill down from coarse-grained business categories to specific attack templates along the path, improving the response efficiency to sudden semantic variations.

[0011] In some embodiments of the first aspect of this application, the step of obtaining reconstructed behavior embedding vectors and corresponding behavior paths for each email through generative reconstruction processing based on each of the said behavior embedding vectors includes: The behavior embedding vectors corresponding to each email are encoded to obtain the corresponding latent variables; Each latent variable is subjected to progressive residual quantization using multiple prototype vector sets until a preset number of levels is reached, thus obtaining the index of each residual quantization level. Specifically, during each residual quantization level, the index of the prototype vector with the smallest distance to the current residual is obtained from the prototype vector set corresponding to that level, and the residual is updated to the difference between the original residual and the prototype vector. By combining the indices corresponding to each latent variable, the behavioral path of each email can be obtained; Based on the prototype vector combination corresponding to each behavior path, the reconstruction behavior embedding vector corresponding to each email is decoded and generated.

[0012] Compared to existing technologies, the above embodiments have the following advantages: latent variables are obtained by encoding the behavior embedding vectors, thereby compressing and denoising high-dimensional behavior features; by using multiple prototype vector sets to perform progressive residual quantization on the latent variables and selecting the prototype vector with the smallest distance from the current residual at each level, the model is forced to learn typical prototype combinations of delivery behavior, forming a stable behavior fingerprint; by combining the indexes at each level to obtain the behavior path, emails sent by the same attack tool can be accurately merged even if the content is different; by decoding the prototype vector combination corresponding to the behavior path to generate a reconstructed behavior embedding vector, a reconstruction benchmark is provided for subsequent anomaly measurement, enabling the system to have the ability to detect unknown protocol escapes without supervision.

[0013] In some embodiments of the first aspect of this application, calculating the reconstruction error between each of the reconstructed behavior embedding vectors and the corresponding behavior embedding vectors includes: For each email, calculate the Euclidean distance between its behavior embedding vector and the reconstructed behavior embedding vector, which is used as the reconstruction error of that email.

[0014] Compared with the prior art, the above embodiments have the following beneficial effects: For each email, the Euclidean distance between its behavior embedding vector and the reconstructed behavior embedding vector is calculated as the reconstruction error. The geometric distance in the vector space is used to quantify the degree of deviation of the email behavior from the normal manifold, so that the system can measure the anomaly of different protocol combinations on a uniform scale, avoid the problem of weight parameter tuning caused by the mixed loss function, and improve the robustness and interpretability of anomaly detection.

[0015] In some embodiments of the first aspect of this application, the step of identifying malicious emails based on the semantic path, behavioral path, and reconstruction error corresponding to each email includes: Emails whose semantic paths belong to a preset whitelist business category and whose reconstruction error is greater than the first threshold are marked as semantic spoofing attack emails. Emails with the same behavioral path are aggregated into the same email set. If the number of semantic path types in a certain email set is greater than the second threshold, the emails in that set are marked as same-origin attack emails. The number of emails in each leaf node of each semantic path is counted within a unit of time. If the growth rate of the number of emails in any leaf node is greater than the third threshold, the emails corresponding to that leaf node are marked as sudden threat emails. The semantic spoofing attack emails, same-origin attack emails, and sudden threat emails are merged as the identification result of malicious emails.

[0016] Compared to existing technologies, the above embodiments have the following beneficial effects: Emails whose semantic paths belong to a preset whitelist of business categories and whose reconstruction error exceeds a first threshold are marked as semantic spoofing attack emails, directly locating high-risk phishing or BEC attacks with compliant content but abnormal delivery; by aggregating emails with the same behavioral path into the same set and marking them as homo-origin attack emails when the number of semantic path types within the set exceeds a second threshold, cross-semantic variant attack merging is achieved, effectively countering attackers' strategies to evade detection through text randomization; by monitoring the growth rate of the number of emails in the leaf nodes of semantic paths and marking them as sudden threat emails when they exceed a third threshold, the system has the ability to automatically perceive new attack templates; by merging and outputting the three types of threat emails, a complete threat view covering spoofing penetration, gang tracing, and outbreak warning is formed, significantly improving the efficiency of security operations.

[0017] Secondly, the present invention also provides a malicious email identification system with mutual verification of semantics and behavior, comprising: a data acquisition module, a semantic path generation module, a behavior path generation module, and an identification output module; The data acquisition module is used to acquire the text content and protocol structure information of several emails to be analyzed; wherein the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint. The semantic path generation module is used to generate corresponding semantic embedding vectors based on the text content, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email. The behavior path generation module is used to generate corresponding behavior embedding vectors according to the protocol structure information, and to obtain reconstructed behavior embedding vectors and behavior paths corresponding to each email through generative reconstruction processing based on the behavior embedding vectors. The identification output module is used to calculate the reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector, and to identify malicious emails based on the semantic path, behavior path and reconstruction error corresponding to each email.

[0018] Compared to existing technologies, the above embodiments of this application have the following beneficial effects: They acquire text content and protocol structure information of several emails to be analyzed, providing a complete input foundation for dual-perspective analysis and ensuring that semantic and behavioral features cover all discriminable dimensions of the emails; they generate semantic embedding vectors based on the text content and perform hierarchical residual clustering on the semantic embedding vectors to obtain semantic paths, constructing a semantic index structure that can be drilled down layer by layer, enabling security operations to quickly locate fine-grained attack variants from macro-level business categories; and they establish a semantic index structure by generating behavioral embedding vectors based on protocol structure information and obtaining reconstructed behavioral embedding vectors and behavioral paths through generative reconstruction processing based on the behavioral embedding vectors. Its ability to discretize delivery behavior prototypes enables stable clustering of emails sent from the same infrastructure. By calculating the reconstruction error between the reconstructed behavior embedding vectors, it quantifies the degree of deviation of emails from the behavior manifold, thereby effectively identifying hidden threats with abnormal protocol combinations but compliant individual aspects. By identifying malicious emails based on semantic paths, behavior paths, and reconstruction errors, it achieves a cross-validation mechanism of multi-dimensional features. The synergistic effect of these features enables the system to simultaneously penetrate semantic masquerading, detect behavioral anomalies, and cluster concordant attacks without relying on labels, thus achieving effective identification of highly disguised malicious emails as a whole.

[0019] In some embodiments of the second aspect of this application, the data acquisition module includes: a first extraction unit, a second extraction unit, and a third extraction unit; The first extraction unit is used to extract the text content of each email; wherein the text content includes the email subject, email body and optical character recognition text of email attachments; The second extraction unit is used to extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email, as the delivery protocol fingerprint of each email. The third extraction unit is used to extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email, as the carrier structure fingerprint of each email.

[0020] Compared to existing technologies, the above embodiments have the following beneficial effects: By extracting the optical character recognition text of the email subject, body, and attachments as text content, semantic analysis is ensured to cover all readable information in the email, avoiding semantic blind spots caused by ignoring attachment content; by extracting the DKIM algorithm type, DKIM standardization method, DKIM version, and the relative order of email header fields from each email as delivery protocol fingerprints, cryptographic parameters and field temporal logic in the delivery protocol are captured, providing highly discriminative protocol features for behavioral modeling; by extracting the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email as carrier structure fingerprints, the complexity and randomness of the email carrier are quantified, enabling the system to perceive the essential structural differences between machine-generated emails and human-generated emails.

[0021] Thirdly, the present invention also provides a computer program product, including a computer program or instructions, characterized in that, when the computer program or instructions are executed, they implement any one of the semantic and behavioral mutual verification malicious email identification methods of the present invention.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any one of the semantic and behavioral verification malicious email identification methods of this invention. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a method for identifying malicious emails based on semantic and behavioral verification, provided in some embodiments of the present invention.

[0024] Figure 2 This is a schematic diagram of the structure of a malicious email identification system that uses semantic and behavioral verification, provided in some embodiments of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: Please refer to Figure 1 To effectively identify highly disguised malicious emails, an embodiment of the present invention provides a method for identifying malicious emails that verifies both semantics and behavior, comprising steps S1 to S4: Step S1: Obtain the text content and protocol structure information of several emails to be analyzed; wherein, the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint.

[0027] Furthermore, step S1 can be implemented through the following preferred embodiments, including steps S11-S13, as follows: S11: Extract the text content of each email; wherein, the text content includes the optical character recognition text of the email subject, email body, and email attachments; S12: Extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email to serve as the delivery protocol fingerprint for each email. S13: Extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email to serve as the carrier structure fingerprint of each email.

[0028] In this preferred embodiment, the text content is extracted from the email subject, body, and attachments using optical character recognition (OCR) to ensure that semantic analysis covers all readable information in the email, avoiding semantic blind spots caused by ignoring attachment content. The delivery protocol fingerprint is obtained by extracting the DKIM algorithm type, DKIM standardization method, DKIM version, and the relative order of email header fields from each email. This captures the cryptographic parameters and temporal logic of the delivery protocol, providing highly discriminative protocol features for behavioral modeling. Furthermore, the carrier structure fingerprint is obtained by extracting the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email. This quantifies the complexity and randomness of the email carrier, enabling the system to perceive the essential structural differences between machine-generated and manually generated emails.

[0029] Step S2: Generate corresponding semantic embedding vectors based on the text content, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email.

[0030] Furthermore, step S2 can be implemented through the following preferred embodiments, including steps S21-S23, as follows: S21: Input the text content of each email into the preset language model to generate the corresponding semantic embedding vector; S22: Iterate through each of the semantic embedding vectors until a preset maximum number of iterations is reached, obtaining the codebook for each iteration and the codeword index of each email in each iteration; wherein, in each iteration, cluster all current semantic embedding vectors to obtain the codebook for this round and the codeword index of each email in the codebook for this round, and determine the cluster center vector to which each email belongs in this round based on the codeword index, calculate the difference between the semantic embedding vector of each email and the cluster center vector to which it belongs in this round, and obtain the residual vector, which is used as the semantic embedding vector for the next iteration; the codebook is a set of cluster center vectors; S23: Combine all the codeword indices of each email in sequence to obtain the semantic path corresponding to each email.

[0031] In this preferred embodiment, the email text content is input into a preset language model to generate semantic embedding vectors. Large-scale semantic prior knowledge is used to accurately vectorize business language. The semantic embedding vectors are subjected to multiple rounds of iterative clustering, and the difference between the current vector and the center vector of its cluster is calculated in each round to obtain a residual vector as the input for the next round. This allows fine-grained semantic differences to be separated step by step in the residual space. By combining the codeword indices of each email in each round in sequence, a semantic path is obtained, forming a discrete code with hierarchical semantic affiliation. This allows security personnel to drill down from coarse-grained business categories to specific attack templates along the path, improving the response efficiency to sudden semantic variants.

[0032] Step S3: Generate corresponding behavior embedding vectors based on the protocol structure information, and obtain reconstructed behavior embedding vectors and behavior paths corresponding to each email through generative reconstruction processing based on the behavior embedding vectors.

[0033] Furthermore, in step S3, generative reconstruction can be implemented through the following preferred embodiments, including steps S31-S34, as follows: S31: Encode the behavior embedding vectors corresponding to each email to obtain the corresponding latent variables; S32: Use multiple prototype vector sets to perform residual quantization on each latent variable step by step until the preset number of levels is reached, and obtain the index of each level of residual quantization; wherein, during each level of residual quantization, obtain the index of the prototype vector with the smallest distance from the current residual in the prototype vector set corresponding to that level, and update the residual to the difference between the original residual and the prototype vector. S33: Combine the indices corresponding to each latent variable to obtain the behavior path of each email; S34: Based on the prototype vector combination corresponding to each behavior path, decode and generate the reconstructed behavior embedding vector corresponding to each email.

[0034] In this preferred embodiment, latent variables are obtained by encoding the behavior embedding vectors, thereby compressing and denoising high-dimensional behavior features. By using multiple prototype vector sets to perform progressive residual quantization on the latent variables and selecting the prototype vector with the smallest distance from the current residual at each level, the model is forced to learn typical prototype combinations of delivery behavior, forming a stable behavior fingerprint. By combining the indices at each level to obtain the behavior path, emails sent by the same attack tool can be accurately merged even if their content is different. By decoding the prototype vector combination corresponding to the behavior path to generate a reconstructed behavior embedding vector, a reconstruction benchmark is provided for subsequent anomaly measurement, enabling the system to have the ability to detect unknown protocol escapes without supervision.

[0035] Step S4: Calculate the reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector, and identify malicious emails based on the semantic path, behavior path and reconstruction error corresponding to each email.

[0036] Furthermore, step S4 can be implemented through the following preferred embodiments, including steps S41-S45, as follows: S41: For each email, calculate the Euclidean distance between its behavior embedding vector and the reconstructed behavior embedding vector, and use it as the reconstruction error of the email.

[0037] In this preferred embodiment, the Euclidean distance between the behavior embedding vector and the reconstructed behavior embedding vector of each email is calculated as the reconstruction error. The geometric distance in the vector space is used to quantify the degree of deviation of the email behavior from the normal manifold, so that the system can measure the anomaly of different protocol combinations on a uniform scale, avoid the problem of weight parameter tuning caused by the mixed loss function, and improve the robustness and interpretability of anomaly detection.

[0038] S42: Filter emails whose semantic paths belong to the preset whitelist business categories and whose reconstruction error is greater than the first threshold, and mark them as semantic spoofing attack emails. S43: Aggregate emails with the same behavioral path into the same email set. If the number of semantic path types in a certain email set is greater than the second threshold, then mark the emails in the set as same-origin attack emails. S44: Count the number of emails in the leaf nodes of each semantic path within a unit of time. If the growth rate of the number of emails in any leaf node is greater than the third threshold, then mark the emails corresponding to that leaf node as sudden threat emails. S45: Merge the semantic spoofing attack emails, same-origin attack emails, and sudden threat emails as the identification result of malicious emails.

[0039] In this preferred embodiment, emails whose semantic paths belong to a preset whitelist of business categories and whose reconstruction error is greater than a first threshold are marked as semantic spoofing attack emails, directly locating high-risk phishing or BEC attacks with compliant content but abnormal delivery; by aggregating emails with the same behavioral path into the same set and marking them as homo-origin attack emails when the number of semantic path types in the set exceeds a second threshold, cross-semantic variant attack merging is achieved, effectively combating attackers' strategies to evade detection through text randomization; by monitoring the growth rate of the number of emails in the leaf nodes of semantic paths and marking them as sudden threat emails when they exceed a third threshold, the system has the ability to automatically detect new attack templates; by merging and outputting the three types of threat emails, a complete threat view covering spoofing penetration, gang tracing, and outbreak warning is formed, significantly improving the handling efficiency of security operations.

[0040] In practice, considering that malicious emails have evolved into two main forms in the existing adversarial environment: 1. Proliferation of variants: Using tools (such as large models) to automatically generate massive amounts of copywriting tweaks, attempting to overwhelm the defense system.

[0041] 2. Covert Attacks: Exploiting zero-day vulnerabilities or extremely covert protocol configurations (such as using legitimate cloud service providers for forwarding) to circumvent reputation checks.

[0042] Existing solutions, such as supervised learning-based classifiers, train binary classification models by collecting black and white samples. However, this approach heavily relies on known labels, rendering it ineffective against zero-day attacks; furthermore, the model is a black box, unable to explain why it was deemed malicious. Another example is traditional K-Means clustering analysis, which flattens email vectors and selects center points for review. Its drawback is fixed granularity, failing to provide a hierarchical view of "looking at the big picture first, then the details"; and Euclidean distance is difficult to measure complex discrete protocol features (such as header order). Yet another approach is rule-based protocol filtering, which checks SPF and DKIM status using hard-coded rules. Its drawbacks include lag, with the rule base always falling behind attackers' bypass techniques; and the inability to automatically detect anomalies through data distribution.

[0043] To address the above shortcomings, this invention provides a malicious email identification scheme based on a dual-perspective (semantic-behavioral) approach, which generally includes: I. Constructing a hierarchical semantic index supporting multi-level drill-down: Text semantic features utilize RQ-KMeans (Residual Quantization K-Means), and behavioral features utilize RQ-VAE (Residual Quantization Variational Autoencoder) to encode email content into a tree-like semantic codebook that can be expanded layer by layer. This achieves a unified view from "coarse-grained business classification" to "fine-grained variant differences," helping operations personnel quickly locate high-value attack variants with specific semantic features in massive amounts of data.

[0044] II. Constructing an abnormal delivery detection mechanism based on behavioral baselines: Utilizing RQ-VAE (Residual Quantization Variational Autoencoder) to generatively model the delivery protocol and carrier structure, learning the delivery behavior distribution of normal emails. DKIM details and header fingerprints are included in the reconstruction scope, and the reconstruction loss is used to directly quantify the degree of abnormality in delivery behavior, thereby discovering unknown threats that are semantically normal but have suspicious delivery paths.

[0045] 3. Establish a "semantic-behavior" mutual verification mechanism: By comparing the distribution consistency between the semantic codebook and the behavior codebook, automatically identify "wolves in sheep's clothing (normal semantics) (abnormal behavior)" and "wolves in packs (behavior homology)".

[0046] The specific solution, combining steps S1-S4 above, is as follows: I. Dual-view feature decoupling: Perspective A: Semantic Content Flow Extract email text, such as subject, body, and attached OCR text, and use a pre-trained Transformer model to generate semantic embedding vectors $V_{sem}$.

[0047] Perspective B: Structural behavioral flow, including: Extract delivery protocol fingerprints: DKIM version / algorithm / canonicalization method, header key field sorting (such as the relative position of Received and From), etc.

[0048] Extract the carrier structure fingerprint: HTML DOM tree depth, tag entropy value, and MIME structure tree.

[0049] The discrete and continuous features mentioned above are concatenated and mapped into a behavior embedding vector $V_{beh}$ in the embedding layer.

[0050] II. Hierarchical Residual Indexing of Semantic Streams (RQ-KMeans): This method trains $V_{sem}$ using multi-level residual quantization to construct an interpretable index. Level-1 clustering: Perform K-Means on $V_{sem}$ to obtain a coarse-grained codebook $C_1$ (corresponding to major business categories, such as "finance" and "HR").

[0051] Residual calculation: Calculate the residual $R_1 = V_{sem} - C_1(i)$.

[0052] Level-2 clustering: Clustering $R_1$ yields a fine-grained codebook $C_2$ (corresponding to specific wording, such as "overdue invoice" or "salary adjustment").

[0053] Recursive iteration: Generate semantic paths of depth $M$: $Path_{sem} = [idx_1, idx_2, ..., idx_M]$.

[0054] After iteration, operators can drill down the path. If a path suddenly branches out into a large number of samples at $Level-3$, it often means that a new attack variant has emerged.

[0055] III. Manifold Reconstruction and Anomaly Measurement (RQ-VAE) of Behavioral Flow: Generative modeling of $V_{beh}$ aims to learn the distribution manifold of normal delivery behavior. Encoder: Maps the behavior vector $V_{beh}$ to a low-dimensional latent space.

[0056] Residual Quantizer: Approximates latent variables by summing multiple codebook vectors within the latent space. This forces the model to learn "prototypes" of behavioral features.

[0057] Decoder: Attempts to reconstruct the original DKIM, Header sequence, and other features.

[0058] Anomaly detection: Calculate the reconstruction error Loss between the input $V_{beh}$ and the reconstructed output $V'_{beh}$.

[0059] Output: Not only outputs the Loss, but also generates a discrete sequence of behavioral codebooks $Path_{beh} = [q_1, q_2,..., q_D]$.

[0060] High loss indicates the discovery of unknown threats: if an attacker uses a script to send emails, the header order may be fixed and different from that of a standard client, or the DKIM signature may pass verification but use rare parameter combinations. These "unseen combinations" prevent the VAE from being effectively reconstructed, resulting in high loss and directly exposing new types of attacks.

[0061] Behavioral codebook sequences can be used to trace the origins of criminal groups: the generated $Path_{beh}$ can be used as behavioral fingerprints to aggregate emails with the same delivery behavior characteristics (such as the same Header fingerprint + DKIM features) to achieve cross-semantic homology analysis.

[0062] IV. Based on the high-value sample mining strategy of dual-stream mutual verification, after obtaining the descriptor $\{Path_{sem}, Path_{beh}, Loss\}$ for each email, the following strategy is executed: Strategy I: Semantic Spoofing Penetration Logic: Filter samples that belong to "high-frequency whitelisted business" (such as daily notifications) but have extremely high loss.

[0063] Meaning: The content is perfectly normal, but the delivery behavior is extremely abnormal, which is highly likely to be a BEC or advanced phishing scam.

[0064] Strategy II: Tracing the Origins of Homogeneous Gangs: Logic: Find email groups with a high degree of consistency in $Path_{beh}$ (behavior codebook path).

[0065] Meaning: Regardless of semantic changes (random text), as long as the attack tools or infrastructure remain the same, the behavioral fingerprint is fixed. This can aggregate scattered attacks into "gang attacks".

[0066] Strategy III: Semantic Burst Warning Logic: Monitor the leaf nodes of the semantic tree RQ-KMeans. If the sample size of a certain leaf node surges in a short period of time, prompt the operations staff to extract the "central sample" of that node to generate interception rules.

[0067] Through the above implementation methods, this invention uses explicit clustering (RQ-KMeans) for "content" to facilitate human understanding and implicit generation (RQ-VAE) for "behavior" to facilitate machine capture of distribution, thus achieving the best match between algorithm characteristics and data characteristics.

[0068] In summary, compared with existing technologies, the embodiments of this application have the following beneficial effects: They acquire text content and protocol structure information of several emails to be analyzed, providing a complete input foundation for dual-perspective analysis and ensuring that semantic and behavioral features cover all discriminable dimensions of the emails; they generate semantic embedding vectors based on the text content and perform hierarchical residual clustering on the semantic embedding vectors to obtain semantic paths, constructing a semantic index structure that can be drilled down layer by layer, enabling security operations to quickly locate fine-grained attack variants from macro-level business categories; and they generate behavioral embedding vectors based on protocol structure information and obtain reconstructed behavioral embedding vectors and behavioral paths through generative reconstruction processing based on the behavioral embedding vectors, thus building... The system establishes the ability to discretize delivery behavior prototypes, enabling stable clustering of emails sent from the same infrastructure. By calculating the reconstruction error between the reconstructed behavior embedding vectors, the system quantifies the degree of deviation of emails from the behavior manifold, thereby effectively identifying hidden threats with abnormal protocol combinations but compliant individual components. The system obtains the identification results of malicious emails by identifying them based on semantic paths, behavior paths, and reconstruction errors, realizing a cross-validation mechanism of multi-dimensional features. The synergistic effect of these features enables the system to simultaneously penetrate semantic masquerading, detect behavioral anomalies, and cluster concordant attacks without relying on labels, thus achieving effective identification of highly disguised malicious emails as a whole.

[0069] Example 2: Please refer to Figure 2 Based on the same inventive concept, the present invention discloses a malicious email identification system with semantic and behavioral mutual verification, comprising: a data acquisition module M1, a semantic path generation module M2, a behavioral path generation module M3, and an identification output module M4. The data acquisition module M1 is used to acquire the text content and protocol structure information of several emails to be analyzed; wherein the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint.

[0070] Furthermore, the data acquisition module M1 includes: a first extraction unit, a second extraction unit, and a third extraction unit; The first extraction unit is used to extract the text content of each email; wherein the text content includes the email subject, email body and optical character recognition text of email attachments; The second extraction unit is used to extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email, as the delivery protocol fingerprint of each email. The third extraction unit is used to extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email, as the carrier structure fingerprint of each email.

[0071] In this preferred embodiment, the text content is extracted from the email subject, body, and attachments using optical character recognition (OCR) to ensure that semantic analysis covers all readable information in the email, avoiding semantic blind spots caused by ignoring attachment content. The delivery protocol fingerprint is obtained by extracting the DKIM algorithm type, DKIM standardization method, DKIM version, and the relative order of email header fields from each email. This captures the cryptographic parameters and temporal logic of the delivery protocol, providing highly discriminative protocol features for behavioral modeling. Furthermore, the carrier structure fingerprint is obtained by extracting the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email. This quantifies the complexity and randomness of the email carrier, enabling the system to perceive the essential structural differences between machine-generated and manually generated emails.

[0072] The semantic path generation module M2 is used to generate corresponding semantic embedding vectors based on the text content, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email.

[0073] Furthermore, the semantic path generation module M2 includes: an embedding unit, a clustering unit, and a path output unit; The embedding unit is used to input the text content of each email into a preset language model to generate a corresponding semantic embedding vector. The clustering unit is used to iterate over each semantic embedding vector until a preset maximum number of iterations is reached, obtaining the codebook for each iteration and the codeword index of each email in each iteration. Specifically, in each iteration, all current semantic embedding vectors are clustered to obtain the codebook for this iteration and the codeword index of each email in the codebook for this iteration. The cluster center vector to which each email belongs in this iteration is determined based on the codeword index. The difference between the semantic embedding vector of each email and the cluster center vector to which it belongs in this iteration is calculated to obtain a residual vector, which serves as the semantic embedding vector for the next iteration. The codebook is a set of cluster center vectors. The path output unit is used to combine all the codeword indices of each email in sequence to obtain the semantic path corresponding to each email.

[0074] In this preferred embodiment, the email text content is input into a preset language model to generate semantic embedding vectors. Large-scale semantic prior knowledge is used to accurately vectorize business language. The semantic embedding vectors are subjected to multiple rounds of iterative clustering, and the difference between the current vector and the center vector of its cluster is calculated in each round to obtain a residual vector as the input for the next round. This allows fine-grained semantic differences to be separated step by step in the residual space. By combining the codeword indices of each email in each round in sequence, a semantic path is obtained, forming a discrete code with hierarchical semantic affiliation. This allows security personnel to drill down from coarse-grained business categories to specific attack templates along the path, improving the response efficiency to sudden semantic variants.

[0075] The behavior path generation module M3 is used to generate corresponding behavior embedding vectors according to the protocol structure information, and to obtain reconstructed behavior embedding vectors and behavior paths corresponding to each email through generative reconstruction processing based on the behavior embedding vectors.

[0076] Furthermore, the behavior path generation module M3 includes: an encoding unit, a step-by-step residual unit, a combination unit, and a decoding unit; The encoding unit is used to encode the behavior embedding vector corresponding to each email to obtain the corresponding latent variables. The progressive residual unit is used to perform progressive residual quantization on each latent variable using multiple prototype vector sets until a preset number of levels is reached, and to obtain the index of each level of residual quantization. Specifically, during each level of residual quantization, the index of the prototype vector with the smallest distance from the current residual is obtained from the prototype vector set corresponding to that level, and the residual is updated to the difference between the original residual and the prototype vector. The combination unit is used to combine the indices corresponding to each latent variable to obtain the behavior path of each email. The decoding unit is used to decode and generate the reconstructed behavior embedding vector for each email based on the prototype vector combination corresponding to each behavior path.

[0077] In this preferred embodiment, latent variables are obtained by encoding the behavior embedding vectors, thereby compressing and denoising high-dimensional behavior features. By using multiple prototype vector sets to perform progressive residual quantization on the latent variables and selecting the prototype vector with the smallest distance from the current residual at each level, the model is forced to learn typical prototype combinations of delivery behavior, forming a stable behavior fingerprint. By combining the indices at each level to obtain the behavior path, emails sent by the same attack tool can be accurately merged even if their content is different. By decoding the prototype vector combination corresponding to the behavior path to generate a reconstructed behavior embedding vector, a reconstruction benchmark is provided for subsequent anomaly measurement, enabling the system to have the ability to detect unknown protocol escapes without supervision.

[0078] The identification output module M4 is used to calculate the reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector, and to identify malicious emails based on the semantic path, behavior path and reconstruction error corresponding to each email.

[0079] Furthermore, the identification output module M4 includes: a distance calculation unit; The distance calculation unit is used to calculate the Euclidean distance between the behavior embedding vector and the reconstructed behavior embedding vector of each email, as the reconstruction error of the email.

[0080] In this preferred embodiment, the Euclidean distance between the behavior embedding vector and the reconstructed behavior embedding vector of each email is calculated as the reconstruction error. The geometric distance in the vector space is used to quantify the degree of deviation of the email behavior from the normal manifold, so that the system can measure the anomaly of different protocol combinations on a uniform scale, avoid the problem of weight parameter tuning caused by the mixed loss function, and improve the robustness and interpretability of anomaly detection.

[0081] Furthermore, the identification output module M4 also includes: a first filtering unit, a second filtering unit, a third filtering unit, and a result integration unit; The first filtering unit is used to filter emails whose semantic paths belong to a preset whitelist business category and whose reconstruction error is greater than a first threshold, and mark them as semantic spoofing attack emails. The second filtering unit is used to aggregate emails with the same behavioral path into the same email set. If the number of semantic path types in a certain email set is greater than the second threshold, the emails in the set are marked as same-origin attack emails. The third filtering unit is used to count the number of emails in the leaf nodes of each semantic path within a unit of time. If the growth rate of the number of emails in any leaf node is greater than the third threshold, the emails corresponding to that leaf node are marked as sudden threat emails. The result integration unit is used to merge the semantic spoofing attack emails, same-origin attack emails, and sudden threat emails as the identification result of malicious emails.

[0082] In this preferred embodiment, emails whose semantic paths belong to a preset whitelist of business categories and whose reconstruction error is greater than a first threshold are marked as semantic spoofing attack emails, directly locating high-risk phishing or BEC attacks with compliant content but abnormal delivery; by aggregating emails with the same behavioral path into the same set and marking them as homo-origin attack emails when the number of semantic path types in the set exceeds a second threshold, cross-semantic variant attack merging is achieved, effectively combating attackers' strategies to evade detection through text randomization; by monitoring the growth rate of the number of emails in the leaf nodes of semantic paths and marking them as sudden threat emails when they exceed a third threshold, the system has the ability to automatically detect new attack templates; by merging and outputting the three types of threat emails, a complete threat view covering spoofing penetration, gang tracing, and outbreak warning is formed, significantly improving the handling efficiency of security operations.

[0083] In summary, compared with existing technologies, the embodiments of this application have the following beneficial effects: They acquire text content and protocol structure information of several emails to be analyzed, providing a complete input foundation for dual-perspective analysis and ensuring that semantic and behavioral features cover all discriminable dimensions of the emails; they generate semantic embedding vectors based on the text content and perform hierarchical residual clustering on the semantic embedding vectors to obtain semantic paths, constructing a semantic index structure that can be drilled down layer by layer, enabling security operations to quickly locate fine-grained attack variants from macro-level business categories; and they establish a semantic index structure by generating behavioral embedding vectors based on protocol structure information and obtaining reconstructed behavioral embedding vectors and behavioral paths through generative reconstruction processing based on the behavioral embedding vectors. Its ability to discretize delivery behavior prototypes enables stable clustering of emails sent from the same infrastructure. By calculating the reconstruction error between the reconstructed behavior embedding vectors, it quantifies the degree of deviation of emails from the behavior manifold, thereby effectively identifying hidden threats with abnormal protocol combinations but compliant individual aspects. By identifying malicious emails based on semantic paths, behavior paths, and reconstruction errors, it achieves a cross-validation mechanism of multi-dimensional features. The synergistic effect of these features enables the system to simultaneously penetrate semantic masquerading, detect behavioral anomalies, and cluster concordant attacks without relying on labels, thus achieving effective identification of highly disguised malicious emails as a whole.

[0084] Example 3: This invention also provides a computer program product, including a computer program or instructions, capable of running on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute any of the semantic and behavioral verification malicious email identification methods of this invention.

[0085] Example 4: This invention also provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is run on a semantically and behaviorally verified malicious email identification system, the semantically and behaviorally verified malicious email identification system performs one of the semantically and behaviorally verified malicious email identification methods described in any of the above method embodiments.

[0086] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. Similarly, for the purpose of simplification and aiding understanding of one or more aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0087] Those skilled in the art will understand that the modules in the system of the embodiments can be adaptively changed and placed in one or more systems different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. A method for identifying malicious emails through semantic and behavioral verification, characterized in that, include: Obtain the text content and protocol structure information of several emails to be analyzed; wherein, the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint; Based on the text content, generate corresponding semantic embedding vectors, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email. Based on the protocol structure information, a corresponding behavior embedding vector is generated, and based on the behavior embedding vector, a reconstructed behavior embedding vector and the behavior path corresponding to each email are obtained through generative reconstruction processing. The reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector is calculated, and the malicious emails are identified based on the semantic path, behavior path and reconstruction error corresponding to each email.

2. The malicious email identification method based on semantic and behavioral verification as described in claim 1, characterized in that, The process of obtaining the text content and protocol structure information of several emails to be analyzed includes: Extract the text content of each email; wherein, the text content includes the optical character recognition text of the email subject, email body, and email attachments; Extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email to serve as the delivery protocol fingerprint for each email. Extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email to serve as the carrier structure fingerprint of each email.

3. The malicious email identification method based on semantic and behavioral verification as described in claim 1, characterized in that, The step of generating corresponding semantic embedding vectors based on the text content, and performing hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email includes: Input the text content of each email into a preset language model to generate corresponding semantic embedding vectors; The semantic embedding vectors are iterated until a preset maximum number of iterations is reached, resulting in a codebook for each iteration and a codeword index for each email in each iteration. In each iteration, all current semantic embedding vectors are clustered to obtain the codebook for this iteration and the codeword index for each email in the codebook for this iteration. The cluster center vector for each email in this iteration is determined based on the codeword index. The difference between the semantic embedding vector of each email and the cluster center vector for this iteration is calculated to obtain a residual vector, which serves as the semantic embedding vector for the next iteration. The codebook is a set of cluster center vectors. The semantic path of each email is obtained by sequentially combining all the codeword indices of each email.

4. The malicious email identification method based on semantic and behavioral verification as described in claim 1, characterized in that, The process of obtaining reconstructed behavior embedding vectors and corresponding behavior paths for each email through generative reconstruction based on the aforementioned behavior embedding vectors includes: The behavior embedding vectors corresponding to each email are encoded to obtain the corresponding latent variables; Each latent variable is subjected to progressive residual quantization using multiple prototype vector sets until a preset number of levels is reached, thus obtaining the index of each residual quantization level. Specifically, during each residual quantization level, the index of the prototype vector with the smallest distance to the current residual is obtained from the prototype vector set corresponding to that level, and the residual is updated to the difference between the original residual and the prototype vector. By combining the indices corresponding to each latent variable, the behavioral path of each email can be obtained; Based on the prototype vector combination corresponding to each behavior path, the reconstruction behavior embedding vector corresponding to each email is decoded and generated.

5. The malicious email identification method based on semantic and behavioral verification as described in claim 1, characterized in that, The calculation of the reconstruction error between each of the reconstructed behavior embedding vectors and the corresponding behavior embedding vectors includes: For each email, calculate the Euclidean distance between its behavior embedding vector and the reconstructed behavior embedding vector, which is used as the reconstruction error of that email.

6. The malicious email identification method based on semantic and behavioral mutual verification as described in claim 1, characterized in that, The process of identifying malicious emails based on their semantic paths, behavioral paths, and reconstruction errors includes: Emails whose semantic paths belong to a preset whitelist business category and whose reconstruction error is greater than the first threshold are marked as semantic spoofing attack emails. Emails with the same behavioral path are aggregated into the same email set. If the number of semantic path types in a certain email set is greater than the second threshold, the emails in that set are marked as same-origin attack emails. The number of emails in each leaf node of each semantic path is counted within a unit of time. If the growth rate of the number of emails in any leaf node is greater than the third threshold, the emails corresponding to that leaf node are marked as sudden threat emails. The semantic spoofing attack emails, same-origin attack emails, and sudden threat emails are merged as the identification result of malicious emails.

7. A malicious email identification system that verifies both semantic and behavioral aspects, characterized in that, include: The module consists of a data acquisition module, a semantic path generation module, a behavior path generation module, and a recognition output module. The data acquisition module is used to acquire the text content and protocol structure information of several emails to be analyzed; wherein the protocol structure information includes delivery protocol fingerprint and carrier structure fingerprint. The semantic path generation module is used to generate corresponding semantic embedding vectors based on the text content, and perform hierarchical residual clustering on the semantic embedding vectors to obtain the semantic path corresponding to each email. The behavior path generation module is used to generate corresponding behavior embedding vectors according to the protocol structure information, and to obtain reconstructed behavior embedding vectors and behavior paths corresponding to each email through generative reconstruction processing based on the behavior embedding vectors. The identification output module is used to calculate the reconstruction error between each reconstructed behavior embedding vector and the corresponding behavior embedding vector, and to identify malicious emails based on the semantic path, behavior path and reconstruction error corresponding to each email.

8. The malicious email identification system with semantic and behavioral mutual verification as described in claim 7, characterized in that, The data acquisition module includes: a first extraction unit, a second extraction unit, and a third extraction unit; The first extraction unit is used to extract the text content of each email; wherein the text content includes the email subject, email body and optical character recognition text of email attachments; The second extraction unit is used to extract the DKIM algorithm type, DKIM standardization method, DKIM version, and relative order of email header fields from each email, as the delivery protocol fingerprint of each email. The third extraction unit is used to extract the HTML document object model tree depth, HTML tag entropy value, and MIME nesting level from each email, as the carrier structure fingerprint of each email.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement a malicious email identification method based on semantic and behavioral verification as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a malicious email identification method based on semantic and behavioral verification as described in any one of claims 1-6.

Images