A spam detection method based on email service asset co-occurrence graph

Abstract

The present invention discloses a spam detection method based on an email service asset co-occurrence graph. The method includes: an offline training phase for a machine learning model: obtaining header information for each sample email in a sample set; the sample set includes several legitimate emails and several spam emails; extracting preset fields from the header information of each sample email, constructing a co-occurrence subgraph corresponding to each sample email, and then merging the obtained co-occurrence subgraphs to obtain an email service asset co-occurrence graph; based on the co-occurrence subgraphs of the sample emails and the email service asset co-occurrence graph, using subgraph representation learning technology to learn the subgraph representation of the corresponding sample email and label it; using the labeled subgraph representation to train the machine learning model; an online detection phase: inputting the subgraph representation of the email to be detected into the trained machine learning model, and outputting the probability that the email to be detected is spam. The present invention makes full use of email asset information for spam detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet mail transmission and processing, and in particular relates to a spam detection method based on a mail service asset co-occurrence graph. Background Art

[0002] Email is a widely used communication tool often used to transmit important information. However, spam has become a prominent problem in email communications. Spammers collect email addresses from various websites, social media platforms, and malware, and then send spam to targeted addresses. This not only wastes users' time and energy, but also wastes email service storage capacity and network bandwidth. Some spam messages even pretend to be from legitimate entities, deceiving users into disclosing sensitive information, downloading malicious attachments, or transferring money, causing significant losses.

[0003] To effectively combat the threat posed by spam, security researchers have developed numerous spam detection and prevention technologies. However, spam delivery methods are constantly evolving and becoming increasingly sophisticated. In recent years, a number of automated tools have emerged to generate forged emails. Some can send emails directly through web pages, while others require senders to set up and configure their own servers. Still others use scripts to send forged emails. These tools have lowered the technical barriers to spam delivery. The content of these constructed emails closely mimics legitimate senders, often employing the latest sender spoofing techniques to circumvent defenses.

[0004] With the development of natural language processing (NLP) and machine learning (ML) technologies, many frameworks for detecting and classifying spam based on artificial intelligence have emerged.

[0005] Fang et al. proposed a spam detection model, THEMIS, based on an improved multi-level vector recurrent convolutional neural network (RCNN) model and an attention mechanism. The model simultaneously models emails at the header, body, character, and word levels. Experimental results show that THEMIS achieves an overall accuracy of 99.848%.

[0006] Almomani et al. proposed a framework for online spam detection. A total of 21 features were extracted from each email, which were divided into four groups: spam, body, URL, and header features.

[0007] Based on a set of features from email clients and email transmission processes, Gascon et al. developed a method that can learn the historical communication patterns of a large number of senders to distinguish different senders and identify forged emails. The method can distinguish thousands of senders and achieve a 90% detection rate in experiments.

[0008] However, over time, many existing methods have become unable to cope with the ever-changing content of spam. Detection methods based on email features are susceptible to forgery and imitation, and some algorithms have high computational costs. Detection methods based on attack vectors slow down detection by analyzing attack vectors and limit detection scope. Sender identity-based identification methods require sender-related attributes and behavioral data, which is difficult to obtain for privacy reasons. Summary of the Invention

[0009] To address the low accuracy and limited versatility of current spam detection methods based on email content and metadata, the present invention aims to provide a spam detection method based on an email service asset co-occurrence graph. This method constructs an email service asset co-occurrence graph based on email header fields, employs graph representation learning techniques to learn subgraph representations, and fully utilizes email service asset information and email content for spam detection.

[0010] To achieve the above objectives, the present invention provides a spam detection method based on the email service asset co-occurrence graph, which includes two stages:

[0011] 1. Offline training phase of machine learning model

[0012] 1) Obtain header information of several legitimate / spam emails;

[0013] 2) According to preset rules, extract preset fields from the header information of each email, construct a co-occurrence subgraph of the corresponding email, and then merge the obtained co-occurrence subgraphs to obtain an email service asset co-occurrence graph;

[0014] 3) Based on the co-occurrence subgraph of the sample emails and the co-occurrence graph of the email service assets, use subgraph representation learning technology to learn the subgraph representation corresponding to each email; subgraph representation learning technology can be found in Adhikari, Bijaya et al. "Sub2Vec: Feature Learning for Subgraphs." PAKDD (2018).

[0015] 4) Input the subgraph representation into the machine learning model and train the machine learning model using the labeled subgraph representation.

[0016] 2. Online spam detection stage

[0017] 1) Obtain the header information of the email to be tested;

[0018] 2) According to the preset rules, extract the preset field information in the header information of the email to be detected and construct the asset co-occurrence subgraph;

[0019] 3) Using subgraph representation learning technology to learn the subgraph representation of the email to be detected;

[0020] 4) The subgraph representation is input into the machine learning model trained in stage 1, and the output is the probability that the email is spam. If the probability is greater than the preset threshold, the email is determined to be spam.

[0021] The technical solution of the present invention is:

[0022] A spam detection method based on an email service asset co-occurrence graph comprises the following steps:

[0023] Offline training phase of machine learning model:

[0024] Obtaining header information of each sample email in a sample set; the sample set includes a number of legitimate emails and a number of spam emails;

[0025] Extracting preset fields from the header information of each sample email, constructing a co-occurrence subgraph corresponding to each sample email, and then merging the obtained co-occurrence subgraphs to obtain an email service asset co-occurrence graph;

[0026] Based on the co-occurrence subgraph of the sample emails and the email service asset co-occurrence graph, using subgraph representation learning technology, learn the subgraph representation of the corresponding sample emails and label them;

[0027] Train machine learning models using labeled subgraph representations;

[0028] Online detection stage:

[0029] Get the header information of the email to be tested;

[0030] Extracting preset field information from the header information of the email to be detected and constructing an asset co-occurrence subgraph of the email to be detected;

[0031] Based on the co-occurrence subgraph of the email to be tested and the co-occurrence graph of the email service assets, subgraph representation learning technology is used to learn the subgraph representation of the email to be tested and input it into the trained machine learning model, and the probability that the email to be tested is spam is output. If the probability is greater than a preset threshold, the email to be tested is determined to be spam.

[0032] Furthermore, the asset-related fields in the header information are used as the preset fields; the assets include domain names, IP addresses, and client identifiers.

[0033] Furthermore, the method for constructing the email service asset co-occurrence graph is:

[0034] 1) Extract graph nodes

[0035] Extract the values ​​in the From, Received, DKIM-Signature, Message-ID, and X-Mailer fields from the header information of each sample email as graph nodes, and merge the same graph nodes; the domain name in the From field is regarded as the first type of graph node and its type is marked as the sender domain; the domain names in the Received, DKIM-Signature, and Message-ID fields are regarded as the second type of graph nodes and their type is marked as the intermediate domain; the IP address in the Received field is regarded as the third type of graph node and its type is marked as the IP address; the client identifier in the X-Mailer field is regarded as the fourth type of graph node and its type is marked as the client identifier;

[0036] 2) Establish edge relationships between graph nodes

[0037] (1) If a graph node of type sender domain and a graph node of type intermediate domain appear in the same sample email, an edge is established between the two corresponding graph nodes;

[0038] (2) If a graph node of type intermediate domain and a graph node of type IP address appear in the same Received entry, an edge is established between the two corresponding graph nodes;

[0039] (3) If the graph node corresponding to the domain name in the Message-ID or DKIM-Signature is different from the graph node of the type "originating domain", an edge is established between the two corresponding graph nodes;

[0040] (4) If a graph node of type sender domain and a graph node of type client identifier appear in the same sample email, an edge is established between the two corresponding graph nodes;

[0041] Connect the nodes of each graph according to the edges obtained from rules (1) to (4) to obtain the co-occurrence subgraph corresponding to each sample email;

[0042] The two co-occurrence subgraphs with common graph nodes are merged into a connected graph to obtain the mail service asset co-occurrence graph.

[0043] Furthermore, the co-occurrence subgraph corresponding to each sample email is regarded as a subgraph in the email service asset co-occurrence graph to learn the embedding vector of the co-occurrence subgraph as the subgraph representation corresponding to the sample email.

[0044] Furthermore, the method for obtaining the subgraph representation of the email to be detected is: searching for a co-occurrence subgraph that has an intersection with the graph nodes in the co-occurrence subgraph corresponding to the email to be detected from the email service asset co-occurrence graph; then inputting each of the co-occurrence subgraphs found and the co-occurrence subgraph corresponding to the email to be detected into the subgraph embedding method to obtain the subgraph representation of the email to be detected.

[0045] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0046] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0047] The advantages of the present invention are as follows:

[0048] This paper constructs a co-occurrence graph of email service assets based on email headers, uses graph representation learning techniques to learn subgraph representations, and combines them with machine learning to design a robust spam detection algorithm. Using subgraph representation learning techniques for subgraph embedding on a public dataset, a machine learning classifier trained with this technique achieves a spam detection accuracy of 98.76%. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is the workflow diagram of the offline model training phase of this method.

[0050] Figure 2 This is the workflow diagram of the online spam detection stage of this method.

[0051] Figure 3 This is the subgraph constructed for the email in Example 1.

[0052] Figure 4 This is the subgraph constructed for the email in Example 2. DETAILED DESCRIPTION

[0053] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0054] Figure 1 The following is a workflow diagram for the offline model training phase of this method. This method constructs an asset co-occurrence graph based on legitimate / fraudulent datasets, uses subgraph representation learning techniques to represent the graph as a low-dimensional feature vector, and combines machine learning techniques to train a classifier offline. The main steps include:

[0055] (1) Obtaining annotated datasets

[0056] This method takes a labeled email dataset as input, which contains two types of emails:

[0057] 1) Spam: Unsolicited emails that users do not need;

[0058] 2) Legitimate emails: normal communication emails of users.

[0059] The labeled email dataset can be a public dataset on the Internet or collected and labeled by yourself. It is required that each email has complete header information.

[0060] (2) Construction of asset co-occurrence graph

[0061] This method defines three types of email service-related assets: domain name, IP address, and client identifier.

[0062] The construction of the asset co-occurrence graph consists of two steps:

[0063] 1) Extract nodes

[0064] This method extracts assets from the following fields of the email header information as graph nodes: From, Received, DKIM-Signature, Message-ID, and X-Mailer. It should be noted that the domain name in the From field is treated as a separate type of node: the sender domain, and all other domain names except the sender domain are collectively referred to as: intermediate domains. That is, the present invention treats the domain name in the From field as the first type of graph node and marks its type as the sender domain; treats the domain names in the Received, DKIM-Signature, and Message-ID fields as the second type of graph node and marks its type as the intermediate domain; treats the IP address in the Received field as the third type of graph node and marks its type as the IP address; and treats the client identifier in the X-Mailer field as the fourth type of graph node and marks its type as the client identifier. Since the Received fields of different email systems are different and may contain host identifiers, there may be overlaps between different Received entries. Therefore, when extracting the domain name in the Received field, the host identifier and the overlapping parts can be removed according to the preset rules and then the extracted domain names can be merged.

[0065] 2) Extract relations

[0066] After extracting the graph nodes, extract the edge relationships according to the following rules:

[0067] (1) If a graph node of type sender domain and a graph node of type intermediate domain appear in the same email, an edge is established between them.

[0068] (2) If a graph node of type intermediate domain and a graph node of type IP address appear in the same Received entry, an edge is established between the graph node corresponding to the intermediate domain and the graph node corresponding to the IP address.

[0069] (3) If the domain name in the Message-ID / DKIM-Signature of the email is different from the domain name in the sender domain of the email, then an edge is established between the graph node corresponding to the sender domain of the email and the graph node corresponding to the intermediate domain in the Message-ID / DKIM-Signature.

[0070] (4) If the email has an X-Mailer field, an edge is established between the graph node corresponding to the sender domain of the email and the graph node corresponding to the X-Mailer client identifier of the email.

[0071] (5) All the above edges are undirected, and the graph we constructed is an undirected heterogeneous graph.

[0072] According to the above method, each email is extracted as a subgraph, the same nodes are merged, and two subgraphs with common nodes are merged into a connected graph. All subgraphs merged together form a set of connected graphs.

[0073] (3) Subgraph representation learning

[0074] This method treats the co-occurrence subgraph corresponding to each email as a subgraph of the asset co-occurrence graph to learn the embedding vector of each co-occurrence subgraph. Using the existing subgraph embedding method, the feature representation of the graph is learned to preserve its structural properties. The input is each subgraph g in the total graph. i , the output is the low-dimensional vector d corresponding to each subgraph i .

[0075] (4) Machine Learning Model Training

[0076] During the machine learning model training phase, supervised machine learning techniques are used to classify subgraph vectors. The machine learning model can be a commonly used classifier including decision tree, random forest, SVM, and multi-layer perceptron.

[0077] The data consists of multiple (d i , Y i ) tuple, where d i is the low-dimensional vector representation of each subgraph, Y i is its class label. The training goal is to find the best classifier and hyperparameters and save it as a model.

[0078] Figure 2 The following is a workflow diagram of the online spam detection phase of this method, which mainly includes the following steps:

[0079] (1) Obtaining the email to be tested

[0080] This method can be applied to spam filtering of a receiving mail server or client. The mail to be detected comes from the receiving mail server or client, and it is necessary to obtain complete mail header field information of the mail to be detected.

[0081] (2) Extracting the email information to be tested

[0082] Extract the assets and relationships in the email headers to be tested and construct the asset co-occurrence subgraph. This extraction method is the same as the subgraph extraction method used in the offline model training phase.

[0083] (3) Learning subgraph representation

[0084] The asset co-occurrence subgraph corresponding to the email to be detected and the set of subgraphs in the original asset co-occurrence graph that intersect with the nodes of the asset co-occurrence subgraph are taken as the input of the subgraph embedding method to obtain a low-dimensional vector representation of the subgraph corresponding to the email to be detected.

[0085] (4) Spam Classification

[0086] The low-dimensional vector representation of the subgraph corresponding to the email to be detected is input into the machine learning model trained in stage 1, and the probability of the email being spam is output. If the probability is greater than the preset threshold, the email is determined to be spam.

[0087] In order to make the above features and advantages of the present invention more clearly understood, examples are given below for illustration.

[0088] Example 1

[0089] We used the Nazario phishing dataset (https: / / monkey.org / jose / phishing) and the DataCon sender forgery dataset (https: / / datacon.qianxin.com / opendata / openpage?resourcesId=2) as annotation datasets for graph construction. Both datasets contain complete email header information, meeting the requirements. Forged emails from the Nazario phishing and DataCon datasets are labeled as spam and given a label of 1; legitimate emails from the DataCon dataset are labeled as authentic and given a label of 0.

[0090] Extract the header assets and relationship information of each email and build a co-occurrence graph of email service assets. Take the following email with the From field being shipment@fedex.com as an example:

[0091]

[0092]

[0093] First, extract asset information from the From, Received, Message-ID, DKIM-Signature, and X-Mailer fields in the email header, including:

[0094] 1) Received field: A rule is pre-defined: only the host name after the from clause is used, and the host name is stripped of the lowest level prefix as the domain name. Therefore, the intermediate domains are extracted: outbound-mail.sendgrid.net, fedex.com; and the IP address is 167.89.100.244.

[0095] 2) DKIM-Signature field: the domain name (middle domain) following “d=”: sendgrid.net.

[0096] 3) From field: The sending domain in the email address: fedex.com.

[0097] 4) Message-ID field: The middle domain after the “@” symbol: fedex.com.

[0098] 5) X-Mailer field: This email does not have an X-Mailer field, so skip it.

[0099] Then extract the relationship information between assets:

[0100] 1) If the sending domain and the intermediate domain appear in the same email, create an edge between them. Create edges between the sending domain fedex.com and the intermediate domains fedex.com, outbound-mail.sendgrid.net, and sendgrid.net, respectively.

[0101] 2) If the intermediate domain and IP address appear in the same Received entry, an edge is created between them. Here, an edge is created between outbound-mail.sendgrid.net and 167.89.100.244.

[0102] 3) If the domain name in the email's Message-ID / DKIM-Signature is different from the sender's domain, an edge is established between the sender's domain and the domain name in the Message-ID / DKIM-Signature. In this case, the domain name in the Message-ID is the same as the sender's domain, but the domain name in the DKIM-Signature is different. Therefore, an edge is established only between the sender's domain (fedex.com) and the domain name in the DKIM-Signature (sendgrid.net).

[0103] 4) All the above edges are undirected, and the graph we constructed is an undirected heterogeneous graph.

[0104] The subgraph constructed for this email is as follows Figure 3 shown.

[0105] Similarly, for each email in the dataset, a subgraph is constructed. Nodes with the same type and name are merged to form the overall asset co-occurrence graph. Each node in the graph is indexed, and each edge is represented by two node indices. The subgraph corresponding to each email is represented as a collection of edges, and all subgraphs are stored in a subgraph library.

[0106] Use the Sub2Vec subgraph embedding model to learn the subgraph structural properties, set the dimension to 128, and represent each subgraph as a 128-dimensional vector.

[0107] Embed all email subgraphs into vectors and email labels, split the training and test sets into a 7:3 ratio, and train them into random forest, support vector machine, and logistic regression models, respectively. The support vector machine model performed best, achieving an F1 score of 97.89%. Save the corresponding SVM model.

[0108] Example 2

[0109] Deploy the sub-image library and the trained model to the receiving email server. When the email server receives a new email, it immediately extracts its header information to build an asset co-occurrence map. For example, the email header information is as follows:

[0110]

[0111]

[0112] The construction method is the same as that in Example 1. The constructed subgraph is as follows: Figure 4 As shown in the figure, a connected graph that intersects with the subgraph of the test email is extracted from the subgraph library. This subgraph set is fed into the Sub2Vec model to learn its structural properties. 128-dimensional low-dimensional vector representations of these subgraphs are obtained. The subgraph representation corresponding to the test email is then fed into the trained support vector machine model. The predicted probability of this email being spam is 0.16, which is less than the preset threshold of 0.5. Therefore, the email is deemed legitimate.

[0113] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A spam detection method based on an email service asset co-occurrence graph, comprising the following steps: Offline training phase of machine learning model: Get the header information of each sample email in the sample set; The sample set includes a number of legitimate emails and a number of spam emails; Extracting preset fields from the header information of each sample email, constructing a co-occurrence subgraph corresponding to each sample email, and then merging the obtained co-occurrence subgraphs to obtain an email service asset co-occurrence graph; Based on the co-occurrence subgraph of the sample emails and the email service asset co-occurrence graph, using subgraph representation learning technology, learn the subgraph representation of the corresponding sample emails and label them; Train machine learning models using labeled subgraph representations; Online detection stage: Get the header information of the email to be tested; Extracting preset field information from the header information of the email to be detected and constructing an asset co-occurrence subgraph of the email to be detected; Based on the co-occurrence subgraph of the email to be detected and the email service asset co-occurrence graph, using subgraph representation learning technology, learn the subgraph representation of the email to be detected and input it into the trained machine learning model, output the probability that the email to be detected is spam, and if the probability is greater than a preset threshold, determine that the email to be detected is spam; The method for constructing the email service asset co-occurrence graph is as follows: 1) Extract graph nodes Extract the values ​​in the From, Received, DKIM-Signature, Message-ID, and X-Mailer fields from the header information of each sample email as graph nodes, and merge the same graph nodes; the domain name in the From field is regarded as the first type of graph node and its type is marked as the sender domain; the domain names in the Received, DKIM-Signature, and Message-ID fields are regarded as the second type of graph nodes and their type is marked as the intermediate domain; the IP address in the Received field is regarded as the third type of graph node and its type is marked as the IP address; the client identifier in the X-Mailer field is regarded as the fourth type of graph node and its type is marked as the client identifier; 2) Establish edge relationships between graph nodes (1) If a graph node of type sender domain and a graph node of type intermediate domain appear in the same sample email, an edge is established between the two corresponding graph nodes; (2) If a graph node of type intermediate domain and a graph node of type IP address appear in the same Received entry, an edge is established between the two corresponding graph nodes; (3) If the graph node corresponding to the domain name in the Message-ID or DKIM-Signature is different from the graph node of the type "originating domain", an edge is established between the two corresponding graph nodes; (4) If a graph node of type sender domain and a graph node of type client identifier appear in the same sample email, an edge is established between the two corresponding graph nodes; Connect the graph nodes according to the edges obtained from rules (1) to (4) to obtain the co-occurrence subgraph corresponding to each sample email; merge the two co-occurrence subgraphs with common graph nodes into a connected graph to obtain the email service asset co-occurrence graph.

2. The method according to claim 1, characterized in that The asset-related fields in the header information are used as the preset fields; the assets include domain names, IP addresses, and client identifiers.

3. The method according to claim 1, characterized in that The co-occurrence subgraph corresponding to each sample email is regarded as a subgraph in the co-occurrence graph of the email service assets to learn the embedding vector of the co-occurrence subgraph as the subgraph representation corresponding to the sample email.

4. The method according to claim 1, wherein The method for obtaining the subgraph representation of the email to be detected is as follows: searching for co-occurrence subgraphs that have intersections with graph nodes in the co-occurrence subgraph corresponding to the email to be detected from the email service asset co-occurrence graph; then inputting each of the found co-occurrence subgraphs and the co-occurrence subgraph corresponding to the email to be detected into a subgraph embedding method to obtain the subgraph representation of the email to be detected.

5. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images