Phishing email detection method and phishing email detection model training method

Abstract

The embodiment of the application provides a phishing email detection method and a phishing email detection model training method. The method comprises the following steps: obtaining a to-be-detected email text; extracting a preset number of effective sentences from the to-be-detected email text to obtain effective text content; inputting the effective text content into a phishing email detection model to determine whether the to-be-detected email text is a phishing email. The scheme no longer detects phishing emails based on a general word library or an expert word library, thereby improving the accuracy and detection efficiency of phishing email detection.

Description

Technical Field

[0001] This application relates to the field of computer network security technology, and in particular to a method for detecting phishing emails, a method for training a phishing email detection model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Phishing emails are forged emails designed to trick recipients into providing account information, passwords, or other sensitive data to a designated recipient. They may also contain hyperlinks that redirect recipients to specially designed phishing websites, prompting them to enter their usernames, passwords, and other private information. Current methods for detecting phishing emails involve extracting features from the email text, subject line, and hyperlinks. These features include specific words such as dangerous or suspicious terms in the text, the number of hyperlinks, whether they redirect, and specific characters contained in the links. These features are then used to learn the probability that the email is a phishing email. If the probability exceeds a certain threshold, the email is identified as a phishing email. This algorithm is effective in detecting phishing emails.

[0003] Current phishing email detection methods mostly rely on general terminology databases or manually compiled sensitive word databases for phishing email detection. These methods have the following drawbacks: (1) The terminology databases are fixed, and simply considering the sensitive words in the database lacks contextual relevance, resulting in a lack of generalization when facing a wide variety of phishing emails. (2) The sensitive word databases are entirely constructed by business experts, which is a complex process and lacks data objectivity. (3) General terminology databases lack contextual relevance, making them less effective for phishing email detection. Summary of the Invention

[0004] This application provides a method for detecting phishing emails, which improves the accuracy and efficiency of phishing email detection.

[0005] This application provides a method for detecting phishing emails, including:

[0006] Get the text of the email to be tested;

[0007] A preset number of valid sentences are extracted from the email text to be detected to obtain valid text content;

[0008] The valid text content is input into the phishing email detection model to determine whether the email text to be detected is a phishing email. The phishing email detection model is obtained by training the parameters of a preset network model based on the classification results of the valid sample content and whether the valid sample content is a phishing email. The valid sample content is obtained by extracting a preset number of valid sentences from multiple sample emails that are known to be phishing emails.

[0009] In one embodiment, extracting a preset number of valid sentences from the email text to be detected to obtain valid text content includes:

[0010] The email text to be detected is preprocessed to obtain a list of words corresponding to the target sentence in the email text to be detected.

[0011] The number of valid characters contained in the target sentence is determined based on the serial numbers corresponding to different words in the dictionary and the word list corresponding to the target sentence in the email text to be detected.

[0012] Target sentences with a number of valid characters greater than a preset value are identified as valid sentences, and a preset number of valid sentences are extracted sequentially from the email text to be detected to obtain the valid text content.

[0013] In one embodiment, the preprocessing of the email text to be detected to obtain a vocabulary list corresponding to the target sentence in the email text includes:

[0014] The email text to be detected is segmented into sentences to obtain the target sentence;

[0015] Compare the number of Chinese words in the target sentence with the number of English words;

[0016] If the number of Chinese words is greater than the number of English words, only the Chinese words are retained to obtain a list of Chinese words corresponding to the target sentence; if the number of Chinese words is less than the number of English words, only the English words are retained to obtain a list of English words corresponding to the target sentence.

[0017] In one embodiment, before segmenting the email text to be detected into sentences to obtain the target sentence, the method further includes:

[0018] The email subject of the email to be detected is decoded, the email body is converted to Traditional Chinese characters, and escape characters are removed.

[0019] In one embodiment, before determining the number of valid characters contained in the target sentence based on the sequence numbers corresponding to different words in the thesaurus and the word list corresponding to the target sentence in the email text to be detected, the method further includes:

[0020] Obtain a sample email set containing multiple sample emails, the sample email set including phishing emails and non-phishing emails;

[0021] The word library is formed by extracting the words with the highest word frequency-inverse document frequency values ​​from each sample email. The word library includes multiple words and the corresponding serial number for each word.

[0022] In one embodiment, before extracting the words with the highest term frequency-inverse document frequency values ​​from each sample email to form the lexicon, the method further includes:

[0023] For each sample email, the body text is extracted from the sample email and then segmented into multiple sample sentences;

[0024] For each sample sentence, the sample sentence is segmented to obtain the vocabulary contained in the sample email.

[0025] In one embodiment, the step of performing word segmentation on each sample sentence to obtain the vocabulary contained in the sample email includes:

[0026] For each sample sentence in the sample email, identify whether the sample sentence is a Chinese sentence;

[0027] If the sample sentence is a Chinese sentence, perform traditional Chinese character replacement and word segmentation on the Chinese sentence to obtain the vocabulary contained in the Chinese sentence;

[0028] If the sample sentence is an English sentence, the English sentence is segmented, and abnormal English words are segmented, while the correct words are retained, to obtain the vocabulary contained in the English sentence.

[0029] In one embodiment, the step of inputting the valid text content into a phishing email detection model to determine whether the email text to be detected is a phishing email includes:

[0030] Input the valid text content into the phishing email detection model to obtain the confidence score output by the phishing email detection model;

[0031] If the confidence level is greater than or equal to the threshold, the email text to be detected is determined to be a phishing email;

[0032] If the confidence level is less than the threshold, the email text to be detected is determined to be a non-phishing email.

[0033] In one embodiment, before inputting the valid text content into the phishing email detection model to determine whether the email text to be detected is a phishing email, the method further includes:

[0034] Retrieve multiple sample emails known to be phishing emails;

[0035] A preset number of valid sentences are extracted from the sample emails to obtain valid sample content;

[0036] Based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, the parameters of the preset network model are trained to obtain a phishing email detection model.

[0037] In one embodiment, the preset network model is a Bi-LSTM+Attention network model, which includes: an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer connected in sequence.

[0038] In one embodiment, the step of training the parameters of a Bi-LSTM+Attention network model based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, and obtaining a phishing email detection model trained by the Bi-LSTM+Attention network model, includes:

[0039] The overall features of the effective sample content are extracted by sequentially connecting the embedding layer, the word long short-term memory layer, the word attention layer, the sentence long short-term memory layer, and the sentence attention layer.

[0040] The overall features are used as input to the classification function layer to obtain the prediction result output by the classification function layer. Based on the prediction result and the known classification result, the parameters of the Bi-LSTM+Attention network model are optimized until the training objective is met, thus obtaining a phishing email detection model trained by the Bi-LSTM+Attention network model.

[0041] In one embodiment, the step of extracting the overall features of the effective sample content through a sequentially connected embedding layer, word long short-term memory layer, word attention layer, sentence long short-term memory layer, and sentence attention layer includes:

[0042] The embedding layer maps the words contained in the effective sample content into a first vector;

[0043] The first vector corresponding to each word in the effective sample content is used as input, and the word features of each word are extracted through the long short-term memory layer of the word.

[0044] The word features of each word in the effective sample content are used as input, and the sentence features of each effective sentence in the effective sample content are obtained through the word attention layer.

[0045] Based on the sentence features of each valid sentence in the valid sample content, the sentence context features of each valid sentence in the valid sample content are obtained through the sentence long short-term memory layer;

[0046] Based on the sentence context features of each valid sentence in the valid sample content, the overall features of the valid sample content are obtained through a sentence attention layer.

[0047] This application also provides a method for training a phishing email detection model, including:

[0048] Retrieve multiple sample emails known to be phishing emails;

[0049] A preset number of valid sentences are extracted from the sample emails to obtain valid sample content;

[0050] Based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, the parameters of the preset network model are trained to obtain a phishing email detection model.

[0051] In one embodiment, the preset network model is a Bi-LSTM+Attention network model, which includes: an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer connected in sequence.

[0052] In one embodiment, the step of training the parameters of a preset network model based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, to obtain a phishing email detection model, includes:

[0053] The overall features of the effective sample content are extracted by sequentially connecting the embedding layer, the word long short-term memory layer, the word attention layer, the sentence long short-term memory layer, and the sentence attention layer.

[0054] The overall features are used as input to the classification function layer to obtain the prediction result output by the classification function layer. Based on the prediction result and the known classification result, the parameters of the Bi-LSTM+Attention network model are optimized until the training objective is met, thus obtaining a phishing email detection model trained by the Bi-LSTM+Attention network model.

[0055] In one embodiment, the step of extracting the overall features of the effective sample content through a sequentially connected embedding layer, word long short-term memory layer, word attention layer, sentence long short-term memory layer, and sentence attention layer includes:

[0056] The embedding layer maps the words contained in the effective sample content into a first vector;

[0057] The first vector corresponding to each word in the effective sample content is used as input, and the word features of each word are extracted through the long short-term memory layer of the word.

[0058] The word features of each word in the effective sample content are used as input, and the sentence features of each effective sentence in the effective sample content are obtained through the word attention layer.

[0059] Based on the sentence features of each valid sentence in the valid sample content, the sentence context features of each valid sentence in the valid sample content are obtained through the sentence long short-term memory layer;

[0060] Based on the sentence context features of each valid sentence in the valid sample content, the overall features of the valid sample content are obtained through a sentence attention layer.

[0061] This application also provides an electronic device, the electronic device comprising:

[0062] processor;

[0063] Memory used to store processor-executable instructions;

[0064] The processor is configured to execute the above-described phishing email detection method or the above-described phishing email detection model training method.

[0065] This application also provides a computer-readable storage medium storing a computer program that can be executed by a processor to perform the above-described phishing email detection method or the above-described phishing email detection model training method.

[0066] The technical solution provided in the above embodiments of this application obtains the email text to be detected; extracts a preset number of valid sentences from the email text to obtain valid text content; inputs the valid text content into a phishing email detection model to determine whether the email text to be detected is a phishing email. Since it no longer detects phishing emails based on general dictionaries or expert dictionaries, it improves the accuracy and efficiency of phishing email detection. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below.

[0068] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0069] Figure 2 This is a flowchart illustrating a phishing email detection method provided in an embodiment of this application;

[0070] Figure 3 yes Figure 2 Detailed flowchart of step S220 in the corresponding embodiment;

[0071] Figure 4 yes Figure 3 Detailed flowchart of step S310 in the corresponding embodiment;

[0072] Figure 5 This is a flowchart illustrating the lexicon construction process provided in the embodiments of this application;

[0073] Figure 6 yes Figure 5 A flowchart illustrating the lexicon construction process based on the corresponding embodiment;

[0074] Figure 7 yes Figure 6 Detailed flowchart of step S620 in the corresponding embodiment;

[0075] Figure 8 This is a flowchart illustrating the training method of the phishing email detection model provided in the embodiments of this application;

[0076] Figure 9 This is a schematic diagram of the architecture of the Bi-LSTM+Attention network model provided in the embodiments of this application;

[0077] Figure 10 yes Figure 8 Detailed flowchart of step S830 in the corresponding embodiment;

[0078] Figure 11 This is a block diagram illustrating a phishing email detection device according to an embodiment of this application;

[0079] Figure 12 This is a block diagram illustrating a training apparatus for a phishing email detection model according to an embodiment of this application. Detailed Implementation

[0080] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0082] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 can be used to execute the phishing email detection method and the phishing email detection model training method provided in the embodiments of this application. Figure 1As shown, the electronic device 100 includes: one or more processors 102 and one or more memories 104 storing processor-executable instructions. The processors 102 are configured to execute the phishing email detection method and the phishing email detection model training method provided in the following embodiments of this application.

[0083] The processor 102 may be a gateway, a smart terminal, or a device that includes a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data from other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.

[0084] The memory 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the phishing email detection method and the phishing email detection model training method described below. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0085] In one embodiment, Figure 1 The electronic device 100 may also include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device 100 may also have other components and structures as needed.

[0086] The input device 106 may be a device used by a user to input commands, and may include one or more of a keyboard, mouse, microphone, and touchscreen. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. The data acquisition device 110 may acquire images of objects and store the acquired images in the memory 104 for use by other components. For example, the data acquisition device 110 may be a camera.

[0087] In one embodiment, the components in the example electronic device 100 used to implement the phishing email detection method and the phishing email detection model training method of the present application embodiments can be integrated or distributed. For example, the processor 102, memory 104, input device 106 and output device 108 can be integrated into one unit, while the data acquisition device 110 can be separated.

[0088] In one embodiment, the example electronic device 100 used to implement the phishing email detection method and the phishing email detection model training method of the embodiments of this application can be implemented as a smart terminal such as a smartphone, tablet computer, desktop computer, server, vehicle equipment, etc.

[0089] Figure 2 This is a flowchart illustrating a phishing email detection method provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps S210-S230.

[0090] Step S210: Obtain the text of the email to be tested.

[0091] Among them, the email text to be detected refers to the email text whose phishing email status is unknown.

[0092] Step S220: Extract a preset number of valid sentences from the email text to be detected to obtain valid text content.

[0093] In this context, a valid sentence is defined as a sentence with a valid vocabulary count greater than a preset value. Vocabulary located in the lexicon can be considered valid vocabulary. In one embodiment, the valid vocabulary count can be obtained by counting how many words in each sentence appear in the lexicon. If the valid vocabulary count is large...

[0094] If the value is within the preset range, it indicates that the sentence is a valid sentence. Valid text content consists of a preset number of valid sentences. In one embodiment, it can start from the first sentence of the email text to be detected and sequentially extract sentences.

[0095] Select a preset number of valid sentences to form valid text content.

[0096] In one embodiment, such as Figure 3 As shown, the above step S220 specifically includes: step S310-step S330.

[0097] Step S310: Preprocess the text of the email to be detected to obtain a list of words corresponding to the target sentence in the text of the email to be detected.

[0098] Here, the target sentence refers to a sentence in the email text to be detected. The vocabulary list refers to a list of words contained in the target sentence. In one embodiment, the preprocessing process may include sentence segmentation and word segmentation.

[0099] In one embodiment, such as Figure 4 As shown, step S310 may specifically include: steps S311-5 and step S313.

[0100] Step S311: Segment the email text to be detected to obtain the target sentence.

[0101] Specifically, the email file to be tested can be segmented into target sentences based on sentence delimiters, such as periods, question marks, and exclamation marks. The email subject is the first sentence by default, and the email body can be segmented based on delimiters and appended to the sentence after the email subject.

[0102] 0 Wherein, prior to step S311 above, the method provided in this application embodiment may further include:

[0103] The email subject of the email to be detected is decoded, the email body is converted to Traditional Chinese characters, and escape characters are removed.

[0104] Decoding refers to the process of decrypting emails. Since most email subjects are transmitted in encrypted form, the email subject can be decoded to obtain more information about the subject.

[0105] Traditional Chinese character conversion refers to replacing traditional Chinese characters in the email body with simplified Chinese characters from the "Traditional Chinese-Simplified Chinese Character Mapping Dictionary". Escape characters, also known as escape symbols, are characters used to represent all ASCII codes using a backslash ("\") followed by a number (usually octal). In C, some letters are preceded by a backslash ("\") to represent common, non-printable ASCII characters, such as \0, \t, and \n. These are called escape characters because the following characters no longer represent their original ASCII meaning.

[0106] Step S312: Compare the number of Chinese words in the target sentence with the number of English words.

[0107] Step S313: If the number of Chinese words is greater than the number of English words, only the Chinese words are retained to obtain a list of Chinese words corresponding to the target sentence; if the number of Chinese words is less than the number of English words, only the English words are retained to obtain a list of English words corresponding to the target sentence.

[0108] Specifically, starting with the first target sentence, we can compare the number of Chinese words in each target sentence with the number of English words. If a sentence has more Chinese words than English words, then only the Chinese words are retained, resulting in a list of Chinese words. Conversely, if a sentence has fewer Chinese words than English words, then only the English words are retained, resulting in a list of English words. Assuming the number of Chinese words equals the number of English words, you can choose to retain only Chinese or only English words as needed.

[0109] Step S320: Determine the number of valid characters contained in the target sentence based on the serial numbers corresponding to different words in the thesaurus and the word list corresponding to the target sentence in the email text to be detected.

[0110] The thesaurus stores words and their corresponding serial numbers. The thesaurus can be constructed using the method described in the examples below, or it can be an existing thesaurus constructed using other methods. The word list can be the Chinese word list or the English word list mentioned above.

[0111] Specifically, for the Chinese vocabulary list, we can sequentially check whether each word in the list appears in the dictionary. If it appears, we assign its index in the dictionary to that word. If it doesn't exist, we can use "full mode" to further subdivide the Chinese words that don't exist in the dictionary, and then use the subdivided words to continue mapping. If they exist, we assign the dictionary mapping index to the corresponding position in the list; otherwise, we assign a value of 0.

[0112] For a list of English words, we can sequentially check whether each word in the list appears in the dictionary. If it does, we assign its index in the dictionary to that word. If it does not exist, we assign the value 0 to the corresponding position in the list.

[0113] Specifically, the number of valid characters in the target sentence can be the number of non-zero ordinal numbers, which represents how many words from the lexicon the target sentence contains.

[0114] Step S330: Target sentences with a number of valid characters greater than a preset value are identified as valid sentences, and a preset number of valid sentences are extracted sequentially from the email text to be detected to obtain the valid text content.

[0115] Specifically, starting with the first target sentence, the number of valid characters in each target sentence is checked sequentially. If the number of valid characters is greater than a preset value, the number of valid sentences is incremented by 1, and the process continues to read the next target sentence. Otherwise, the process continues to read the next target sentence until the preset number of valid sentences is obtained, at which point the process ends. Otherwise, the process continues to read the next target sentence until no further target sentences are available. The preset number of valid sentences constitutes the valid text content.

[0116] It should be noted that if the email text to be tested is too long, the above method can extract the valid text content, improving data processing efficiency. If the email text to be tested is not long enough, zeros can be padded to it.

[0117] Step S230: Input the effective text content into the phishing email detection model to determine whether the email text to be detected is a phishing email. The phishing email detection model is obtained by training the parameters of a preset network model based on the classification results of the effective sample content and whether the effective sample content is a phishing email. The effective sample content is obtained by extracting a preset number of effective sentences from multiple sample emails that are known to be phishing emails.

[0118] It should be noted that the phishing email detection model can be constructed using the method described in the embodiments below, or it can be an existing model constructed using other methods. Specifically, valid text content can be input into the phishing email detection model to obtain the confidence score output by the model. The confidence score is used to characterize the probability that the email text to be detected is a phishing email. If the confidence score is greater than or equal to a threshold, the email text to be detected is determined to be a phishing email; if the confidence score is less than the threshold, the email text to be detected is determined to be a non-phishing email.

[0119] Figure 5 This is a flowchart illustrating the lexicon construction process provided in an embodiment of this application. For example... Figure 5 As shown, the lexicon construction process includes the following steps S510-S520.

[0120] Step S510: Obtain a sample email set containing multiple sample emails, the sample email set including phishing emails and non-phishing emails.

[0121] Phishing emails are spoofed emails used to trick users into revealing their account information, passwords, etc. Non-phishing emails are legitimate emails. A sample email collection can include multiple phishing emails and multiple non-phishing emails; these are collectively referred to as sample emails.

[0122] In one embodiment, prior to step S520 below, as follows Figure 6 As shown, the method provided in this application embodiment may further include: step S610-step S620.

[0123] Step S610: For each sample email, extract the body portion from the sample email and segment the body portion into multiple sample sentences.

[0124] Specifically, the sample emails can be parsed using the MIME standard (Multipurpose Internet Mail Extensions) to obtain the body portions of both phishing and non-phishing emails, removing escape characters from the body portions. Then, the text is segmented based on sentence delimiters, breaking the sample email body portion into individual sentences for differentiation; these are called sample sentences.

[0125] Step S620: For each sample sentence, perform word segmentation on the sample sentence to obtain the vocabulary contained in the sample email.

[0126] The vocabulary included in the sample emails can be composed of the vocabulary contained in all sample sentences. By performing word segmentation on each sample sentence, the vocabulary contained in each sample sentence can be obtained, and thus the vocabulary contained in the sample emails can be obtained.

[0127] In one embodiment, such as Figure 7 As shown, step S620 may include steps S621-S622'.

[0128] Step S621: For each sample sentence in the sample email, identify whether the sample sentence is a Chinese sentence.

[0129] In one embodiment, a sample sentence can be distinguished as either a Chinese sentence or an English sentence based on the range of its character encoding. If the encoding falls within a certain range, it is considered a Chinese sentence; otherwise, it is considered an English sentence.

[0130] Step S622: If the sample sentence is a Chinese sentence, perform traditional Chinese character replacement and word segmentation on the Chinese sentence to obtain the vocabulary contained in the Chinese sentence.

[0131] This process involves replacing traditional Chinese characters in sentences with simplified Chinese characters based on a "Traditional Chinese to Simplified Chinese character mapping dictionary." Then, a "precise mode" is used for word segmentation of the Chinese sentences. This "precise mode" precisely segments the text, eliminating redundant words and repetitive word combinations, thus obtaining the vocabulary contained in each Chinese sentence.

[0132] Step S622': If the sample sentence is an English sentence, perform word segmentation on the English sentence, segment abnormal English words, retain correct words, and obtain the vocabulary contained in the English sentence.

[0133] Specifically, natural language processing toolkits can be used to segment each English sentence. The segmented English words contain many abnormal words, which are referred to here as abnormal English words, such as English words composed of multiple words or English words containing ".".

[0134] Abnormal English words can be further segmented to extract more keywords. There are two segmentation methods: a) Processing abnormal English words containing ".": Separate abnormal English words according to ".", and then retain the valid words as the correct words. b) Processing words composed of multiple words: Use a sliding window to extract buffer words as the correct words.

[0135] Furthermore, the correctness of English words can also be determined. a) Add specific words, such as "dhl" and "boe," which are considered incorrect in conventional dictionaries but are considered correct in this embodiment and added to the dictionary of this embodiment, becoming important features in phishing text detection. b) Delete English names. Phishing emails contain a large number of names. In the word segmentation step of the previous step, many English names are segmented into multiple correct words. However, these words will affect the detection of phishing emails and reduce the proportion of blacklisted words. Therefore, in this embodiment, English names are considered incorrect and cannot be added to the dictionary of this embodiment; they must be deleted. In other words, in this embodiment, some specific words such as "dhl" and "boe" can be considered correct words and retained, while English names are considered incorrect words and deleted. The correct words ultimately retained in the English sentence are the words contained in the English sentence.

[0136] Step S520: Extract the words with the highest word frequency-inverse document frequency values ​​from each sample email to form the vocabulary, which includes multiple words and the corresponding serial number for each word.

[0137] Term Frequency-Inverse Document Frequency (TF-IDF) is used to evaluate the importance of a word to a specific sample email in a dataset. If a word has a high TF frequency in one sample email but rarely appears in others, it is considered to have good class-discriminating ability and is suitable for classification. TF-IDF = TF * IDF, where TF is Term Frequency and IDF is Inverse Document Frequency. TF represents the frequency of a word in the sample emails. The main idea behind IDF is that the fewer sample emails containing a word, the more important the word is, indicating its class-discriminating ability.

[0138] Specifically, from each sample email, a vocabulary corpus can be constructed by extracting the highest TF-IDF values ​​for both Chinese and English words. The corpus can include the extracted words and their serial numbers. These serial numbers can be obtained by sequentially numbering the words in the corpus.

[0139] Figure 8 This is a flowchart illustrating the training method of the phishing email detection model provided in this application embodiment. Figure 8 As shown, the method includes the following steps S810-S830.

[0140] Step S810: Obtain multiple sample emails that are known to be phishing emails.

[0141] The sample emails used to train the phishing email detection model and the sample emails used to build the vocabulary can come from the same set of sample emails or different sets of sample emails. It is known whether a sample email is a phishing email; that is, the classification result of the sample email is known.

[0142] Step S820: Extract a preset number of valid sentences from the sample emails to obtain valid sample content.

[0143] To distinguish it from the valid text content in step S220, a predetermined number of valid sentences extracted from the sample emails constitute the text, which can be referred to as valid sample content. The extraction method for valid sample content is the same as that for valid text content, as described in step S220 above.

[0144] Step S830: Based on the content of the valid samples and the classification result of whether the content of the valid samples is a phishing email, train the parameters of the preset network model to obtain the phishing email detection model.

[0145] The preset network model can be a Bi-LSTM+Attention (bidirectional long short-term memory network + self-attention mechanism) network model.

[0146] To distinguish them, the network model before training can be called the Bi-LSTM+Attention network model, and after the parameters of the Bi-LSTM+Attention network model are optimized, it can be called the phishing email detection model.

[0147] Specifically, the effective sample content is used as the input to the Bi-LSTM+Attention network model, and the parameters of the Bi-LSTM+Attention network model are adjusted to minimize the error between the prediction result output by the Bi-LSTM+Attention network model and the known classification result of the sample email.

[0148] In one embodiment, such as Figure 9As shown, the Bi-LSTM+Attention network model includes: an embedding layer, a word LSTM layer, a word attention layer, a sentence LSTM layer, a sentence attention layer, and a softmax layer, all connected sequentially. After determining the network model architecture, appropriate batch size (input size in one training iteration), word embedding length, number of training epochs, and the number of nodes in the LSTM hidden layer are selected.

[0149] like Figure 10 As shown, the above step S830 specifically includes: step S831-step S832.

[0150] Step S831: Extract the overall features of the effective sample content through the sequentially connected embedding layer, word long short-term memory layer, word attention layer, sentence long short-term memory layer, and sentence attention layer.

[0151] The overall features are used to characterize the features of the sample emails and can be represented in the form of feature vectors. Specifically, the forward propagation training of the effective sample content first maps the words contained in the effective sample content into a first vector through the embedding layer (that is, mapping each word to a low-dimensional vector). Using the first vector corresponding to each word in the effective sample content as input, the word features of each word are extracted through the Word LSTM Layer. The Word LSTM Layer can extract the contextual relationships in a sentence. For sequence modeling, future information and historical information at each time step are equally important. The standard LSTM model, following the sequence, cannot capture future information; therefore, a bidirectional LSTM model is used to obtain equal contextual relationships.

[0152] The lexical features of each word in the valid sample content are used as input, and the sentence features of each valid sentence in the valid sample content are obtained through the Word Attention layer. The Word Attention layer can generate a weight vector, merge the lexical features at each position into a sentence feature vector, and multiply and sum the weight vectors to obtain the sentence features.

[0153] Subsequently, based on the sentence features of each valid sentence in the valid sample content, the sentence context features of each valid sentence in the valid sample content are obtained through the Sentence LSTM Layer. Based on the sentence context features of each valid sentence in the valid sample content, the overall features of the valid sample content are obtained through the Sentence Attention Layer. The Sentence LSTM Layer can extract the relationship between sentences, and the Sentence Attention Layer generates another weight vector, multiplying the sentence context features at each position with the weights and then summing the results (i.e., weighted summation) to form the overall features of a single valid sample content.

[0154] Step S832: Use the overall features as input to the classification function layer to obtain the prediction result output by the classification function layer. Based on the prediction result and the known classification result, optimize the parameters of the Bi-LSTM+Attention network model until the training objective is met, and obtain the phishing email detection model trained by the Bi-LSTM+Attention network model.

[0155] The prediction result refers to the probability value of whether the sample email output by the Bi-LSTM+Attention network model is a phishing email, which can also be called the training output. The Categorical cross-entropy loss value is calculated based on the training output and classification results. The Rmsprop optimizer is used to optimize the hyperparameter values ​​of the entire Bi-LSTM+Attention network model based on the loss value, thereby reducing the network's loss value.

[0156] The training objective could be that the number of training iterations equals the number of epochs, or that the loss value remains relatively constant across multiple training iterations. If the training objective is not achieved, the training continues iteratively to optimize the hyperparameters of the entire Bi-LSTM+Attention network model. Once training is complete, a phishing email detection model is obtained, which can be used for... Figure 2 In a corresponding embodiment, phishing emails are detected.

[0157] The technical solution provided in the above embodiments of this application utilizes a large number of sample emails, combines deep learning and autonomous learning to construct a lexicon, and the phishing email detection model based on the self-attention mechanism can use the association of word context to assign different weights to words and sentences, thereby determining whether it is a phishing email, improving the accuracy and efficiency of email detection.

[0158] Due to the wide variety of phishing emails, the technical solution provided by the above embodiments of this application can detect Japanese or English using traditional Chinese characters through traditional Chinese character replacement processing. For example, in the email text, "安全認證" is replaced with "安全认证", and "驗證" is replaced with "验证". Through decoding technology, emails with encoded and encrypted subjects can be detected. For example, the email subject "?UTF-8?B?UmU6IEF0dGVudGlvbiBNYWlsIFVzZXI gIsKg?" is decoded to "Re:Attention Mail User"\xc2\xa0. Through splitting English words, many words spliced together can be processed, and correct words can be extracted from abnormal words. For example, "messagecenter" is split into u'message' and u'center', and "globalsources" is split into u'source' and u'global', adding more keywords.

[0159] The technical solution provided by the above embodiments of this application uses Tf-Idf to build a word library. For each email sample, the words with the highest Tf-Idf scores in each email are extracted using Tf-Idf to build a word library. After removing duplicates from the words, a word library is obtained. Compared with expert screening, Tf-Idf can well extract the feature words of each text, enabling better classification of emails.

[0160] Given the particularity of phishing emails, there are a large number of sensitive words and keywords in the subject. The subject of phishing emails often contains some fascinating or warning information, which is different from legitimate emails. By adding the email subject and using the Attention mechanism, different weights are autonomously assigned to the email subject and the email body, enabling the model to focus on processing more diverse and useful information from the email subject.

[0161] The following is an embodiment of the device of this application, which can be used to execute the above embodiments of the phishing email detection method of this application. For details not disclosed in the embodiment of the device of this application, please refer to the embodiment of the phishing email detection method of this application.

[0162] Figure 11 It is a block diagram of a phishing email detection device shown in an embodiment of this application. As Figure 11 shown, the device includes:

[0163] An email acquisition module 1101, which can be used to acquire the text of the email to be detected;

[0164] A text extraction module 1102, which is used to extract a preset number of valid sentences from the text of the email to be detected to obtain the valid text content;

[0165] The email detection module 1103 is used to input the valid text content into the phishing email detection model to determine whether the email text to be detected is a phishing email. The phishing email detection model is obtained by training the parameters of a preset network model based on the classification results of valid sample content and whether the valid sample content is a phishing email. The valid sample content is obtained by extracting a preset number of valid sentences from multiple sample emails that are known to be phishing emails.

[0166] The specific implementation process of the functions and roles of each module in the above-mentioned device can be found in the implementation process of the corresponding steps in the above-mentioned phishing email detection method, and will not be repeated here.

[0167] Figure 12 This is a block diagram illustrating a training apparatus for a phishing email detection model according to an embodiment of this application. Figure 12 As shown, the device:

[0168] The sample acquisition module 1201 is used to acquire multiple sample emails that are known to be phishing emails;

[0169] Content extraction module 1202 is used to extract a preset number of valid sentences from the sample email to obtain valid sample content;

[0170] The model training module 1203 is used to train the parameters of a preset network model based on the content of the effective samples and the classification result of whether the content of the effective samples is a phishing email, so as to obtain a phishing email detection model.

[0171] The specific implementation process of the functions and roles of each module in the above-mentioned device can be found in the corresponding steps of the training method of the above-mentioned phishing email detection model, and will not be repeated here.

[0172] The apparatuses and methods disclosed in the several embodiments provided in this application can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0173] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0174] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A method for detecting phishing emails, characterized in that, include: Get the text of the email to be tested; A preset number of valid sentences are extracted from the email text to be detected to obtain valid text content; The valid text content is input into a phishing email detection model to determine whether the email text to be detected is a phishing email. The phishing email detection model is obtained by training the parameters of a preset network model based on the classification results of valid sample content and whether the valid sample content is a phishing email. The valid sample content is obtained by extracting a preset number of valid sentences from multiple sample emails known to be phishing emails. The phishing email detection model includes: an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer connected in sequence. The embedding layer is used to map the words contained in the valid text content into a first vector. The word long short-term memory layer receives the first vector corresponding to each word in the effective text content and extracts the lexical features of each word; the word attention layer receives the lexical features of each word in the effective text content and outputs the sentence features of each effective sentence in the effective text content; the sentence long short-term memory layer receives the sentence features of each effective sentence in the effective text content and outputs the sentence context features of each effective sentence in the effective text content; the sentence attention layer receives the sentence context features of each effective sentence in the effective text content and outputs the overall features of the effective text content; the classification function layer receives the overall features and outputs the classification result. The step of extracting a preset number of valid sentences from the email text to be detected to obtain valid text content includes: The email text to be detected is preprocessed to obtain a list of words corresponding to the target sentence in the email text to be detected. The number of valid characters contained in the target sentence is determined based on the serial numbers corresponding to different words in the dictionary and the word list corresponding to the target sentence in the email text to be detected. Target sentences with a number of valid characters greater than a preset value are identified as valid sentences, and a preset number of valid sentences are extracted sequentially from the email text to be detected to obtain the valid text content.

2. The method according to claim 1, characterized in that, The preprocessing of the email text to be detected yields a vocabulary list corresponding to the target sentence in the email text, including: The email text to be detected is segmented into sentences to obtain the target sentence; Compare the number of Chinese words in the target sentence with the number of English words; If the number of Chinese words is greater than the number of English words, only the Chinese words are retained to obtain a list of Chinese words corresponding to the target sentence; if the number of Chinese words is less than the number of English words, only the English words are retained to obtain a list of English words corresponding to the target sentence.

3. The method according to claim 2, characterized in that, Before performing sentence segmentation on the email text to be detected to obtain the target sentence, the method further includes: The email subject of the email to be detected is decoded, the email body is converted to Traditional Chinese characters, and escape characters are removed.

4. The method according to claim 1, characterized in that, Before determining the number of valid characters contained in the target sentence based on the serial numbers corresponding to different words in the thesaurus and the word list corresponding to the target sentence in the email text to be detected, the method further includes: Obtain a sample email set containing multiple sample emails, the sample email set including phishing emails and non-phishing emails; The word library is formed by extracting the words with the highest word frequency-inverse document frequency values ​​from each sample email. The word library includes multiple words and the corresponding serial number for each word.

5. The method according to claim 4, characterized in that, Before extracting the words with the highest term frequency-inverse document frequency values ​​from each sample email to form the vocabulary, the method further includes: For each sample email, the body text is extracted from the sample email and then segmented into multiple sample sentences; For each sample sentence, the sample sentence is segmented to obtain the vocabulary contained in the sample email.

6. The method according to claim 5, characterized in that, For each sample sentence, word segmentation is performed to obtain the vocabulary contained in the sample email, including: For each sample sentence in the sample email, identify whether the sample sentence is a Chinese sentence; If the sample sentence is a Chinese sentence, perform traditional Chinese character replacement and word segmentation on the Chinese sentence to obtain the vocabulary contained in the Chinese sentence; If the sample sentence is an English sentence, the English sentence is segmented, and abnormal English words are segmented, while the correct words are retained, to obtain the vocabulary contained in the English sentence.

7. The method according to claim 1, characterized in that, The step of inputting the valid text content into the phishing email detection model to determine whether the email text to be detected is a phishing email includes: Input the valid text content into the phishing email detection model to obtain the confidence score output by the phishing email detection model; If the confidence level is greater than or equal to the threshold, the email text to be detected is determined to be a phishing email; If the confidence level is less than the threshold, the email text to be detected is determined to be a non-phishing email.

8. The method according to claim 1, characterized in that, Before inputting the valid text content into the phishing email detection model to determine whether the email text to be detected is a phishing email, the method further includes: Retrieve multiple sample emails known to be phishing emails; A preset number of valid sentences are extracted from the sample emails to obtain valid sample content; Based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, the parameters of the Bi-LSTM + Attention network model are trained to obtain the phishing email detection model trained by the Bi-LSTM + Attention network model.

9. The method according to claim 8, characterized in that, The Bi-LSTM + Attention network model includes, in sequence, an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer.

10. The method according to claim 9, characterized in that, The step of training the parameters of the Bi-LSTM + Attention network model based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, and obtaining the phishing email detection model trained by the Bi-LSTM + Attention network model, includes: The overall features of the effective sample content are extracted by sequentially connecting the embedding layer, the word long short-term memory layer, the word attention layer, the sentence long short-term memory layer, and the sentence attention layer. The overall features are used as input to the classification function layer to obtain the prediction result output by the classification function layer. Based on the prediction result and the known classification result, the parameters of the Bi-LSTM + Attention network model are optimized until the training objective is met, thus obtaining a phishing email detection model trained by the Bi-LSTM + Attention network model.

11. The method according to claim 10, characterized in that, The extraction of overall features of the effective sample content through sequentially connected embedding layers, word long short-term memory layers, word attention layers, sentence long short-term memory layers, and sentence attention layers includes: The embedding layer maps the words contained in the effective sample content into a first vector; The first vector corresponding to each word in the effective sample content is used as input, and the word features of each word are extracted through the long short-term memory layer of the word. The lexical features of each word in the effective sample content are used as input, and the sentence features of each effective sentence in the effective sample content are obtained through the word attention layer. Based on the sentence features of each valid sentence in the valid sample content, the sentence context features of each valid sentence in the valid sample content are obtained through the sentence long short-term memory layer; Based on the sentence context features of each valid sentence in the valid sample content, the overall features of the valid sample content are obtained through a sentence attention layer.

12. A training method for a phishing email detection model, characterized in that, include: Retrieve multiple sample emails known to be phishing emails; A preset number of valid sentences are extracted from the sample emails to obtain valid sample content; wherein, the valid sentences are sentences in the sample emails that contain a number of valid characters greater than a preset value; the number of valid characters is determined based on the serial numbers corresponding to different words in the thesaurus and the word list corresponding to each sentence in the sample emails; Based on the content of the valid samples and the classification result of whether the content of the valid samples is a phishing email, the parameters of the preset network model are trained to obtain a phishing email detection model. The phishing email detection model comprises, in sequence, an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer. The embedding layer maps the words contained in the valid sample content to a first vector. The word long short-term memory layer receives the first vector corresponding to each word in the valid sample content and extracts the lexical features of each word. The word attention layer receives the lexical features of each word in the valid sample content and outputs the sentence features of each valid sentence in the valid sample content. The sentence long short-term memory layer receives the sentence features of each valid sentence in the valid sample content and outputs the sentence context features of each valid sentence in the valid sample content. The sentence attention layer receives the sentence context features of each valid sentence in the valid sample content and outputs the overall features of the valid sample content. The classification function layer receives the overall features and outputs the classification result.

13. The method according to claim 12, characterized in that, The preset network model is a Bi-LSTM + Attention network model, which includes: an embedding layer, a word long short-term memory layer, a word attention layer, a sentence long short-term memory layer, a sentence attention layer, and a classification function layer connected in sequence.

14. The method according to claim 13, characterized in that, The step of training the parameters of a preset network model based on the content of the valid samples and the classification result of whether the valid sample content is a phishing email, to obtain a phishing email detection model, includes: The overall features of the effective sample content are extracted by sequentially connecting the embedding layer, the word long short-term memory layer, the word attention layer, the sentence long short-term memory layer, and the sentence attention layer. The overall features are used as input to the classification function layer to obtain the prediction result output by the classification function layer. Based on the prediction result and the known classification result, the parameters of the Bi-LSTM + Attention network model are optimized until the training objective is met, thus obtaining a phishing email detection model trained by the Bi-LSTM + Attention network model.

15. The method according to claim 14, characterized in that, The extraction of overall features of the effective sample content through sequentially connected embedding layers, word long short-term memory layers, word attention layers, sentence long short-term memory layers, and sentence attention layers includes: The embedding layer maps the words contained in the effective sample content into a first vector; The first vector corresponding to each word in the effective sample content is used as input, and the word features of each word are extracted through the long short-term memory layer of the word. The lexical features of each word in the effective sample content are used as input, and the sentence features of each effective sentence in the effective sample content are obtained through the word attention layer. Based on the sentence features of each valid sentence in the valid sample content, the sentence context features of each valid sentence in the valid sample content are obtained through the sentence long short-term memory layer; Based on the sentence context features of each valid sentence in the valid sample content, the overall features of the valid sample content are obtained through a sentence attention layer.

16. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the phishing email detection method according to any one of claims 1-11 or the phishing email detection model training method according to any one of claims 12-15.

17. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to perform the phishing email detection method according to any one of claims 1-11 or the phishing email detection model training method according to any one of claims 12-15.

Images