Junk mail filtering method and device, electronic equipment and readable storage medium

Abstract

The invention relates to a junk mail filtering method and device, electronic equipment and a readable storage medium, and belongs to the technical field of communication. The method comprises the following steps: establishing a junk mail set and a non-junk mail set; performing word segmentation based on a word bag model technology to obtain corresponding independent character strings, and calculating corresponding word frequencies; establishing a junk mail hash table and a non-junk mail hash table; performing word segmentation on the new mail based on a bag-of-word model technology to obtain independent character strings and calculate word frequencies, establishing a temporary hash table, and calculating word frequencies of the independent character strings in the temporary hash table in the junk mail hash table and the non-junk mail hash table; calculating a basic word frequency threshold value and adding the basic word frequency threshold value to the temporary hash table; and judging whether the new mail is a junk mail based on a size relationship between the word frequency of the independent character string in the temporary hash table and a basic word frequency threshold value, and updating the junk mail hash table and the non-junk mail hash table. The problem that an existing spam filtering method is low in filtering accuracy is solved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a spam filtering method, device, electronic device and readable storage medium. Background Art

[0002] As an early pioneer of digital communications, email still supports the operation of many business processes around the world and has become an indispensable part of daily work. However, along with its widespread use, the problem of spam is becoming increasingly serious. Early spam was mainly limited to unsolicited commercial advertisements, but with the rapid development of Internet technology, its form has expanded to include malware, computer viruses, and content disguised as topics of interest to users, which seriously threatens personal information security and network security. Therefore, effectively dealing with spam has become an important issue in maintaining the health of the digital communication environment.

[0003] Existing anti-spam technologies mainly rely on methods such as keyword filtering, blacklist filtering, and filtering based on regular expression rules. These methods are widely used in mail server plug-ins, anti-spam gateways, and client functions. However, these technologies have significant limitations: First, they are easy to bypass. Spammers can easily circumvent keyword-based filtering mechanisms by using techniques such as word splitting, word grouping, or special character replacement. Even the more advanced Bayesian filter algorithms need to be frequently updated to cope with the ever-changing spam strategies, which poses a timeliness challenge. Secondly, the high false alarm rate is another major problem. If a normal email contains keywords or sentences similar to spam, it may be mistakenly classified as spam, causing inconvenience to users. Finally, although the filter review mechanism alleviates the false alarm problem to a certain extent, it requires users to manually check the spam box, which increases time cost and workload, and reduces overall communication efficiency. Therefore, the filtering accuracy of the spam filtering method in the prior art is relatively low.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a spam filtering method, device, electronic device and readable storage medium to solve the technical problems of high false alarm rate and low filtering accuracy of the above-mentioned email filtering method.

[0006] According to one aspect of an embodiment of the present application, the present application provides a spam filtering method, comprising:

[0007] Establish corresponding spam and non-spam collections based on spam and non-spam in the mail system;

[0008] Based on the bag-of-words model technology, each email in the spam email set and the non-spam email set is segmented to obtain a corresponding independent string, the word frequency of the independent string in different emails is calculated, and the corresponding average word frequency is calculated based on the word frequency in the different emails;

[0009] A spam hash table is established based on the correspondence between each independent string and each email in the spam email set in combination with the word frequency and average word frequency of each independent string in the spam email set; a non-spam hash table is established based on the correspondence between each independent string and each email in the non-spam email set in combination with the word frequency and average word frequency of each independent string in the non-spam email set;

[0010] Based on the bag-of-words model technology, the new email is segmented to obtain the independent character strings corresponding to the new email and the corresponding word frequencies are calculated, a temporary hash table is established based on the independent character strings in the new email and the corresponding word frequencies, and the word frequencies of each independent character string in the temporary hash table in the spam hash table and the non-spam hash table are calculated;

[0011] Calculate a basic word frequency threshold by combining the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and add the basic word frequency threshold to the temporary hash table;

[0012] Based on the relationship between the word frequency of each independent character string in the temporary hash table and the corresponding basic word frequency threshold, it is determined whether the corresponding new email is spam; based on the temporary hash table, the spam hash table and the non-spam hash table are updated; based on the updated spam hash table and the non-spam hash table, it is determined whether the next new email is spam.

[0013] Optionally, the step of establishing corresponding spam email sets and non-spam email sets based on spam emails and non-spam emails in the email system respectively includes:

[0014] Obtain spam data and non-spam data in the existing mail system;

[0015] Performing data cleaning on the spam data and the non-spam data respectively to obtain spam and non-spam;

[0016] Marking each of the spam emails to obtain a first mark, and marking each of the non-spam emails to obtain a second mark;

[0017] A spam email set is generated from each of the spam emails based on the first annotation, and a non-spam email set is generated from each of the non-spam emails based on the second annotation.

[0018] Optionally, the bag-of-words model technology is used to segment each email in the spam email set and the non-spam email set to obtain a corresponding independent string, calculate the word frequency of the independent string in different emails, and calculate the corresponding average word frequency based on the word frequency in the different emails, including:

[0019] Preprocessing each email in the spam email set and the non-spam email set respectively;

[0020] Based on the bag-of-words model technology, each email in the spam email set is segmented to obtain a first independent string, and each email in the non-spam email set is segmented to obtain a second independent string;

[0021] Calculate the first word frequency of the first independent string in different emails based on the ratio of the number of each first independent string in each email corresponding to the spam email set to the sum of the number of first independent strings in each email; calculate the second word frequency of the second independent string in different emails based on the ratio of the number of each second independent string in each email corresponding to the non-spam email set to the sum of the number of second independent strings in each email;

[0022] A first average word frequency corresponding to the first independent string is calculated based on the first word frequency corresponding to the first independent string of each email in the spam set, and a second average word frequency corresponding to the second independent string is calculated based on the second word frequency corresponding to the second independent string of each email in the non-spam set.

[0023] Optionally, the preprocessing of each email in the spam email set and the non-spam email set respectively includes:

[0024] Performing text analysis on each email in the spam email set and the non-spam email set to extract corresponding text features;

[0025] Performing text cleaning on non-text features in the spam email set and the non-spam email set to obtain first text features, wherein the text cleaning includes removing HTML tags, and / or escaping special characters, and / or unifying line breaks;

[0026] Removing punctuation marks from the first text features corresponding to the spam email set and the non-spam email set to obtain second text features;

[0027] Stop words are removed from the second text features corresponding to the spam email set and the non-spam email set respectively to obtain a pre-processed spam email set and a pre-processed non-spam email set.

[0028] Optionally, the word segmentation of each email in the spam email set to obtain a first independent string based on the bag-of-words model technology, and the word segmentation of each email in the non-spam email set to obtain a second independent string include:

[0029] Determine email text dictionary thresholds;

[0030] Scan the text of each email in the spam email set and the non-spam email set from left to right based on a forward maximum matching algorithm;

[0031] Selecting the word with the largest length in the email text dictionary threshold to match the spam email set, and if the match is successful, using the word as the first independent character string;

[0032] A word with the largest length in the email text dictionary threshold is selected to match the non-spam email set, and if the match is successful, the word is used as the second independent character string.

[0033] Optionally, the step of establishing a spam hash table based on the correspondence between each independent string in the spam set and each email in combination with the word frequency and average word frequency of each independent string in the spam set, and establishing a non-spam hash table based on the correspondence between each independent string in the non-spam set and the email in combination with the word frequency and average word frequency of each independent string in the non-spam set includes:

[0034] Initializing a hash table template, including setting a key map in the hash table template to a first independent string and a second independent string, and setting a value map in the hash table template to a first independent string quantity, a first word frequency, and a first average word frequency in each email in the spam email set, and a second independent string quantity, a second word frequency, and a second average word frequency in each email in the non-spam email set;

[0035] Filling each first independent character string, first word frequency and corresponding first average word frequency of the spam email set into the hash table template to obtain a spam email hash table;

[0036] The second independent character strings, the second word frequencies and the corresponding second average word frequencies in the non-spam set are filled into the hash table template to obtain a non-spam hash table.

[0037] Optionally, the word segmentation of the new email based on the bag-of-words model technology to obtain independent character strings corresponding to the new email and calculate the corresponding word frequencies, establishing a temporary hash table based on the independent character strings in the new email and the corresponding word frequencies, and calculating the word frequencies of each independent character string in the temporary hash table in the spam hash table and the non-spam hash table, includes:

[0038] Obtain new emails and pre-process the new emails;

[0039] Extracting the third independent string corresponding to the preprocessed new email based on the bag-of-words model technology, and calculating the third word frequency of the third independent string in the new email based on the ratio of the number of the third independent string in the new email and the sum of the numbers of each third independent string in the new email;

[0040] Establishing a temporary hash table based on the number of the third independent character string in the new email and the corresponding third word frequency;

[0041] The fourth word frequency of each of the third independent character strings in the temporary hash table in the spam hash table and the fifth word frequency in the non-spam hash table are calculated respectively.

[0042] Optionally, the acquiring of new emails and preprocessing of the new emails includes:

[0043] Obtaining new emails updated in the email system, and performing text parsing on the new emails to extract corresponding new email body features;

[0044] Performing text cleaning on the new email body feature to obtain a first new email body feature, wherein the text cleaning includes removing HTML tags, and / or escaping special characters, and / or unifying line breaks;

[0045] Removing punctuation marks from the first new email body feature to obtain a second new email body feature;

[0046] Stop words are removed from the second new email body feature to obtain a preprocessed new email.

[0047] Optionally, the step of calculating a basic word frequency threshold by combining the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and adding the basic word frequency threshold to the temporary hash table comprises the following steps:

[0048] Determine the sum of the fourth word frequency and the fifth word frequency of the third independent character string in the temporary hash table;

[0049] Calculating a basic word frequency threshold based on a ratio of a fourth word frequency of the third independent character string and a sum of the fourth word frequency and the fifth word frequency;

[0050] The basic word frequency threshold is added to the temporary hash table to correspond to the third independent character string to update the temporary hash table.

[0051] Optionally, judging whether the corresponding new email is spam based on the relationship between the word frequency of each independent string in the temporary hash table and the corresponding basic word frequency threshold, updating the spam hash table and the non-spam hash table based on the temporary hash table, and judging whether the next new email is spam based on the updated spam hash table and the non-spam hash table, includes:

[0052] Determine a magnitude relationship between the third word frequency of the third independent character string in the temporary hash table and the corresponding basic word frequency threshold;

[0053] If the third word frequency is greater than the basic word frequency threshold, the output result is 1, otherwise the output result is 0;

[0054] Summing all the output results to obtain a final output result, and judging whether the new email is a spam based on the ratio of the final output result to the number of all third independent character strings of the new email, if the ratio is 1, the new email is a spam, otherwise the new email is not a spam;

[0055] The spam hash table and the non-spam hash table are updated based on the temporary hash table, and whether the next email is a spam is determined based on the updated spam hash table and the non-spam hash table.

[0056] Optionally, updating the spam hash table and the non-spam hash table based on the temporary hash table, and determining whether the next email is a spam based on the updated spam hash table and the non-spam hash table, includes:

[0057] If the new email is a spam email, updating the spam email hash table based on the temporary hash table corresponding to the new email;

[0058] Determine whether the next email is spam by combining the updated spam hash table with the unupdated non-spam hash table;

[0059] If the new email is not a spam email, updating the non-spam email hash table based on the temporary hash table corresponding to the new email;

[0060] It is determined whether the next email is a spam email by combining the updated non-spam email hash table with the unsealed spam email hash table.

[0061] Optionally, the spam filtering method further includes:

[0062] If there are emails that fail to be judged as spam, manually mark the emails that fail to be judged as spam;

[0063] Extract the corresponding independent string according to the manual marking result and update it into the corresponding email hash table;

[0064] If the email that fails to be determined is a spam email, the spam email hash table is updated based on the independent character string;

[0065] If the email that fails to be determined is not a spam email, the waste spam email hash table is updated based on the independent character string.

[0066] According to another aspect of the embodiment of the present application, the present application provides a spam filtering device, including:

[0067] An email set building module, used to build corresponding spam email sets and non-spam email sets based on spam emails and non-spam emails in the email system;

[0068] The email word segmentation module is used to segment each email in the spam email set and the non-spam email set based on the bag-of-words model technology to obtain the corresponding independent strings, calculate the word frequency of the independent strings in different emails, and calculate the corresponding average word frequency based on the word frequency in different emails;

[0069] The email hash table construction module is used to build a spam hash table based on the correspondence between each independent string in the spam email set and each email, combined with the word frequency and average word frequency of each independent string in the spam email set, and to build a non-spam hash table based on the correspondence between each independent string in the non-spam email set and the email, combined with the word frequency and average word frequency of each independent string in the non-spam email set;

[0070] A new mail processing module is used to segment new mails based on the bag-of-words model technology to obtain independent strings corresponding to the new mails and calculate the corresponding word frequencies, establish a temporary hash table based on the independent strings in the new mails and the corresponding word frequencies, and calculate the word frequencies of each independent string in the temporary hash table in the spam mail hash table and the non-spam mail hash table;

[0071] A new email hash table construction module, used to calculate a basic word frequency threshold by combining the word frequency of each independent string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and add the basic word frequency threshold to the temporary hash table;

[0072] The spam judgment module is used to judge whether the corresponding new email is spam based on the relationship between the word frequency of each independent string in the temporary hash table and the corresponding basic word frequency threshold, update the spam hash table and the non-spam hash table based on the temporary hash table, and judge whether the next new email is spam based on the updated spam hash table and the non-spam hash table.

[0073] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0074] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0075] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0076] The present application can dynamically adapt to the changing strategies of spam senders by continuously collecting new spam and non-spam samples and updating the spam hash table and non-spam hash table based on these samples. This autonomous learning mechanism significantly improves the accuracy and timeliness of the filtering system. By calculating the word frequency of independent strings in spam and non-spam, and setting the basic word frequency threshold, the category of new emails can be judged more accurately, which is more complex and sophisticated than simple keyword matching, and reduces the possibility of false positives and missed negatives. The present invention not only considers the word frequency of independent strings, but also combines the distribution of these strings in spam and non-spam, and performs a comprehensive evaluation by calculating the basic word frequency threshold. This multi-factor consideration method reduces false positives caused by single keyword matching errors. As the hash table is continuously updated during the processing of new emails, more features about spam and non-spam can be gradually learned, thereby further improving the accuracy of classification and reducing the false positive rate. Through automated hash table updates and email classification, the need for users to manually check spam boxes is reduced, reducing time costs and workload. This solution improves the accuracy and efficiency of email filtering, ensures that normal emails can be delivered to users in a timely manner, and improves overall communication efficiency. This solution can continuously learn and adapt to new spam characteristics, and maintain high filtering accuracy even in the face of ever-changing spam strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0079] Figure 1 A schematic diagram of a hardware environment for a spam filtering method provided according to an embodiment of the present application;

[0080] Figure 2 A schematic diagram of a spam filtering method according to an embodiment of the present application;

[0081] Figure 3 A block diagram of a spam filtering device provided according to an embodiment of the present application;

[0082] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0084] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0085] In the related art, spam filtering through keyword filtering, blacklist or regular expression methods has the problems of easy bypass, high false alarm rate and cumbersome review, which is prone to false alarms and leads to low accuracy of spam filtering.

[0086] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a spam filtering method is provided.

[0087] Optionally, in the embodiment of the present application, the above method can be applied to Figure 1In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0088] A spam filtering method in the embodiment of the present application can be executed by the server 103, or can be executed by the server 103 and the terminal 101 together, such as Figure 2 As shown, the method may include the following steps:

[0089] S202: establishing corresponding spam email sets and non-spam email sets based on the spam emails and non-spam emails in the email system.

[0090] Specifically, S202 includes:

[0091] Obtain spam data and non-spam data in the existing mail system;

[0092] Perform data cleaning on spam data and non-spam data respectively to obtain spam and non-spam;

[0093] Marking each spam email to obtain a first mark, and marking each non-spam email to obtain a second mark;

[0094] A spam email set is generated from each spam email based on the first annotation, and a non-spam email set is generated from each non-spam email based on the second annotation.

[0095] In this embodiment, it is assumed that the emails obtained from the email system include email 1, email 2, email 3, email 4 and email 5, wherein email 1, email 2 and email 3 are non-spam emails, email 4 and email 5 are spam emails, email 1 specifically states: I love machine learning, email 2 specifically states: Machine learning is very interesting, email 3 specifically states: I love interesting activities, email 4 specifically states: Activities make me happy, and email 5 specifically states: Live happily every day. Email 1, email 2 and email 3 are respectively labeled to obtain a first label [good], and email 4 and email 5 are respectively labeled to obtain a second label [bad]. According to the first label [good], email 1, email 2 and email 3 are automatically merged into the same email set to obtain a non-spam email set [good_emil] = [emil 1, emil 2, emil 3], and according to the second label [bad], email 4 and email 5 are automatically merged into the same email set to obtain a spam email set [bad_emil] = [emil 4, emil5].

[0096] In the above embodiments, noise, errors, duplications or irrelevant information in spam and non-spam data can be removed by data cleaning, thereby improving the accuracy and quality of the data set. By clearly labeling each email, the email category in the data set is ensured, which helps the model learn the correct classification boundary. The spam and non-spam sets constructed based on the cleaned and labeled data can more comprehensively reflect the distribution and characteristics of emails in the actual email system. By pre-constructing spam and non-spam sets, the deployment and iteration speed of the email classification system can be accelerated. When a new email enters the system, it can be quickly classified directly using the trained model without the need for tedious data cleaning and labeling, thereby improving the overall processing efficiency and response speed of the system.

[0097] S204, based on the bag-of-words model technology, segment each email in the spam email set and the non-spam email set to obtain a corresponding independent string, calculate the word frequency of the independent string in different emails, and calculate the corresponding average word frequency based on the word frequency in different emails.

[0098] Specifically, S204 includes:

[0099] Pre-process each email in the spam email set and the non-spam email set respectively;

[0100] Based on the bag-of-words model technology, each email in the spam email set is segmented to obtain a first independent string, and each email in the non-spam email set is segmented to obtain a second independent string;

[0101] Calculating the first word frequency of the first independent string in different emails based on the ratio of the number of each first independent string in each email corresponding to the spam email set to the sum of the number of the first independent string in each email, and calculating the second word frequency of the second independent string in different emails based on the ratio of the number of each second independent string in each email corresponding to the non-spam email set to the sum of the number of the second independent string in each email;

[0102] A first average word frequency corresponding to the first independent string is calculated based on the first word frequency corresponding to the first independent string of each email in the spam email set, and a second average word frequency corresponding to the second independent string is calculated based on the second word frequency corresponding to the second independent string of each email in the non-spam email set.

[0103] In this embodiment, firstly, each email in the obtained spam email set and non-spam email set is preprocessed, and the preprocessing mainly includes text cleaning, punctuation removal and stop word removal. By analyzing the non-spam email set [emil 1, emil 2, emil 3] and the spam email set [emil 4, emil 5], it can be seen that the present embodiment does not need to preprocess the non-spam email set [emil 1, emil 2, emil 3] and the spam email set [emil 4, emil 5]. The non-spam email set and the spam email set are segmented by the bag-of-words model technology, and the first independent character strings included in the spam email set [emil 4, emil 5] are respectively "activity", "make", "I", "change", "happy", "live well", and "everyday", and the second independent character strings included in the non-spam email set [emil 1, emil 2, emil 3] are respectively "I", "love", "machine", "learning", "is", "very", "interesting", "of", and "activity". Then, the number of each first independent string in Mail 4 and Mail 5 in the spam mail set is counted, and the first word frequency of each first independent string in each mail is obtained by dividing the counted number by the number of all first independent strings in Mail 4 and the number of all first independent strings in Mail 5, and the first average word frequency is calculated based on the first word frequency of the corresponding first independent string in Mail 4 and Mail 5. The number of each second independent string in Mail 1, Mail 2, and Mail 3 in the non-spam mail set is counted, and the second word frequency of each second independent string in each mail is obtained by dividing the counted number by the number of all second independent strings in Mail 1, the number of all second independent strings in Mail 2, and the number of all second independent strings in Mail 3, and the second average word frequency is calculated based on the second word frequency of the corresponding second independent string in Mail 1, Mail 2, and Mail 3.

[0104] In the above embodiment, by preprocessing to remove stop words and punctuation marks, the noise in the data can be reduced, thereby improving the quality of subsequent feature extraction and model training. The bag-of-words model can effectively convert text data into a numerical form that can be processed by the machine learning algorithm, which is convenient for subsequent word frequency calculation and feature extraction. By calculating the word frequency and average word frequency of each independent string in different email sets, the feature differences between different email categories can be captured. The calculation of the average word frequency further enhances the representativeness of the features, allowing the model to more accurately capture the category information of the email.

[0105] Furthermore, each email in the spam email set and the non-spam email set is pre-processed respectively, including:

[0106] Perform text analysis on each email in the spam email set and the non-spam email set to extract the corresponding text features;

[0107] Performing text cleaning on non-text features in the spam email set and the non-spam email set to obtain first text features, wherein the text cleaning includes removing HTML tags, and / or escaping special characters, and / or unifying line breaks;

[0108] Removing punctuation marks from the first text features corresponding to the spam email set and the non-spam email set to obtain second text features;

[0109] Stop words are removed from the second text features corresponding to the spam email set and the non-spam email set respectively to obtain the pre-processed spam email set and the non-spam email set.

[0110] In the above embodiment, the text content of the email can be accurately extracted through text parsing, excluding non-core content such as email headers and attachments. By removing HTML tags, escaping special characters, unifying line breaks and other operations, the noise and format inconsistency in the data are reduced, the standardization of the data is improved, and it is conducive to the consistency of subsequent processing. Although punctuation marks can express emotions or emphasize, they may be regarded as noise in machine learning models, affecting the performance of the model. After removing punctuation marks, text data can be more concise and feature expression can be more direct. Removing stop words can reduce the dimension of the feature space, improve the model processing speed and classification accuracy, and also highlight the key information in the text and enhance the expressiveness of the features.

[0111] Furthermore, based on the bag-of-words model technology, each email in the spam email set is segmented to obtain a first independent string, and each email in the non-spam email set is segmented to obtain a second independent string, including:

[0112] Determine email text dictionary thresholds;

[0113] Scan the text of each email in the spam email set and the non-spam email set from left to right based on the forward maximum matching algorithm;

[0114] Select the word with the longest length in the email text dictionary threshold to match the spam email set, and if the match is successful, use the word as the first independent string;

[0115] The word with the largest length in the email text dictionary threshold is selected to match the non-spam email set, and if the match is successful, the word is used as the second independent string.

[0116] In the above embodiment, by setting the email text dictionary threshold and using the forward maximum matching algorithm for word segmentation, meaningful vocabulary units can be extracted from the email text more accurately, reducing semantic errors caused by improper segmentation and improving the accuracy of subsequent text processing and analysis. By scanning the text from left to right, the longest word can be quickly located and matched, reducing unnecessary comparisons and judgments, and improving overall processing efficiency. By using a unified dictionary threshold and matching algorithm in the word segmentation process, it helps to build a more consistent and robust text representation, thereby enhancing the generalization ability of machine learning models based on these representations and improving performance on different data sets.

[0117] S206, a spam hash table is established based on the correspondence between each independent string in the spam set and each email, combined with the word frequency and average word frequency of each independent string in the spam set; a non-spam hash table is established based on the correspondence between each independent string in the non-spam set and the email, combined with the word frequency and average word frequency of each independent string in the non-spam set.

[0118] Specifically, S206 includes:

[0119] Initializing a hash table template, including setting a key map in the hash table template to a first independent string and a second independent string, and setting a value map in the hash table template to a first independent string quantity, a first word frequency, and a first average word frequency in each email in the spam email set, and a second independent string quantity, a second word frequency, and a second average word frequency in each email in the non-spam email set;

[0120] Fill each first independent character string, first word frequency and corresponding first average word frequency of the spam email set into the hash table template to obtain a spam email hash table;

[0121] Fill each second independent character string, second word frequency and corresponding second average word frequency in the non-spam set into the hash table template to obtain a non-spam hash table.

[0122] In this embodiment, a hash table template is initialized, the key of the hash table template corresponds to the first independent string or the second independent string, and the hash table template corresponds to the number of second independent strings, the second word frequency and the second average word frequency in Mail 1, Mail 2 and Mail 3 or the number of first independent strings, the first word frequency or the first average word frequency in Mail 4 and Mail 5. The second independent strings "I", "love", "machine", "learning", "yes", "very", "interesting", "of", "activity" and the corresponding second word frequency and second average word frequency of Mail 1, Mail 2 and Mail 3 in the non-spam set are filled into the non-spam hash table to obtain the non-spam hash table ht_good shown in Table 1. The vocabulary in Table 1 represents the corresponding second independent string, the P1 value in Table 1 represents the second word frequency, and the AVG1 (P value) probability value in Table 1 represents the second average word frequency.

[0123] Table 1 Non-spam hash table (ht_good)

[0124]

[0125] The first independent character strings "activity", "make", "I", "change", "happy", "live well", "every day" and the corresponding first word frequency and first average word frequency of email 4 and email 3 in the spam set are filled into the spam hash table to obtain the spam hash table ht_bad shown in Table 2. The vocabulary in Table 2 represents the corresponding first independent character strings, the P2 value in Table 2 represents the first word frequency, and the AVG2 (P value) probability value in Table 2 represents the first average word frequency.

[0126] Table 2 Spam hash table (ht_bad)

[0127]

[0128] In the above embodiment, by using independent character strings (vocabulary) in the email as keys and relevant statistical information (such as quantity, word frequency, average word frequency) as values ​​to construct a corresponding hash table, email data can be efficiently organized and retrieved. By constructing a spam hash table and a non-spam hash table separately, the difference between the two types of emails can be captured more accurately, thereby improving the accuracy of spam identification.

[0129] S208, segmenting the new email based on the bag-of-words model technology to obtain independent character strings corresponding to the new email and calculate the corresponding word frequencies, establishing a temporary hash table based on the independent character strings in the new email and the corresponding word frequencies, and calculating the word frequencies of each independent character string in the temporary hash table in the spam hash table and the non-spam hash table.

[0130] Specifically, S208 includes:

[0131] Get new emails and pre-process them;

[0132] Extracting the third independent string corresponding to the preprocessed new email based on the bag-of-words model technology, and calculating the third word frequency of the third independent string in the new email based on the ratio of the number of the third independent string in the new email and the sum of the numbers of each third independent string in the new email;

[0133] A temporary hash table is established based on the number of the third independent character string in the new email and the corresponding third word frequency;

[0134] The fourth word frequency of each third independent character string in the temporary hash table in the spam hash table and the fifth word frequency in the non-spam hash table are calculated respectively.

[0135] In this embodiment, firstly, new emails in the email system are obtained, and then the new emails are preprocessed. The third independent string corresponding to the preprocessed new email is extracted by the bag-of-words model technology. It is assumed that the obtained third independent string includes str1, str2...strN, and the third word frequency P3 of each third independent string is calculated. According to the third independent string, the number of third independent strings in the new email and the third word frequency P3, a temporary hash table ht_temp is constructed. The obtained temporary hash table is shown in Table 3, and the basic probability value in the table represents the basic word frequency threshold. The word frequency of each third independent string in the temporary hash table ht_temp in the non-spam hash table is calculated as the fourth word frequency, and the word frequency of each third independent string in the temporary hash table ht_temp in the spam hash table is calculated as the fifth word frequency. Among them, the calculation formulas for the fourth word frequency and the fifth word frequency are as follows:

[0136] P4(strN)=tn1, indicating the word frequency value of strN in ht_good (the fourth word frequency);

[0137] P5(strN)=tn2, indicating the word frequency value of strN in ht_bad (fifth word frequency).

[0138] Table 3 Temporary hash table ( h t_temp )

[0139]

[0140]

[0141] In the above embodiment, preprocessing the new email helps to standardize the vocabulary representation and improve the generalization ability of the classifier. Based on the third independent string extracted by the bag-of-words model technology, the third word frequency of the third independent string in the new email can be calculated to quantify their contribution to the email content and provide important features for subsequent email classification. By establishing a temporary hash table, it is possible to quickly compare the spam email set and the non-spam email set. By storing the independent strings and their word frequencies in the new email in the hash table, the performance of these words in the known email set can be efficiently queried and matched. The word frequencies of the words in the temporary hash table in the spam email hash table and the non-spam email hash table are calculated respectively, which helps to comprehensively evaluate the distribution of these words in the two types of emails. By comparing the third independent string and the third word frequency of the new email with the features of the known email set, the category of the new email can be more accurately determined, which has higher accuracy and robustness than simple keyword matching or rule judgment.

[0142] Furthermore, new emails are obtained and pre-processed, including:

[0143] Obtain new emails updated in the email system, perform text analysis on the new emails and extract the corresponding new email body features;

[0144] Performing text cleaning on the new email body feature to obtain a first new email body feature, wherein the text cleaning includes removing HTML tags, and / or escaping special characters, and / or unifying line breaks;

[0145] Removing punctuation marks from the first new email body feature to obtain a second new email body feature;

[0146] Stop words are removed from the second new email body feature to obtain a preprocessed new email.

[0147] In the above embodiment, by parsing and cleaning the text of new emails, such as removing HTML tags, escaping special characters and unifying line breaks, it is helpful to purify the email content and remove interference information, thereby improving the accuracy and efficiency of subsequent text analysis and classification. By removing punctuation marks and stop words, it is possible to focus more on the core content of the email and optimize text features. The preprocessed email content is more standardized, which enables subsequent spam detection to be based on a more accurate and consistent data set, thereby improving the accuracy and reliability of spam detection.

[0148] S210, calculating a basic word frequency threshold by combining the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and adding the basic word frequency threshold to the temporary hash table.

[0149] Specifically, S210 includes:

[0150] Determine the sum of the fourth word frequency and the fifth word frequency of the third independent character string in the temporary hash table;

[0151] A basic word frequency threshold is calculated based on the ratio of the fourth word frequency of the third independent character string and the sum of the fourth word frequency and the fifth word frequency;

[0152] The basic word frequency threshold is added to the temporary hash table to correspond to the third independent character string to update the temporary hash table.

[0153] In this embodiment, the sum of the first word frequency P4 and the fifth word frequency P5 corresponding to the third independent string in the temporary hash table ht_temp is first calculated, that is, P4(strN)+P5(strN), and the basic word frequency threshold is calculated based on the ratio of the fourth word frequency P4 and the sum of the fourth word frequency P4 and the fifth word frequency P5 P4(strN)+P5(strN), and the basic word frequency threshold is P(strN)=P4(strN) / [P4(strN)+P5(strN)]. The obtained basic word frequency is added to the temporary hash table ht_temp, as shown in Table 2, assuming that the basic word frequency threshold corresponding to the third independent string "str1" is 0.5, and the basic word frequency threshold corresponding to the third independent string "str2" is 0.2.

[0154] In the above embodiment, by combining the word frequency information of each independent string in the temporary hash table in the spam hash table and the non-spam hash table, spam and non-spam can be distinguished more accurately. The calculation of the basic word frequency threshold is based on the relative distribution of specific strings in the two types of emails, which helps to divide the email types more finely, thereby improving the accuracy of email classification.

[0155] S212, judging whether the corresponding new email is spam based on the relationship between the word frequency of each independent character string in the temporary hash table and the corresponding basic word frequency threshold, updating the spam hash table and the non-spam hash table based on the temporary hash table, and judging whether the next new email is spam based on the updated spam hash table and the non-spam hash table.

[0156] Specifically, S212 includes:

[0157] Determine the magnitude relationship between the third word frequency of the third independent character string in the temporary hash table and the corresponding basic word frequency threshold;

[0158] If the third word frequency is greater than the basic word frequency threshold, the output result is 1, otherwise the output result is 0;

[0159] All output results are summed up to obtain a final output result, and whether the new email is spam is determined based on the ratio of the final output result to the number of all third independent character strings of the new email. If the ratio is 1, the new email is spam, otherwise the new email is not spam;

[0160] The spam hash table and the non-spam hash table are updated based on the temporary hash table, and whether the next email is a spam is determined based on the updated spam hash table and the non-spam hash table.

[0161] In this embodiment, the third word frequency P3 of each third independent string is compared with the basic word frequency threshold P based on the basic word frequency threshold in the temporary hash table ht_temp. If the third word frequency P3 ≥ the basic word frequency threshold, the output result is 1, and if the third word frequency P3 < the basic word frequency threshold, the output result is 0. The output results of each third independent string are added, and the result obtained by the addition is divided by the number of third independent strings in the new email to obtain the final output result. If the final output result is equal to 100%, the new email is judged to be a spam email. If the final output result is not equal to 100%, the new email is judged to be a non-spam email. As shown in Table 3, the third word frequency P3 of the third independent string "str1" is 1 / 2, which is just equal to the corresponding basic word frequency threshold, and the output result is 1. The third word frequency P3 of the third independent string "str2" is 1 / 2, which is greater than the corresponding basic word frequency threshold, and the output result is 1. Then (1+1) / (1+1)=100% is obtained, so the new email is judged to be a spam email. The third independent character string in the temporary hash table ht_temp of the new email, as well as the corresponding relationship and quantity with the new email are updated to the spam hash table ht_bad, and the average word frequency of ht_bad in the spam hash table is updated, and the next email is judged based on the updated spam hash table.

[0162] In the above embodiment, by comparing the word frequency of independent character strings in the email with the preset basic word frequency threshold, it is possible to more accurately determine whether the email is spam, taking into account the actual distribution of the email content, which is more flexible and accurate than simple keyword matching. Dynamically updating the spam hash table and the non-spam hash table according to the content of the new email can continuously learn and adapt to new spam patterns and improve the ability to identify future spam. Using a hash table to store and update the word frequency information of a vocabulary can efficiently process a large amount of data and has good adaptability to newly emerging vocabulary or spam patterns. By comprehensively considering the relationship between the word frequency of all independent character strings in the email and the threshold, rather than relying solely on a few keywords, it is possible to reduce misjudgments caused by mismatching of keywords.

[0163] Further, the spam hash table and the non-spam hash table are updated based on the temporary hash table, and whether the next email is a spam email is determined based on the updated spam hash table and the non-spam hash table, including:

[0164] If the new email is spam, the spam hash table is updated based on the temporary hash table corresponding to the new email;

[0165] Determine whether the next email is spam by combining the updated spam hash table with the unupdated non-spam hash table;

[0166] If the new email is not spam, the non-spam hash table is updated based on the temporary hash table corresponding to the new email;

[0167] The updated non-spam hash table is combined with the unsealed spam hash table to determine whether the next email is spam.

[0168] In the above embodiment, real-time learning and adaptation of new email content can be achieved. Whether it is spam or non-spam, the corresponding hash table will be updated according to its content, which can enable the system to continuously adapt to new email patterns and vocabulary changes. By continuously updating the hash table and making judgments based on the hash table, it is possible to reduce misjudgments or missed judgments caused by changes in email content. For example, some new spam patterns may be learned and updated to the spam hash table in a timely manner, thereby avoiding missed judgments of these emails. When judging whether the next email is spam, the hash table can be directly used for fast matching and comparison, thereby improving the judgment efficiency. Since the hash table is learned based on a large amount of email data, it can improve the accuracy of the judgment.

[0169] Furthermore, the spam filtering method in this embodiment also includes:

[0170] If there are emails that fail to be identified as spam, manually mark the emails that fail to be identified as spam;

[0171] Extract the corresponding independent string according to the manual marking result and update it into the corresponding email hash table;

[0172] If the email that failed to be judged is spam, the spam hash table is updated based on the independent string;

[0173] If the email that fails to be judged is not spam, the hash table of waste spam is updated based on the independent string.

[0174] In the above embodiment, by manually marking the emails that failed to be judged, the errors in automatic judgment can be corrected, and the independent character strings in these emails are updated to the corresponding email hash table. The system can learn and adapt to the email patterns that were previously judged incorrectly, thereby improving the accuracy of subsequent judgments. The system can learn and update the hash table from the emails that failed to be judged, which enhances the self-learning ability and adaptability of the system. As time goes by, the system can continuously accumulate and optimize the email pattern library and improve the ability to identify spam. By manually marking and updating the hash table, the misjudgment and missed judgment caused by automatic judgment errors can be reduced.

[0175] According to another aspect of the embodiment of the present application, Figure 3 As shown, a spam filtering device is provided, comprising:

[0176] The mail set building module 301 is used to build corresponding spam mail sets and non-spam mail sets based on spam mails and non-spam mails in the mail system;

[0177] The email word segmentation module 303 is used to segment each email in the spam email set and the non-spam email set based on the bag-of-words model technology to obtain corresponding independent character strings, calculate the word frequencies of the independent character strings in different emails, and calculate the corresponding average word frequencies based on the word frequencies in different emails;

[0178] The email hash table building module 305 is used to build a spam hash table based on the correspondence between each independent string in the spam email set and each email, combined with the word frequency and average word frequency of each independent string in the spam email set, and to build a non-spam hash table based on the correspondence between each independent string in the non-spam email set and the email, combined with the word frequency and average word frequency of each independent string in the non-spam email set;

[0179] New email processing module 307, used to segment new emails based on the bag-of-words model technology to obtain independent character strings corresponding to the new emails and calculate the corresponding word frequencies, establish a temporary hash table based on the independent character strings in the new emails and the corresponding word frequencies, and calculate the word frequencies of each independent character string in the temporary hash table in the spam email hash table and the non-spam email hash table;

[0180] A new email hash table construction module 309 is used to calculate a basic word frequency threshold by combining the word frequency of each independent string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and add the basic word frequency threshold to the temporary hash table;

[0181] The spam judgment module 311 is used to judge whether the corresponding new email is spam based on the relationship between the word frequency of each independent character string in the temporary hash table and the corresponding basic word frequency threshold, update the spam hash table and the non-spam hash table based on the temporary hash table, and judge whether the next new email is spam based on the updated spam hash table and the non-spam hash table.

[0182] It should be noted that the email set construction module 301 in this embodiment can be used to execute step S202 in the embodiment of the present application, the email segmentation module 303 in this embodiment can be used to execute step S204 in the embodiment of the present application, the email hash table construction module 305 in this embodiment can be used to execute step S206 in the embodiment of the present application, the new email processing module 307 in this embodiment can be used to execute step S208 in the embodiment of the present application, the new email hash table construction module 309 in this embodiment can be used to execute step S210 in the embodiment of the present application, and the spam judgment module 311 in this embodiment can be used to execute step S212 in the embodiment of the present application.

[0183] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 402, a processor 403, a communication interface 404 and a communication bus 405. The memory 402 stores a computer program that can be run on the processor 404. The memory 402 and the processor 404 communicate through the communication interface 406 and the communication bus 408. When the processor 404 executes the computer program, the steps of the above method are implemented.

[0184] The electronic device provided in the embodiment of the present application may specifically be a module capable of realizing a communication function or a terminal device including the module, etc. The terminal device may be a mobile terminal or a smart terminal. The mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a laptop computer, etc.; the smart terminal may specifically be a terminal including a wireless communication module such as a smart car, a smart watch, a shared bicycle, a smart cabinet, etc.; the module may specifically be a wireless communication module, such as any one of a 2G communication module, a 3G communication module, a 4G communication module, a 5G communication module, and a NB-IOT communication module.

[0185] The memory and processor in the above electronic device communicate through the communication bus and the communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0186] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0187] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0188] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above embodiments.

[0189] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to execute the following steps:

[0190] S202, establishing corresponding spam and non-spam sets respectively based on spam and non-spam in the mail system;

[0191] S204, segmenting each email in the spam email set and the non-spam email set based on the bag-of-words model technology to obtain corresponding independent character strings, calculating the word frequencies of the independent character strings in different emails, and calculating the corresponding average word frequencies based on the word frequencies in different emails;

[0192] S206, establishing a spam hash table based on the correspondence between each independent string in the spam set and each email, combined with the word frequency and average word frequency of each independent string in the spam set, and establishing a non-spam hash table based on the correspondence between each independent string in the non-spam set and the email, combined with the word frequency and average word frequency of each independent string in the non-spam set;

[0193] S208, segmenting the new email based on the bag-of-words model technology to obtain independent character strings corresponding to the new email and calculating the corresponding word frequencies, establishing a temporary hash table based on the independent character strings in the new email and the corresponding word frequencies, and calculating the word frequencies of each independent character string in the temporary hash table in the spam email hash table and the non-spam email hash table;

[0194] S210, calculating a basic word frequency threshold based on the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and adding the basic word frequency threshold to the temporary hash table;

[0195] S212, judging whether the corresponding new email is spam based on the relationship between the word frequency of each independent character string in the temporary hash table and the corresponding basic word frequency threshold, updating the spam hash table and the non-spam hash table based on the temporary hash table, and judging whether the next new email is spam based on the updated spam hash table and the non-spam hash table.

[0196] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0197] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0198] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application, or a combination thereof.

[0199] For software implementation, the technology of this article can be implemented by a unit that performs the functions of this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0200] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0202] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0205] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a ..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0206] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A spam filtering method, characterized in that: The steps include: Based on the spam and non-spam emails in the email system, corresponding spam and non-spam email sets are established respectively; Based on the bag-of-words model technology, each email in the spam email set and the non-spam email set is segmented to obtain a corresponding independent string, the word frequency of the independent string in different emails is calculated, and the corresponding average word frequency is calculated based on the word frequency in the different emails; A spam hash table is established based on the correspondence between each independent string and each email in the spam email set in combination with the word frequency and average word frequency of each independent string in the spam email set; a non-spam hash table is established based on the correspondence between each independent string and each email in the non-spam email set in combination with the word frequency and average word frequency of each independent string in the non-spam email set; Based on the bag-of-words model technology, the new email is segmented to obtain the independent character strings corresponding to the new email and the corresponding word frequencies are calculated, a temporary hash table is established based on the independent character strings in the new email and the corresponding word frequencies, and the word frequencies of each independent character string in the temporary hash table in the spam hash table and the non-spam hash table are calculated; Calculate a basic word frequency threshold by combining the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and add the basic word frequency threshold to the temporary hash table; Based on the relationship between the word frequency of each independent character string in the temporary hash table and the corresponding basic word frequency threshold, it is determined whether the corresponding new email is spam; based on the temporary hash table, the spam hash table and the non-spam hash table are updated; based on the updated spam hash table and the non-spam hash table, it is determined whether the next new email is spam.

2. The spam filtering method according to claim 1, characterized in that: The method of establishing corresponding spam email sets and non-spam email sets based on spam emails and non-spam emails in the email system respectively includes: Obtain spam data and non-spam data in the existing mail system; Performing data cleaning on the spam data and the non-spam data respectively to obtain spam and non-spam; Marking each of the spam emails to obtain a first mark, and marking each of the non-spam emails to obtain a second mark; A spam email set is generated from each of the spam emails based on the first annotation, and a non-spam email set is generated from each of the non-spam emails based on the second annotation.

3. The spam filtering method according to claim 1, characterized in that: The bag-of-words model technology is used to segment each email in the spam email set and the non-spam email set to obtain a corresponding independent string, calculate the word frequency of the independent string in different emails, and calculate the corresponding average word frequency based on the word frequency in the different emails, including: Preprocessing each email in the spam email set and the non-spam email set respectively; Based on the bag-of-words model technology, each email in the spam email set is segmented to obtain a first independent string, and each email in the non-spam email set is segmented to obtain a second independent string; Calculate the first word frequency of the first independent string in different emails based on the ratio of the number of each first independent string in each email corresponding to the spam email set to the sum of the number of first independent strings in each email; calculate the second word frequency of the second independent string in different emails based on the ratio of the number of each second independent string in each email corresponding to the non-spam email set to the sum of the number of second independent strings in each email; A first average word frequency corresponding to the first independent string is calculated based on the first word frequency corresponding to the first independent string of each email in the spam set, and a second average word frequency corresponding to the second independent string is calculated based on the second word frequency corresponding to the second independent string of each email in the non-spam set.

4. The spam filtering method according to claim 1, characterized in that: The step of establishing a spam hash table based on the correspondence between each independent string in the spam set and each email combined with the word frequency and average word frequency of each independent string in the spam set, and establishing a non-spam hash table based on the correspondence between each independent string in the non-spam set and the email combined with the word frequency and average word frequency of each independent string in the non-spam set includes: Initializing a hash table template, including setting a key map in the hash table template to a first independent string and a second independent string, and setting a value map in the hash table template to a first independent string quantity, a first word frequency, and a first average word frequency in each email in the spam email set, and a second independent string quantity, a second word frequency, and a second average word frequency in each email in the non-spam email set; Filling each first independent character string, first word frequency and corresponding first average word frequency of the spam email set into the hash table template to obtain a spam email hash table; The second independent character strings, the second word frequencies and the corresponding second average word frequencies in the non-spam set are filled into the hash table template to obtain a non-spam hash table.

5. The spam filtering method according to claim 1, characterized in that: The method of segmenting new emails based on the bag-of-words model technology to obtain independent character strings corresponding to the new emails and calculating corresponding word frequencies, establishing a temporary hash table based on the independent character strings in the new emails and the corresponding word frequencies, and calculating the word frequencies of each independent character string in the temporary hash table in the spam hash table and the non-spam hash table includes: Obtain new emails and pre-process the new emails; Extracting the third independent string corresponding to the preprocessed new email based on the bag-of-words model technology, and calculating the third word frequency of the third independent string in the new email based on the ratio of the number of the third independent string in the new email and the sum of the numbers of each third independent string in the new email; Establishing a temporary hash table based on the number of the third independent character string in the new email and the corresponding third word frequency; The fourth word frequency of each of the third independent character strings in the temporary hash table in the spam hash table and the fifth word frequency in the non-spam hash table are calculated respectively.

6. The spam filtering method according to claim 5, characterized in that: The step of calculating a basic word frequency threshold by combining the word frequency of each independent character string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and adding the basic word frequency threshold to the temporary hash table includes: Determine the sum of the fourth word frequency and the fifth word frequency of the third independent character string in the temporary hash table; Calculating a basic word frequency threshold based on a ratio of a fourth word frequency of the third independent character string and a sum of the fourth word frequency and the fifth word frequency; The basic word frequency threshold is added to the temporary hash table to correspond to the third independent character string to update the temporary hash table.

7. The spam filtering method according to claim 6, characterized in that: The method of determining whether the corresponding new email is a spam email based on the relationship between the word frequency of each independent string in the temporary hash table and the corresponding basic word frequency threshold, updating the spam email hash table and the non-spam email hash table based on the temporary hash table, and determining whether the next new email is a spam email based on the updated spam email hash table and the non-spam email hash table, includes: Determine a magnitude relationship between the third word frequency of the third independent character string in the temporary hash table and the corresponding basic word frequency threshold; If the third word frequency is greater than the basic word frequency threshold, the output result is 1, otherwise the output result is 0; Summing all the output results to obtain a final output result, and judging whether the new email is a spam based on the ratio of the final output result to the number of all third independent character strings of the new email, if the ratio is 1, the new email is a spam, otherwise the new email is not a spam; The spam hash table and the non-spam hash table are updated based on the temporary hash table, and whether the next email is a spam is determined based on the updated spam hash table and the non-spam hash table.

8. A spam filtering device, applicable to the spam filtering method according to any one of claims 1 to 7, characterized in that: include: An email set building module, used to build corresponding spam email sets and non-spam email sets based on spam emails and non-spam emails in the email system; The email word segmentation module is used to segment each email in the spam email set and the non-spam email set based on the bag-of-words model technology to obtain the corresponding independent strings, calculate the word frequency of the independent strings in different emails, and calculate the corresponding average word frequency based on the word frequency in different emails; The email hash table construction module is used to build a spam hash table based on the correspondence between each independent string in the spam email set and each email, combined with the word frequency and average word frequency of each independent string in the spam email set, and to build a non-spam hash table based on the correspondence between each independent string in the non-spam email set and the email, combined with the word frequency and average word frequency of each independent string in the non-spam email set; A new mail processing module is used to segment new mails based on the bag-of-words model technology to obtain independent strings corresponding to the new mails and calculate the corresponding word frequencies, establish a temporary hash table based on the independent strings in the new mails and the corresponding word frequencies, and calculate the word frequencies of each independent string in the temporary hash table in the spam mail hash table and the non-spam mail hash table; A new email hash table construction module, used to calculate a basic word frequency threshold by combining the word frequency of each independent string in the temporary hash table in the spam hash table and the word frequency in the non-spam hash table, and add the basic word frequency threshold to the temporary hash table; The spam judgment module is used to judge whether the corresponding new email is spam based on the relationship between the word frequency of each independent string in the temporary hash table and the corresponding basic word frequency threshold, update the spam hash table and the non-spam hash table based on the temporary hash table, and judge whether the next new email is spam based on the updated spam hash table and the non-spam hash table.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the processor implements the instructions of the spam filtering method described in any one of 1-7 above.

10. A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, the computing device executes the spam filtering method according to any one of claims 1 to 7.

Images