A spam filtering method based on feature selection
Automatically filtering of emails through feature selection technology solves the inefficiency problem of manually identifying spam in traditional technology, realizes efficient spam filtering, and improves the security of email processing.
Patent Information
- Application Number
- CN202311791685.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-25
AI Technical Summary
Traditional technology requires manual identification of spam, which leads to wasting time and energy. At the same time, fraud and false information in spam cause losses to user property and rights.
Using feature selection-based spam filtering method, the text information of sample email is obtained for preprocessing, converted into sample feature vectors, and feature selection is performed using bag-of-word model and gain analysis, and the email classification model is constructed to automatically identify and filter spam. .
Improves the efficiency of email processing, saves time and energy, reduces the adverse impact of spam on users, and enhances the security of emails.
Smart Images

Figure CN117474510B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text processing, and in particular to a junk mail filtering method based on feature selection. Background Art
[0002] With the rapid development of network technology, e-mail is widely used in people's life and work, and has become an important tool for people's daily communication and information transmission. It provides functions such as sending, receiving, storing, and searching emails, bringing great convenience to people's work and life.
[0003] However, there are currently unscrupulous businesses that use technical means to randomly distribute advertising emails, and network hackers maliciously spread virus files through emails, causing great trouble to users' work and life. The above advertising emails and virus emails are all classified as spam. In traditional technologies, users need to manually judge and delete spam emails one by one when they receive them, which wastes time and energy and reduces the efficiency of email processing. At the same time, the fraud and false information in spam emails are also very likely to cause losses to users' property and rights.
[0004] Therefore, a spam filtering method based on feature selection is proposed. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a spam filtering method based on feature selection, which is used to solve the problem of inconvenience in manually identifying spam in traditional technologies.
[0006] An embodiment of the present invention provides a spam filtering method based on feature selection, comprising:
[0007] Get sample emails;
[0008] Preprocessing the text information in the sample email to obtain preprocessed text information;
[0009] Converting the preprocessed text information into corresponding sample feature vectors through a bag-of-words model;
[0010] Perform feature selection on the sample feature vector based on gain analysis and simplified processing to obtain the optimized sample feature vector;
[0011] Constructing an email classification model according to the optimized sample feature vector;
[0012] Get pending mails;
[0013] Inputting the to-be-processed email into the email classification model to obtain corresponding classification results;
[0014] According to the classification results, spam emails are filtered out and normal emails are retained.
[0015] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the step of preprocessing the text information in the sample email to obtain the preprocessed text information comprises:
[0016] Obtaining text information in the sample email;
[0017] Identify the language type in the text information, perform corresponding word segmentation processing according to the language type, and build a feature phrase library; including:
[0018] When the language type is Chinese, the text information is segmented using the JIEBA segmentation database to obtain Chinese feature phrases, which are stored in the feature phrase database;
[0019] When the language type is English, the text information is segmented by spaces in the text information to obtain English feature phrases, which are stored in the feature phrase library;
[0020] Meaningless information in the feature phrase library is filtered out to obtain preprocessed text information.
[0021] Preferably, the present invention provides a junk mail filtering method based on feature selection, wherein the meaningless information includes: filtering out function words, punctuation marks and low-frequency phrases in the feature phrase library.
[0022] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the step of converting the preprocessed text information into a corresponding sample feature vector through a bag-of-words model comprises:
[0023] Calculate the weight of each phrase in the preprocessed text information, and set i The weight corresponding to the phrase is α i The preprocessed text information contains n phrases;
[0024] ;
[0025] in, f i For the i The frequency of occurrence of a phrase in the text processing information, M is the number of all phrases in the text library, m i For the i The number of phrases in the text base;
[0026] According to the weight corresponding to each phrase, construct the sample feature vector T =[ α1 , …, α n ].
[0027] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the step of performing feature selection on a sample feature vector based on gain analysis and simplified processing to obtain an optimized sample feature vector comprises:
[0028] Performing gain analysis based on the preprocessed text information, performing feature selection on the feature vectors in the sample feature vectors, and obtaining an initial optimized feature vector;
[0029] The initial optimized feature vector is simplified several times to construct a standard feature vector set, and the trial probabilities of all feature vectors in the standard feature vector set are subjected to threshold probability judgment to obtain an optimized sample feature vector.
[0030] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the steps of performing gain analysis based on preprocessed text information, performing feature selection on feature vectors in the sample feature vectors, and obtaining an initial optimized feature vector include:
[0031] Calculate the gain value of each phrase in the preprocessed text information, assuming i The gain value of the phrase is g i ;
[0032] ;
[0033] in, P ( May j ) is a sample email in j The probability of appearing in a category, F i For the i The probability of a phrase appearing in a sample email, For the i The probability that a phrase does not appear in the sample emails, For the i The phrase in j The probability of appearing in a category, For the i The phrase in j The probability of not appearing in a category;
[0034] Setting a gain threshold, obtaining a phrase that does not meet the gain threshold in the text processing information, filtering a feature vector corresponding to the phrase from a sample feature vector, and obtaining an initial optimized feature vector; including:
[0035] Wheni The gain value of the phrase g i is less than the gain threshold Tr g ,Will α i From the sample feature vector T Filter out and obtain the initial optimized feature vector T’ =[ α 1 ’ ,…, α n1 ’ ].
[0036] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the steps of: performing several simplification processes on the initial optimized feature vector, constructing a standard feature vector set, and performing threshold probability judgment processing on the trial probabilities of all feature vectors in the standard feature vector set to obtain an optimized sample feature vector include:
[0037] The initial optimized feature vector is t Simplify the process and get the corresponding t A subset of standard eigenvectors;
[0038] Will t A subset of standard feature vectors is constructed as a standard feature vector set;
[0039] Obtain the test probability of all feature vectors in the standard feature vector set and construct a vector probability set; suppose the first feature vector in the standard feature vector set i The experimental probability of the feature vector p i ;
[0040] ;
[0041] in, The eigenvector in the initial optimized eigenvector conduct t The number of times the simplified process occurs;
[0042] The test probability of all feature vectors in the vector probability set is judged by threshold probability. When the threshold probability condition is not met, the corresponding feature vector in the standard feature vector set is removed to construct an optimized sample feature vector. T’’ =[ α 1 ’’ ,…, α n2 ’’ ].
[0043] Preferably, the present invention provides a spam filtering method based on feature selection, the steps of: t Simplify the process and get the corresponding t A subset of standard feature vectors; these include:
[0044] By using the granular sphere neighborhood rough set algorithm, according to the initial optimized feature vector T’ Construct vector space;
[0045] Randomly select any eigenvector in the vector space as a starting particle;
[0046] Obtaining a certain characteristic vector in the vector space that satisfies a similarity condition with the initial particle, constructing a growth vector subspace, and calculating a vector mean of the growth vector subspace;
[0047] According to the vector mean of the growth vector subspace, obtaining other eigenvectors in the vector space that meet the similarity condition with the growth vector subspace, adding the eigenvectors to the growth vector subspace, and recalculating the vector mean of the growth vector subspace;
[0048] Until all eigenvectors in the vector space are traversed and calculated;
[0049] Remove the eigenvectors in the vector space that are not added to the growing vector subspace, and construct an optimized vector space according to the eigenvectors in the growing vector subspace;
[0050] Analyzing the properties of the optimized vector space, when a certain feature vector in the optimized vector space is removed and the properties of the optimized vector space do not change, filtering out the feature vector; when a certain feature vector in the optimized vector space is removed and the properties of the optimized vector space change, retaining the feature vector in the standard feature vector subset;
[0051] Until all the eigenvectors in the optimized vector space are traversed, a standard eigenvector subset is obtained.
[0052] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the step of constructing an email classification model according to the optimized sample feature vector comprises:
[0053] Obtaining the optimized sample feature vector and classification result set;
[0054] According to the sample emails, the learning parameters corresponding to the classification results are calculated to build an email classification model;
[0055] Assume the number of sample emails is K The classification result isC S , calculate the average degree of attachment, average degree of non-attachment and average degree of boundary of sample emails;
[0056] ;
[0057] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average level of attachment, A ik For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S degree of attachment;
[0058] ;
[0059] in, l ik To optimize the feature vector in the sample feature vector α i ’’ In the k Sample emails May k The number of occurrences in is the feature vector α i ’’ The mean , is the feature vector α i ’’ The standard deviation of
[0060] ;
[0061] in, D S ( May k ) is the k Sample emails May k Falling into classification results C S The judgment value of
[0062] ;
[0063] ;
[0064] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average degree of non-dependence, E ik For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The degree of non-dependence;
[0065] ;
[0066] ;
[0067] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average boundary degree of
[0068] The learning parameters SP , including the average degree of attachment, average degree of non-attachment and average degree of boundary of sample emails, .
[0069] Preferably, the present invention provides a spam filtering method based on feature selection, wherein the step of inputting the to-be-processed email into the email classification model to obtain the corresponding classification result comprises:
[0070] Count pending mail newmail the average degree of attachment, average degree of non-attachment, and average degree of boundary;
[0071] Based on pending mail newmailThe average level of attachment , average non-dependence and the average boundary degree Analysis and learning parameters SP The degree of homogeneity of C newmail ;
[0072] ;
[0073] in, sim [] is the homogeneity function, argmax [] is a set classification function.
[0074] Compared with the traditional technology, the beneficial effects of the present invention are as follows: the text information in the sample email is preprocessed and converted into a sample feature vector; in the actual processing process, the converted sample feature vector is high-dimensional data with high redundancy, and the redundant vectors in the sample feature vector are filtered out by performing feature selection on the sample feature vector, thereby improving the classification performance of the constructed email classification model;
[0075] The above method is applied to the user's email processing process. The email to be processed is identified through the email classification model, which solves the defect of traditional technology that requires manual identification of spam. While saving time and energy, it improves the efficiency of email processing and filters out the identified spam, avoiding the adverse effects of fraud and false information in spam on users, and further improving the security of users' use of email.
[0076] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0077] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] Figure 1 The present invention provides a flow chart of a spam filtering method based on feature selection. DETAILED DESCRIPTION
[0079] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0080] Embodiment 1:
[0081] The embodiment of the present invention provides a spam filtering method based on feature selection, referring to Figure 1 ,include:
[0082] Get sample emails;
[0083] Preprocessing the text information in the sample email to obtain preprocessed text information;
[0084] The preprocessed text information is converted into corresponding sample feature vectors through the bag-of-words model;
[0085] Perform feature selection on the sample feature vector based on gain analysis and simplified processing to obtain the optimized sample feature vector;
[0086] Build an email classification model based on the optimized sample feature vector;
[0087] Get pending mails;
[0088] Input the emails to be processed into the email classification model to obtain the corresponding classification results;
[0089] Filter out spam based on the classification results and keep normal emails.
[0090] In the above embodiments, the text information in the sample emails is preprocessed, and the preprocessed text information is converted into corresponding sample feature vectors using a bag-of-words model; feature selection is performed on the sample feature vectors based on gain analysis and simplification processing to obtain optimized sample feature vectors and build an email classification model; after the email classification model receives the email to be processed, it performs classification processing to obtain the classification results, and filters out spam based on the classification results.
[0091] The beneficial effects of the above technology are: by preprocessing the text information in the sample emails, the text information is converted into a sample feature vector; in the actual processing process, the converted sample feature vector is high-dimensional data with high redundancy. By performing feature selection on the sample feature vector, the redundant vectors in the sample feature vector are filtered out, thereby improving the classification performance of the constructed email classification model;
[0092] The above method is applied to the user's email processing process. The email to be processed is identified through the email classification model, which solves the defect of traditional technology that requires manual identification of spam. While saving time and energy, it improves the efficiency of email processing and filters out the identified spam, avoiding the adverse effects of fraud and false information in spam on users, and further improving the security of users' use of email.
[0093] Embodiment 2:
[0094] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of preprocessing text information in a sample email to obtain preprocessed text information include:
[0095] Get the text information in the sample email;
[0096] Identify the language type in the text information, perform corresponding word segmentation processing according to the language type, and build a feature phrase library; including:
[0097] When the language type is Chinese, the text information is segmented through the JIEBA word segmentation database to obtain Chinese feature phrases and store them in the feature phrase database;
[0098] When the language type is English, the text information is segmented by spaces in the text information to obtain English feature phrases, which are stored in a feature phrase library;
[0099] Filter out meaningless information in the feature phrase library and obtain preprocessed text information.
[0100] In the above embodiment, corresponding word segmentation processing is performed according to the language type of the text information in the sample email, a feature phrase library is constructed, and meaningless information in the feature phrase library is removed to obtain pre-processed text information.
[0101] In the above embodiment, when the language type of the text information in the sample email is Chinese, the text information is segmented using the JIEBA word segmentation database to obtain Chinese feature phrases.
[0102] In the above embodiment, when the language type of the text information in the sample email is English, the text information is segmented according to the spaces in the text information to obtain English feature phrases.
[0103] The beneficial effects of the above technology are: performing word segmentation processing on the text information in the sample email and filtering out meaningless information, realizing preprocessing of the text information, and obtaining preprocessed text information.
[0104] Embodiment 3:
[0105] The embodiment of the present invention provides a junk mail filtering method based on feature selection, meaningless information, including: filtering out function words, punctuation marks and low-frequency phrases in a feature phrase library.
[0106] In the above embodiments, the text information in the sample email is preprocessed by using the function words, punctuation marks and low-frequency phrases contained in the meaningless information.
[0107] Embodiment 4:
[0108] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: converting preprocessed text information into corresponding sample feature vectors through a bag-of-words model, including:
[0109] Calculate the weight of each phrase in the preprocessed text information, and set i The weight corresponding to the phrase is α i , there are a total of preprocessed text information n phrases;
[0110] ;
[0111] in, f i For the i The frequency of occurrence of the phrase in the text processing information, M is the number of all phrases in the text library, m i For the i The number of phrases in the text base;
[0112] According to the weight corresponding to each phrase, construct the sample feature vector T =[ α 1 , …, α n ].
[0113] In the above embodiments, the weight of each word group in the pre-processed text information is calculated to obtain a sample feature vector, thereby achieving vectorized expression of the pre-processed text information.
[0114] Embodiment 5:
[0115] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: performing feature selection on a sample feature vector based on gain analysis and simplified processing to obtain an optimized sample feature vector, including:
[0116] Perform gain analysis based on preprocessed text information, perform feature selection on feature vectors in sample feature vectors, and obtain an initial optimized feature vector;
[0117] The initial optimized feature vector is simplified several times to construct a standard feature vector set, and the test probabilities of all feature vectors in the standard feature vector set are subjected to threshold probability judgment to obtain the optimized sample feature vector.
[0118] In the above embodiments, gain analysis is performed based on preprocessed text information, feature vectors with less information in the sample feature vectors are filtered out, and an initial optimized feature vector is obtained; the initial optimized feature vector is simplified, feature vectors with low similarity and consistent attributes are filtered out, a standard feature vector set is constructed, and a threshold probability judgment is performed on the trial probabilities of all feature vectors in the standard feature vector set, feature vectors with no advantage in classification performance are filtered out, and an optimized sample feature vector is obtained.
[0119] The beneficial effects of the above technology are: through gain analysis of preprocessed text information, feature selection is performed on the feature vectors in the sample feature vector, and the information expression of the feature vector in the initial optimized feature vector is enhanced; the initial optimized feature vector is simplified, and the threshold probability judgment is performed on the experimental probability of the obtained standard feature vector set, so as to improve the classification performance of the feature vector in the optimized sample feature vector, thereby providing reliable samples for email classification model training.
[0120] Embodiment 6:
[0121] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: performing gain analysis according to preprocessed text information, performing feature selection on feature vectors in sample feature vectors, and obtaining an initial optimized feature vector, including:
[0122] Calculate the gain value of each phrase in the preprocessed text information, assuming i The gain value of a phrase is g i ;
[0123] ;
[0124] in, P ( May j ) is a sample email in j The probability of appearing in a category, F i For the i The probability of a phrase appearing in a sample email, For the i The probability that a phrase does not appear in the sample emails, For the i The phrase in j The probability of appearing in a category, For the i The phrase in j The probability of not appearing in a category;
[0125] Setting a gain threshold, obtaining phrases that do not meet the gain threshold in the text processing information, filtering out the feature vectors corresponding to the phrases in the sample feature vectors, and obtaining an initial optimized feature vector; including:
[0126] When i The gain value of the phrase g i Less than gain threshold Tr g ,Will α i From the sample feature vector T Filter out and obtain the initial optimized feature vector T’ =[ α 1 ’ ,…, α n1 ’ ].
[0127] In the above embodiment, the gain value of each phrase in the preprocessed text information is calculated. When the gain value of the phrase does not meet the gain threshold, the feature vector corresponding to the phrase is filtered out in the sample feature vector to obtain the initial optimized feature vector.
[0128] The beneficial effects of the above technology are: by calculating the gain value of each phrase in the preprocessed text information, the information contained in the phrase is measured, and the higher the gain value, the more important the feature is; and by setting the gain threshold, the feature vector containing less information is filtered out to obtain the initial optimized feature vector after feature selection.
[0129] Embodiment 7:
[0130] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: performing several simplification processes on the initial optimized feature vector, constructing a standard feature vector set, and performing threshold probability judgment process on the trial probabilities of all feature vectors in the standard feature vector set to obtain an optimized sample feature vector, including:
[0131] The initial optimized feature vector t Simplify the process and get the corresponding t A subset of standard eigenvectors;
[0132] Will t A subset of standard feature vectors is constructed as a standard feature vector set;
[0133] Get the test probability of all feature vectors in the standard feature vector set and construct a vector probability set; let the first feature vector in the standard feature vector set i The experimental probability of the feature vector p i ;
[0134] ;
[0135] in, The eigenvector in the initial optimized eigenvector conduct t The number of times the simplified process occurs;
[0136] The test probability of all feature vectors in the vector probability set is judged by the threshold probability. When the threshold probability condition is not met, the corresponding feature vector in the standard feature vector set is removed to construct the optimized sample feature vector. T’’ =[ α 1 ’’ ,…, α n2 ’’ ].
[0137] In the above embodiment, the initial optimized feature vector is t Simplify the process and get the corresponding t A standard subset of eigenvectors, t The standard feature vector subsets are constructed into a standard feature vector set; the test probability of all feature vectors in the standard feature vector set is calculated, and the threshold probability judgment is performed. When the threshold probability condition is not met, the corresponding feature vector is eliminated to construct an optimized sample feature vector.
[0138] The beneficial effect of the above technology is that: since the simplification process has a certain randomness, the initial optimized feature vector is simplified multiple times based on the law of large numbers, a standard feature vector subset is obtained, a standard feature vector set is constructed, and judgment is made by setting a threshold probability. When the test probability of the feature vector does not meet the threshold probability condition, it is judged that it has no advantage in classification performance, and it is filtered out to construct an optimized sample feature vector with classification performance.
[0139] Embodiment 8:
[0140] The embodiment of the present invention provides a spam filtering method based on feature selection, comprising the steps of: performing t Simplify the process and get the corresponding t A subset of standard feature vectors; these include:
[0141] Through the granular sphere neighborhood rough set algorithm, according to the initial optimized feature vector T’ Construct vector space;
[0142] Randomly select any eigenvector in the vector space as the starting particle;
[0143] Obtain a certain characteristic vector in the vector space that meets the similarity condition with the initial particle, construct a growth vector subspace, and calculate the vector mean of the growth vector subspace;
[0144] According to the vector mean of the growth vector subspace, other eigenvectors in the vector space that meet the similarity condition with the growth vector subspace are obtained, the eigenvectors are added to the growth vector subspace, and the vector mean of the growth vector subspace is recalculated;
[0145] Until all eigenvectors in the calculation vector space are traversed;
[0146] Remove the eigenvectors in the vector space that are not added to the growth vector subspace, and construct an optimized vector space based on the eigenvectors in the growth vector subspace;
[0147] The properties of the optimized vector space are analyzed. When the properties of the optimized vector space do not change after a certain feature vector in the optimized vector space is removed, the feature vector is filtered out; when the properties of the optimized vector space change after a certain feature vector in the optimized vector space is removed, the feature vector is retained in the standard feature vector subset;
[0148] Until all eigenvectors in the optimized vector space are traversed, a standard eigenvector subset is obtained.
[0149] In the above embodiments, the initial optimized feature vector is simplified, a vector space is constructed based on the initial optimized feature vector based on the granular sphere neighborhood rough set algorithm, and a growing vector subspace is constructed for the feature vectors that meet the similarity condition in the vector space, and the feature vectors in the vector space that are not added to the growing vector subspace are removed to implement the first simplification of the initial optimized feature vector and obtain the optimized vector space; the overall properties of the optimized vector space and the properties of all feature vectors in the optimized vector space are analyzed, the feature vector is moved out of the optimized vector space, and when the properties of the optimized vector space do not change, the feature vector is filtered out to implement the second simplification of the initial optimized feature vector and obtain the standard feature vector subset.
[0150] The beneficial effects of the above technology are: through similarity detection and attribute analysis, two simplification processes of the feature vector in the initial optimized feature vector are achieved, the feature vectors with low similarity and consistent attributes in the initial optimized feature vector are filtered out, and feature selection of the feature vector is achieved, avoiding repeated calculation of redundant vectors in the subsequent email classification model processing.
[0151] Embodiment 9:
[0152] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: constructing an email classification model according to an optimized sample feature vector, including:
[0153] Obtain optimized sample feature vectors and classification result sets;
[0154] Based on the sample emails, calculate the learning parameters of the corresponding classification results and build an email classification model;
[0155] Assume the number of sample emails is K The classification result is C S , calculate the average degree of attachment, average degree of non-attachment and average degree of boundary of sample emails;
[0156] ;
[0157] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average level of attachment, A ik For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S degree of attachment;
[0158] ;
[0159] in, l ik To optimize the feature vector in the sample feature vector α i ’’ In the k Sample emails May k The number of occurrences in is the feature vector α i ’’ The mean , is the feature vector α i ’’ The standard deviation of
[0160] ;
[0161] in, DS ( May k ) is the k Sample emails May k Falling into classification results C S The judgment value of
[0162] ;
[0163] ;
[0164] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average degree of non-dependence, E ik For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The degree of non-dependence;
[0165] ;
[0166] ;
[0167] in, For the k Sample emails May k Middle i feature vector α i ’’ Belongs to the classification results C S The average boundary degree of
[0168] Learning parameters SP , including the average degree of attachment, average degree of non-attachment and average degree of boundary of sample emails, .
[0169] In the above embodiment, the learning parameters of the email classification model are calculated based on the sample emails, the optimized sample feature vectors and the classification result set. The learning parameters include the average attachment degree, the average non-attachment degree and the average boundary degree of the sample emails.
[0170] In the above embodiments, the average degree of attachment, average degree of non-attachment and average degree of boundary of the sample emails are specifically the set of the average degree of attachment, average degree of non-attachment and average degree of boundary of the sample emails to all feature vectors in the optimized sample feature vector.
[0171] The beneficial effect of the above technology is that the above email classification model is based on the three-category decision-making idea, and sets the email classification into normal emails, spam emails and other emails. Other emails are located at the decision boundary, which is very likely to lead to misclassification and accidental deletion. The model does not make judgments on other emails, and the user makes the judgment, which avoids the situation where the model mistakenly identifies normal emails as spam, thereby improving the robustness of the model.
[0172] Embodiment 10:
[0173] The embodiment of the present invention provides a spam filtering method based on feature selection, the steps of: inputting the email to be processed into an email classification model to obtain the corresponding classification result, including:
[0174] Count pending mail newmail the average degree of attachment, average degree of non-attachment, and average degree of boundary;
[0175] Based on pending mail newmail The average level of attachment , average non-dependence and the average boundary degree Analysis and learning parameters SP The degree of homogeneity of C newmail ;
[0176] ;
[0177] in, sim [] is the homogeneity function, argmax [] is a set classification function.
[0178] In the above embodiment, the obtained emails to be processed are input into the email classification model, and the email classification model calculates the average degree of attachment, average degree of non-attachment and average degree of boundary of the emails to be processed, and calculates the degree of homogeneity with the learning parameters obtained by training in the model, and obtains the final classification result of the emails to be processed through the set classification function.
[0179] In one embodiment, the homogeneity function may be implemented as:
[0180] ;
[0181] The beneficial effect of the above technology is that after the email classification model receives the email to be processed, the above technical solution is used to obtain the corresponding classification results of the email to be processed, and specifically realizes the identification of spam and normal emails.
[0182] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A spam filtering method based on feature selection, characterized in that: include: Get sample emails; Preprocessing the text information in the sample email to obtain preprocessed text information; including: Get the text information in the sample email; Identify the language type in the text information, perform corresponding word segmentation processing according to the language type, and build a feature phrase library; including: When the language type is Chinese, the text information is segmented through the JIEBA word segmentation database to obtain Chinese feature phrases and store them in the feature phrase database; When the language type is English, the text information is segmented by spaces in the text information to obtain English feature phrases, which are stored in a feature phrase library; Filter out meaningless information in the feature phrase library to obtain preprocessed text information; filter out function words, punctuation marks and low-frequency phrases in the feature phrase library; The preprocessed text information is converted into a corresponding sample feature vector through a bag-of-words model; including: Calculate the weight of each phrase in the preprocessed text information, and set the weight corresponding to the i-th phrase to be α i , there are n phrases in the preprocessed text information; Among them, f i is the frequency of the i-th phrase in the text processing information, M is the number of all phrases in the text library, and m i is the number of the i-th phrase in the text library; According to the weight corresponding to each phrase, construct the sample feature vector T = [α1,…,α n ]; The sample feature vector is subjected to feature selection based on gain analysis and simplification processing to obtain an optimized sample feature vector; including: Perform gain analysis based on preprocessed text information, perform feature selection on feature vectors in sample feature vectors, and obtain initial optimized feature vectors; including: Calculate the gain value of each phrase in the preprocessed text information, and set the gain value of the i-th phrase to be g i ; Among them, P(mail j ) is the probability of a sample email appearing in the jth category, F i is the probability of the i-th phrase appearing in the sample email, is the probability that the i-th phrase does not appear in the sample email, P(mail j |i) is the probability of the i-th phrase appearing in the j-th category, is the probability that the i-th phrase does not appear in the j-th category; J represents the category set of j; Setting a gain threshold, obtaining phrases that do not meet the gain threshold in the text processing information, filtering out the feature vectors corresponding to the phrases in the sample feature vectors, and obtaining an initial optimized feature vector; including: When the gain value g of the i-th phrase i Less than the gain threshold Tr g , and α i Filter out from the sample feature vector T and obtain the initial optimized feature vector T'=[α1',…,α n1 '] The initial optimized feature vector is simplified several times to construct a standard feature vector set, and the test probability of all feature vectors in the standard feature vector set is judged by threshold probability to obtain the optimized sample feature vector; including: Perform t simplification processes on the initial optimized eigenvector to obtain corresponding t standard eigenvector subsets; Construct t standard feature vector subsets into a standard feature vector set; Get the test probability of all feature vectors in the standard feature vector set and construct a vector probability set; let the test probability p of the i-th feature vector in the standard feature vector set be i ; in, is the eigenvector α' in the initial optimized eigenvector i The number of times t simplification processes occur; The test probability of all feature vectors in the vector probability set is judged by the threshold probability. When the threshold probability condition is not met, the corresponding feature vector in the standard feature vector set is removed to construct the optimized sample feature vector T”=[α1”,…,α n2 ”]; Constructing an email classification model according to the optimized sample feature vector; Get pending mails; Inputting the to-be-processed email into the email classification model to obtain corresponding classification results; including: Obtaining the optimized sample feature vector and classification result set; According to the sample emails, the learning parameters corresponding to the classification results are calculated to construct an email classification model; including: Suppose the number of sample emails is K and the classification result is C S , calculate the average attachment level of sample emails Average non-dependence and the average boundary degree in, mail is the kth sample email k The i-th eigenvector α in i " belongs to classification result C S The average degree of attachment, A ik mail is the kth sample email k The i-th eigenvector α in i " belongs to classification result C S degree of attachment; Among them, l ik To optimize the feature vector α in the sample feature vector i "In the kth sample email k The number of occurrences in is the feature vector α i The mean of " is the eigenvector α i "Standard deviation of Among them, D S (mail k ) is the kth sample email mail k Falling into classification result C S The judgment value of in, mail is the kth sample email k The i-th eigenvector α in i " belongs to classification result C S The average degree of non-attachment, E ik mail is the kth sample email k The i-th eigenvector α in i " belongs to classification result C S The degree of non-dependence; in, mail is the kth sample email k The i-th eigenvector α in i " belongs to classification result C S The average boundary degree of The learning parameter SP includes the average attachment degree of the sample emails Average non-dependence and the average boundary degree Filter out spam based on the classification results and keep normal mails, including: Calculate the average attachment level of the pending mail newmail Average non-dependence and the average boundary degree According to the average degree of attachment of the pending mail newmail Average non-dependence and the average boundary degree Analyze the homogeneity with the learning parameter SP and obtain the classification result C newmail ; Among them, sim[] is the homogeneity function, argmax[] is a set classification function.
2. The spam filtering method based on feature selection according to claim 1, characterized in that: The step of performing t simplification processes on the initial optimized feature vector to obtain corresponding t standard feature vector subsets comprises: Constructing a vector space according to the initial optimized feature vector T' by using a granular sphere neighborhood rough set algorithm; Randomly select any eigenvector in the vector space as a starting particle; Obtaining a certain characteristic vector in the vector space that satisfies a similarity condition with the initial particle, constructing a growth vector subspace, and calculating a vector mean of the growth vector subspace; According to the vector mean of the growth vector subspace, obtaining other eigenvectors in the vector space that meet the similarity condition with the growth vector subspace, adding the eigenvectors to the growth vector subspace, and recalculating the vector mean of the growth vector subspace; Until all eigenvectors in the vector space are traversed and calculated; Remove the eigenvectors in the vector space that are not added to the growing vector subspace, and construct an optimized vector space according to the eigenvectors in the growing vector subspace; Analyzing the properties of the optimized vector space, when a certain feature vector in the optimized vector space is removed and the properties of the optimized vector space do not change, filtering out the feature vector; when a certain feature vector in the optimized vector space is removed and the properties of the optimized vector space change, retaining the feature vector in the standard feature vector subset; Until all the eigenvectors in the optimized vector space are traversed, a standard eigenvector subset is obtained.
Citation Information
Patent Citations
Method for filtering Chinese junk mail based on Logistic regression
CN101227435A
Junk short message text feature selection method oriented to word and combined word
CN107193804A
Junk mail identification method and system
CN111079427A
Bilingual corpus sentence alignment method and device, readable storage medium and computer equipment
CN111259652A
Text error correction method and device, equipment and storage medium
CN113901797A
