System and method for generating heuristic rules for identifying spam emails
By generating statistical data-based heuristic rules, using hash and regular expression combinations to identify spam emails, the problems of high false positive rates and difficult privacy protection in the existing technology are solved, and the accuracy of spam email recognition is improved and computer security.
Patent Information
- Application Number
- CN201911159541.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-17
- Filing Date
- 2019-11-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-11-22
AI Technical Summary
The prior art is difficult to obtain accurate results without analyzing the body of the email when identifying spam emails, and the false positive rate is high, affecting computer security and user experience.
By collecting and analyzing email statistics, heuristic rules are generated to identify spam emails, using hash and regular expression combinations, filtering based on the characteristics of email titles and hyperlinks, reducing direct analysis of email content.
Improves the identification accuracy and computer security of spam emails, reduces the false positive rate, avoids unnecessary access to email content, and protects user privacy.
Smart Images

Figure CN111752973B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer security, and more particularly to a system and method for identifying junk emails. Background Art
[0002] At present, the demand for providing data security and computer security is constantly growing. When a user receives unsolicited or unwanted content, security may be compromised and fraudulent activities may ensue. One way to send unsolicited or unwanted content is to use junk emails, which are a type of electronic content sent by email for the purpose of advertising a specific product or service and for the purpose of implementing fraudulent actions. Today, there are many technologies that can be used to send junk emails. For example, a proxy server, a free email server that enables email to be sent in an automatic manner, a botnet, etc. can be used to send junk emails. In another example, junk emails can be sent in social networks, blogs, forums, dating websites, etc. by means of instant messaging protocols. In another example, junk emails can be distributed using short message service (Short Message Service, SMS) messages or multimedia messaging service (Multimedia Messaging Service, MMS) messages.
[0003] To illustrate the impact of junk email in modern communications, junk email sent by email accounts for 70%-90% of the total email volume. The amount of junk email sent by computer networks is so large that it causes great inconvenience to email users. In particular, junk email reduces network bandwidth, wastes resources of message exchange systems, and increases the processing time of email by users and computers. Therefore, users must constantly fight junk email.
[0004] Various mechanisms have been developed to combat spam emails, such as creating blacklists of spam email senders, creating greylists of senders that need to repeatedly access the mail server to send spam emails, contextual filtering of spam emails, etc. Typically, spam email filtering focuses on analyzing the content of email messages. Filtering-based methods typically require exact matches in order to find emails containing spam information and to distinguish emails containing spam information from legitimate emails. Unfortunately, the analysis of content introduces other problems. First, a large amount of analysis requires a lot of time, which slows down the speed of email processing. Bulk spam email sending further exacerbates the impact on processing time. Second, analysis requires opening all emails to access the information. This type of access is not allowed in most cases unless the user's consent is obtained before sending the email. In addition, in some cases, emails may contain confidential information.
[0005] One way to improve the effectiveness of spam email filtering is to reduce the large-scale analysis or introduce preliminary analysis. The reduction of analysis can be based on the analysis of the email title instead of the analysis of the entire email. For example, the email title and the title characters used to form the title can be analyzed. However, the current technology based on title analysis does not contain an effective mechanism that can quickly analyze and obtain statistically accurate results at the same time. For example, when the probability of the first type of error (i.e., false alarm) is greater than a predetermined threshold, a legitimate email may be classified as a spam email at an unacceptable level. In some systems, legitimate emails classified as spam emails may be isolated or automatically deleted without notifying the sender and / or recipient. Therefore, misclassification may lead to the loss of legitimate emails. One way to reduce false alarms is to use different thresholds that can make spam emails pass through filters. Unfortunately, senders of spam emails are also becoming more and more creative and continue to work hard to consider various known spam email filtering mechanisms. For example, spam emails can avoid being automatically detected by adding randomization or data similar to legitimate emails to the email content.
[0006] Therefore, there is a need to improve the detection scheme, for example, to adopt statistical data analysis of the emails received by users, so as to effectively detect junk emails while maintaining the confidentiality of the email content. That is, it is necessary to identify emails containing junk information without analyzing the text contained in the email body. Summary of the invention
[0007] Various aspects of the present invention relate to the field of computer security, and more particularly to systems and methods for generating heuristic rules for identifying spam emails.
[0008] In one exemplary aspect, a method for generating heuristic rules for identifying spam emails containing unsolicited or unwanted content is implemented in a computer including a hardware processor, the method comprising: collecting statistical data about the content of a plurality of emails; analyzing the collected statistical data to identify different types of content of the emails, the different types of content comprising one or more of a title and a hyperlink of the emails; grouping the plurality of emails into one or more clusters based on the identified different types of content in the emails, wherein at least one cluster comprises one or more groups of fields in the title of the emails; selecting at least one most common group combination of data in each cluster; generating a hash from the at least one most common group combination; forming at least one regular expression based on an analysis of a hyperlink of the email corresponding to the generated hash; and generating at least one heuristic rule for identifying spam emails by combining at least one hash with a corresponding regular expression, wherein at least one hash is derived from a sequence of fields in the title of the email.
[0009] According to one aspect of the present invention, a system for generating heuristic rules for identifying spam emails is provided, the system comprising a hardware processor, the hardware processor being configured to: collect statistical data about the contents of a plurality of emails; analyze the collected statistical data to identify different types of contents of the emails, the different types of contents comprising one or more of titles and hyperlinks of the emails; group the plurality of emails into one or more clusters based on the identified different types of contents in the emails, wherein at least one cluster comprises one or more groups of fields in the titles of the emails; select at least one most common group combination of data in each cluster; generate a hash from the at least one most common group combination; form at least one regular expression based on an analysis of hyperlinks of the emails corresponding to the generated hash; and generate at least one heuristic rule for identifying spam emails by combining at least one hash with a corresponding regular expression, wherein at least one hash is derived from a sequence of fields in the title of the email.
[0010] In one exemplary aspect, a non-transitory computer-readable medium is provided, on which a set of instructions is stored, the instruction set being used to: collect statistical data about the content of a plurality of emails; analyze the collected statistical data to identify different types of content of the emails, the different types of content including one or more of titles and hyperlinks of the emails; group the plurality of emails into one or more clusters based on the identified different types of content in the emails, wherein at least one cluster includes one or more groups of fields in the titles of the emails; select at least one most common group combination of data in each cluster; generate a hash from the at least one most common group combination; form at least one regular expression based on an analysis of hyperlinks of the emails corresponding to the generated hash; and generate at least one heuristic rule for identifying spam emails by combining at least one hash with a corresponding regular expression, wherein at least one hash is derived from a sequence of fields in the title of the email.
[0011] In one aspect, the regular expression is formed by dividing each identified hyperlink into a plurality of segments according to established delimiters; for each hyperlink, calculating the number of segments obtained by the division; comparing hyperlinks having the same number of segments segment by segment; and combining the hyperlinks into a regular expression according to their lengths, wherein if the segments are the same, the segments are kept unchanged; and if the segments are not the same, the segments are converted into a form in which only the same values are retained.
[0012] In one aspect, the method further comprises, prior to generating the heuristic rules, performing, for each hash, a preliminary check on a set of hashes containing only hashes of non-spam emails, wherein the hash is used in generating the heuristic rules only if the preliminary check does not find a match with any hash in the set of hashes.
[0013] In one aspect, the set of hashes includes at least a set of hashes corresponding to legitimate emails and a set of hashes corresponding to emails containing spam.
[0014] In one aspect, the most common group combinations are identified based on at least one of: a threshold established for the number of emails analyzed to obtain the statistical data, or a threshold that exceeds or reaches a threshold for grouping together a predetermined percentage of the emails from which the statistical data is obtained.
[0015] In one aspect, the statistical data is obtained based on lexical analysis of the email.
[0016] In one aspect, the method further comprises: validating the generated heuristic rule on a set of legitimate emails; and sending the heuristic rule to a user when the validated heuristic rule does not match any email.
[0017] In one aspect, the hashing comprises at least MD5.
[0018] According to the teachings of the present invention, the identification of junk emails containing unwanted or unsolicited content improves computer security. The present invention can solve the above-mentioned shortcomings and problems by creating heuristic rules and checking emails for the presence of junk information, wherein the heuristic rules enable the analysis of the sequence of fields of the email header of the email being checked. The improvement is achieved in the following manner: collecting statistical data about the content of a plurality of emails; analyzing the collected statistical data to identify different types of content of the emails, the different types of content including one or more of the header and hyperlink of the email; grouping the plurality of emails into one or more clusters based on the identified different types of content in the emails, wherein at least one cluster includes one or more groups of fields in the header of the email; selecting at least one most common group combination of data in each cluster; generating a hash from the at least one most common group combination; forming at least one regular expression based on the analysis of the hyperlink of the email corresponding to the generated hash; and generating at least one heuristic rule for identifying junk emails by combining at least one hash with a corresponding regular expression, wherein at least one hash is derived from the sequence of fields of the header of the email. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more exemplary aspects of the present invention and, together with the detailed description, serve to explain the principles and implementations of these exemplary aspects.
[0020] Figure 1 A block diagram is shown of an exemplary system for forming heuristic rules for identifying emails that include unwanted content according to aspects of the present invention.
[0021] Figure 2 An example of data acquired from an email of a user is shown, wherein the data corresponds to a title of the email.
[0022] Figure 3 An example of statistical data obtained from a header of an email and converted into a specific form is shown.
[0023] Figure 4A flow chart illustrating an exemplary method of forming heuristic rules for identifying emails that include unwanted content.
[0024] Figure 5 An example of a general purpose computer system is shown upon which aspects of the invention may be implemented. DETAILED DESCRIPTION
[0025] Exemplary aspects are described herein in the context of systems, methods, and computer programs that form heuristic rules for identifying junk emails. One of ordinary skill in the art will recognize that the following description is merely exemplary and is not intended to be limiting in any way. Other aspects will readily suggest themselves to one of ordinary skill in the art who appreciates the advantages of the present invention. Reference will now be made in detail to the implementation of the exemplary aspects as shown in the accompanying drawings. Throughout the drawings and the following description, the same reference numerals will be used to refer to the same or similar items as much as possible.
[0026] In order to clearly present the teachings of the present invention, a number of terms and concepts used to describe various aspects of the present invention are defined herein.
[0027] An email is a piece of mail that is sent electronically and has a defined structure. Therefore, emails are written using special programs that support structure, such as the structure of an email. This structure consists of two parts: the header and the body of the email.
[0028] The header, in turn, contains a series of fields designed to be filled in, each defined by its own name (field of the email header) and a respective value. The value of a field in the series of fields contains a specific type of information. For example, for a field designed to indicate the sender of the email, the name is "from" and the value will be in the form of the sender's email address, such as username@kaspersky.com. Examples of other fields of the email header are as follows:
[0029] Sender - the sender of the email, usually similar to "from";
[0030] To - the name and email address of the recipient, a required field to be filled in;
[0031] Subject - The subject of the email. It is not a required field, but it is expected.
[0032] CarbonCopy (Cc) - the addresses of other subscribers who receive a copy of the message;
[0033] Date - the date and time the message was sent;
[0034] Reply-to - the email address to which replies are sent (this email address can be different from the sender's address);
[0035] Received - Various Internet servers relay the message from the sender to the receiver;
[0036] Subject - the content of the message being sent;
[0037] Content-type - the format used to compose the message being sent and the encoding used to create the email;
[0038] Content-Transfer-Encoding - the method of data transmission (7th bit, 8th bit message, etc.);
[0039] Message-ID - a unique identifier for the message; and
[0040] · X-mailer - an email message sending program.
[0041] Typically, the size of an email must not exceed certain values, such as 500kb or 2Mb, etc. For example, some email servers do not relay emails larger than 1Mb.
[0042] Figure 1 1 is a block diagram illustrating an exemplary system 100 for forming heuristic rules for identifying spam emails according to aspects of the present invention. As used herein, the term "spam email" includes, but is not limited to, unsolicited or unwanted messages sent in bulk to one or more recipients via email.
[0043] The system 100 comprises at least a data processor 110, a hash generator 120, a regular expression generator 130, a rule database 140, and a heuristic rule generator 150. The system 100 forms heuristic rules by acquiring statistical data 160 from a data collection and storage device 170, and processing and analyzing the acquired statistical data 160.
[0044] The hash generator 120 may be an MD5 hash generator that takes a string of arbitrary length and generates a 128-bit hash. Note that an MD5 hash is not encryption, but is designed to be difficult to reverse engineer to identify the hash of a given input of the original string.
[0045] In one aspect, the data collection and storage device 170 comprises a cloud server. Figure 1 , the cloud server collects and stores data about emails 175 of email clients 180A, 180B, ..., 180N. The term "cloud server" refers to, for example, a device / server of the Kaspersky Security Network (KSN) system of the company "Kaspersky Lab AO". The term "client" refers to a user's email client established on the user's computer or the user's (one or more) email servers. For example, in Figure 1 , clients 180A, 180B, ..., 180N are email clients of users. It should be noted that the data collection and storage device 170 does not collect and maintain identifiable data during the collection of data about the client's emails. Therefore, the data collection and storage device 170 can be configured not to collect information related to the user, or to depersonalize the collected information so that the information is no longer identifiable. Depersonalization refers to converting information, for example, with the help of a hash function. Information from the body of the email (e.g., the user's text message) and information from the email header (e.g., the email address) are depersonalized or not collected at all. In order to depersonalize the information, each client 180A, 180B, ..., 180N includes an agent (such as Figure 1 As shown), the agent converts the email before sending it to the data collection and storage device 170.
[0046] Figure 2 An example 200 of data obtained from a user's email is shown, where the data corresponds to the header of the email. The data is converted and sent to the data acquisition and storage device 170. In other words, a portion of the information about an email that has been converted (the header) is sent to the data acquisition and storage device 170. The header of the email contains a series of technical data and fields of the email header (such as the sender (From), recipient (To), carbon copy (CC), subject message ID (Subject Message-ID) and content type (Content-Type) and hidden values of these fields). The storage of information about the collected emails and the sending of information about the collected emails to the system 100 are carried out in the form of statistical data 160. The data acquisition and storage device 170 forms statistical data about each email by another conversion of the data collected from the client. In one aspect, the statistical data 160 is formed based on the conversion of the emails collected from the client with the help of lexical analysis. An example of such a conversion is Figure 3 Shown in.
[0047] Figure 3 An example 300 of statistical data obtained from the title of an email and converted into a specific form (e.g., an anonymous form) is shown. For example, the information corresponding to the title of an email and the different fields of the title are converted into an anonymous form (i.e., a form that does not contain the private or personal information of the client). It should be noted that the body of the email will also be converted. The statistical information about the body of the email only includes information about the Uniform Resource Locator (URL) address (also referred to as a hyperlink) contained in the email, information about various block strategies formed based on the data of the email, and information about the categories assigned to the email. In this regard, "category" refers to the internal email classification created. For example, the classification can include categories such as junk emails, non-junk emails, possible junk emails, possible junk emails with phishing elements, and junk emails with suspicious hyperlinks.
[0048] The data processor 110 receives statistical data 160 about emails and performs a preliminary analysis on the received statistical data 160. The preliminary analysis includes identifying various attributes in each email, where the attributes are at least: a sequence of fields in the email header (e.g., Figure 3 , selected text, hyperlinks from the body of the email, various identifiers (IDs) and hashes (e.g., md5) of the email data, etc. Next, the data processor 110 performs clustering on the received statistical data 160, in which at least one cluster is formed based on the identified data. Each formed cluster contains emails grouped according to a specific data type. These data types (attributes) include at least the fields of the email header and the values of these fields.
[0049] In one aspect, clustering of statistical data 160 is performed based on a data clustering algorithm, Density-Based Spatial Clustering of Applications with Noise (DBSCAN).
[0050] In one aspect, the grouping according to a particular data type of the emails includes forming clusters according to fields contained in the header of each email. For example, suppose the groups for at least one cluster are the following email fields of the email header: "hdr_seq2", "boundary_type", "mailer_name", "msgid_type", "urlsQty", "content_type", "attach". Then, Figure 3 The emails represented in the statistical data form in are assigned to at least one cluster formed based on the following fields: "hdr_seq2", "from", "to", "subject", "date", "content-type", and "x-mailer".
[0051] After forming the clusters, the hash generator 120 selects at least one most common group combination in each formed cluster and converts the found group combination into a hash. The group combination is at least a series of field values from the email header. The most common combination is the field set that brings together the largest number of emails in the analyzed emails. The generated hash should be understood as at least MD5, a hash of the found group combination, or a flexible hash. A threshold value can be specified for the number of emails, and once the threshold value is exceeded, each group combination will be considered the most common. In this case, all group combinations exceeding the threshold value will be further analyzed. For example, the threshold value can be set to 80% of the total number of emails. It should be noted that if no group combination reaches the above threshold value, the group combination closest to the given threshold value will be selected.
[0052] In one aspect, when more than one group set has been formed, the data processor 110 can perform prioritization on the group sets. The prioritization can be used to form heuristic rules for emails that contain spam and are frequently encountered. The prioritization parameters will be, for example, information about the assigned category of the email from which the statistical data 160 was obtained, or any additional information that can be used to assess the importance of the email.
[0053] To illustrate, consider an example of searching and generating hashes based on two emails. Assume that one email (M1) contains a sequence of fields of an email header, the sequence of fields consisting of five fields: F1, F2, F8, F9, and F20, where F corresponds to a specific field of the email header and the number represents the field type in a specific classification. The classification of the fields is beyond the scope of the present invention and is an input parameter. For example, F1 corresponds to the field "from", F2 corresponds to the field "to", F8 corresponds to the field "x-mailer", F9 corresponds to the field "cc", and F20 corresponds to the field "message-id". Another email (M2) contains the following fields: F1, F3, F5, F8, F9, F15, and F20. The hash is unique for each email and is formed based on each sequence. For example, for the sequence F1F2F8F9F20 of email M1, a hash md5(from:to:x-mailer:message-id:cc) is formed which would have the following representation: “c72c4c829a3863d1056634d3a306871f”.
[0054] In one aspect, when clustering N (e.g., tens of thousands) of emails, a hash of other hashes can be formed, which in turn are formed from the most common groups in order to capture the maximum number of emails analyzed. For example, 10,000 emails are analyzed, and 2,000 of them are grouped into Group 1 based on the following fields of the email's header: F1:A1, F2:A2, F8:A8, F11:A11 (i.e., these headers are encountered in all 2,000 emails); and 7,500 emails are grouped into Group 2 based on the following fields of the email's header: F1:A1, F5:A5, F9:A9. Thus, for Group 1, a hash "Hash 1" is formed, and for Group 2, a hash "Hash 2" is formed. Thereafter, another hash is formed by combining the two hashes created in this way, where the indicated hash is a flexible hash.
[0055] On the other hand, when no single group set of clusters reaches a given threshold, multiple sets can be combined to exceed the given threshold. For example, two or three hashes will be combined by an "or" symbol. In this case, such a combination will also further correspond to the attribute "hash" in the text.
[0056] Next, the hash generator 120 performs a preliminary check on each selected hash in the set of hashes belonging to emails that do not contain spam (i.e., legitimate emails). It should be noted that this preliminary check can be one of the main steps in forming a heuristic rule, and can be an optional step or an additional step. The set of such hashes is stored in the database 140. If the selected hash matches a hash from the mentioned set, the selected hash is discarded in the process of forming the heuristic rule later. Otherwise, if the selected hash does not match any hash from the mentioned set, the hash will be sent to the heuristic rule generator 150, where the hash will be one of the mandatory conditions, and the hash generator 120 will notify the regular expression generator 130 of the sending of the data. However, if the preliminary check is not performed, each hash is considered to not match any hash from the mentioned set.
[0057] In one aspect, rules database 140 contains at least a set of hashes corresponding to legitimate emails, a set of hashes corresponding to emails containing spam, and a set of flexible hashes for both types of emails.
[0058] Then, the regular expression generator 130 forms at least one regular expression based on the analysis of the hyperlinks contained in the emails corresponding to each hash sent to the heuristic rule generator 150. In order to form the regular expression, data containing information about the hyperlinks is identified based on the statistical data obtained from each of the above-mentioned emails. The data containing information about the hyperlinks includes hyperlinks, wherein each hyperlink (e.g., URL) is a unique identifier of a resource located in an information network (such as the Internet). Another condition for generating the heuristic rule is a regular expression, which is a string that specifies a rule for finding the indicated link.
[0059] In one aspect, the regular expression is formed as follows:
[0060] In the first step of forming a regular expression, the regular expression generator 130 divides each link identified from the statistical data into segments according to the separators contained in the link. The separators include the following symbols: slash " / ", dot "." and symbol "@". Next, the regular expression generator 130 determines the length of each link by counting the number of segments obtained after such division (i.e., after applying the separator). For example, for the hyperlink http: / / app.ingos.ru / email / road_rules / ? utm_source=newsletter&utm_medium=email&utm_campaign=digest_02_2019_feb&utm_content=road_rules&email=username@kaspersky.com, it can be divided into eight segments. Among them, the first segment is "app", the second segment is "ingos", the third segment is "ru", the fourth segment is "email", the fifth segment is "road_rules", the sixth segment is "?utm_source=newsletter&utm_medium=email&utm_campaign=digest_02_2019_feb&utm_content=road_rules&email=username", the seventh segment is "kaspersky" and the eighth segment is "com". Therefore, for this link, the length is equal to eight.
[0061] In the second step of forming a regular expression, the regular expression generator 130 compares the hyperlinks of the same length from different emails with each other segment by segment to form a regular expression corresponding to a hyperlink of a given length. It should be noted that these emails correspond to the same cluster from which the hash generator 120 previously formed a hash. The segments are compared according to the order of the segments, that is, when comparing hyperlinks of the same length, the first segment of the first link from one email is compared with the first segment of the first link in another email, and so on. If the segments match, the symbol set of the regular expression of the segment will completely correspond to the segment in an unchanged form. Otherwise, if the segments are not the same, the segment of the regular expression will have the form of [x] +, where x is the range of symbols contained in both segments. Therefore, when comparing hyperlinks of the same length of different emails, the result obtained is N regular expressions.
[0062] In the third step of forming regular expressions, the regular expression generator 130 compares the regular expressions with each other. The comparison is performed in the same manner as in the previous step, i.e., only regular expressions corresponding to hyperlinks of the same length are compared, while following the principle that "the more general the regular expression, the higher the priority of the regular expression". Thus, less general regular expressions are excluded from further analysis. Thus, a list is formed, each of which contains a regular expression for a hyperlink of one link length.
[0063] After forming the list of regular expressions, the regular expression generator 130 sends the list of regular expressions to the heuristic rule generator 150 .
[0064] The heuristic rule generator 150 generates heuristic rules based on the data received from the hash generator 120 and the regular expression generator 130. The heuristic rule generator 150 determines the correspondence between each hash obtained from the hash generator 120 and each regular expression obtained from the regular expression generator 130.
[0065] After determining the correspondence between each hash obtained from the hash generator 120 and each regular expression obtained from the regular expression generator 130, the heuristic rule generator 150 combines them into the following types of heuristic rules:
[0066] “
[0067] {Bind2(Functor1Equal,GetFullHeaderSeqMd5,'25219f817f0470dc40b578ee99a45cfd')},
[0068] {BodyMatch,where='urls',regexp=[[\w+\.win / [az]+\.? $]]}
[0069] ”,
[0070] Therein, each rule contains at least two specified mandatory conditions, a first condition corresponding to a hash of a sequence of fields of a header of an e-mail and a second condition corresponding to a regular expression corresponding to a hyperlink of a defined length.
[0071] In one aspect, the heuristic rule generator 150 forms additional conditions in addition to the conditions indicated when creating the heuristic rule. The additional conditions are also formed based on the statistical data 160, while allowing the main conditions that have been created. Thus, for example, one of the additional conditions can be created based on an analysis of at least one field of the title contained in the sequence of fields of the title of the email, wherein a hash of the first condition for the heuristic rule is formed based on the sequence of fields of the title of the email. Another additional condition can be created based on an analysis of data (such as hyperlinks) obtained from the body of the email. The formation of each additional condition is similar to the formation of the first condition, which is the hash. To this end, the heuristic rule generator 150 interacts with the data processor 110. Thus, a type of field of the title of the email is selected, wherein for the field of the type, grouping is performed according to clusters based on the values indicated in the corresponding fields of the email being analyzed. For example, if an email contains the field "Content-Type" or the field "X-mailer", grouping can be performed based on analysis of the values of the various fields indicated in the email being analyzed to form additional conditions for evaluating whether spam exists in the email when creating a heuristic rule (an example is given below).
[0072] In another aspect of forming additional conditions, the body of the emails may be analyzed. For example, the analysis may include determining the number of hyperlinks contained in the body of each email. Thus, the maximum and minimum number of hyperlinks in the emails being analyzed are counted. These emails correspond to the clusters from which the hash for the mandatory condition is formed. Additional conditions are formed for the heuristic rule based on the counts. Thus, the formation of additional conditions allows minimizing the possibility of false alarms during the satisfaction of the generated heuristic rule.
[0073] In a preferred aspect, the selection of attributes for forming the additional conditions is done such that information about the attributes is contained in all emails corresponding to the hash.Let us assume that the additional conditions for the heuristic rules have been formed based on the attributes from the example indicated above.
[0074] Then, the heuristic rule can have the following presentation:
[0075] “
[0076] {Bind2(Functor1Equal,GetFullHeaderSeqMd5,'25219f817f0470dc40b578ee99a45cfd')},
[0077] {Bind2(Functor1Equal,UrlsCountS,'1')},
[0078] {Bind2(Functor1Equal,GetBoundaryType,'no_boundary')},
[0079] {Bind2(Functor1Equal,GetMailerType,'none')},
[0080] {BodyMatch,where='urls',regexp=[[\w+\.win / [az]+\.? $]]},
[0081] ”,
[0082] Among them, the first condition indicates that the hash of the sequence of email header fields should correspond to the indicated Md5, the second condition indicates that the email should contain a hyperlink, the third condition indicates that the "Content-type" field lacks a boundary, the fourth condition indicates that the email lacks an "X-mailer" field, and the fifth condition indicates that the hyperlink contained in the email falls within the indicated regular expression.
[0083] Thus, the heuristic rule generator 150 forms a rule for each previously created regular expression. After creating the heuristic rule, the heuristic rule generator 150 can additionally perform validation of the heuristic rule on additional sets of legitimate emails. If the heuristic rule is not triggered on the validated set, the heuristic rule is sent to the client 180A, 180B, ..., 180N. Otherwise, if the heuristic rule is triggered, the heuristic rule is discarded.
[0084] In yet another aspect, the heuristic rule generator 150 evaluates the created rules to identify at least one heuristic rule that covers the maximum number of emails in the cluster. The evaluation is based on comparing the created rules with each other by:
[0085] - (i) if the rules are described by the same regular expression, the conditions of the rules are combined into a single rule using the "OR" symbol; and
[0086] -(ii) If the rules are described by different regular expressions, but other conditions match, then down to one, then the most preferred regular expression will be determined, where the most preferred regular expression will be the regular expression that meets the following criteria: covers N% of the emails in the cluster and the more stringent form of the regular expression, the size of the coverage being the most important criterion. For example, if a rule with a more stringent regular expression describes less than 90% of the emails compared to another rule, then that rule will be less preferred than the other rule, and therefore the rule with the less stringent regular expression will be selected.
[0087] Therefore, the less preferred rules are discarded and the more preferred rules are compared with the remaining rules, resulting in at least one heuristic rule remaining that most completely describes the larger portion of the cluster used to form the hash in the rule by the hash generator 120, as well as the rule itself. Thereafter, rule validation can be performed on a collection of legitimate emails.
[0088] It should be noted that hash generator 120 and regular expression generator 130 will exchange information as needed during the performance of their functions, for example, sending data regarding hashes prepared for a cluster and sending data regarding information used to form a regular expression.
[0089] In one aspect, the system 100 is part of the data collection and storage device 170 , and thus, the system 100 will perform analysis statistics and formation of heuristic rules within the device 170 .
[0090] On the other hand, the system 100 will automatically identify events that indicate the need to create heuristic rules for identifying spam emails being sent. Such events are at least events that indicate the collection of a certain amount of statistical data 160, events that indicate false alarms of heuristic rules on the devices of the clients 180A, 180B, ..., 180N, events that indicate the need to convert one of the previously created clusters, etc. For this reason, the system 100 itself or through the data acquisition and storage device 170 monitors whether such events exist from the devices of the clients 180A, 180B, ..., 180N. Then, the system 100 either requests the data needed for creating the heuristic rules, or obtains the data in parallel with the identified events.
[0091] Figure 4 Flowchart of an exemplary method 400 for forming a heuristic rule for identifying emails including unwanted content (e.g., junk emails) is shown. The method 400 is implemented with the aid of the system 100. Assume that the system 100 has received an event indicating the need to create a heuristic rule for searching and identifying junk emails. Then, as described below, the method 400 is implemented to form the heuristic rule.
[0092] In step 410, method 400 uses data processor 110 to collect statistical data about the content of multiple emails from data collection and storage device 170. Statistical data 160 is presented in a specific (predefined) form. For example, the statistical data can be presented in an anonymous form to contain information about the emails, that is, in such a way that information about the client (e.g., client 180A, 180B, ..., 180N) and information about the client cannot be determined. Figure 3 As described above, an example of such statistics for an email is shown in Figure 1. Data containing information about email headers is obtained, and some of the data is depersonalized and presented in hashed form.
[0093] In step 420 , the method 400 analyzes the collected statistical data by the data processor 110 to identify different types of content of the email, where the different types of content include one or more of a title and a hyperlink of the email.
[0094] In step 430, method 400, by data processor 110, groups the plurality of emails into one or more clusters based on the different types of the content identified in the emails, wherein at least one cluster includes one or more groups of fields in the headers of the emails. In one aspect, at least one cluster is formed, wherein each cluster contains emails grouped by data type. In one aspect, one of the data types used to form the clusters is the data type used for the field corresponding to the header of each email.
[0095] In one aspect, when one of the data types is a data type for a field corresponding to a title of each email, in step 430, the method further performs an analysis on the acquired statistical data, wherein the analysis includes at least identifying a sequence of fields of the title of each email. Then, the method further forms clusters according to the fields using at least one sequence of the identified fields of the title of each email. The clusters are formed using the most preferred data clustering algorithm. In one aspect, the clusters are selected by trial and error (such as using the algorithm DBSCAN). The above combined Figure 1 An example of the formation of clusters is described.
[0096] In step 435, method 400 selects at least one most common group combination of data in each cluster via hash generator 120. The combination of groups is at least a series of most common fields of the headers of the emails being analyzed. The most common combination is the set of fields that brings together the largest number of emails among those being analyzed. The hash is at least MD5, or a flexible hash of the group combination found. In one aspect, a threshold specified for the total number of graphs being analyzed can be used as a basis for determining the most common group combination.
[0097] In step 440, method 400 generates a hash from the at least one most common group combination.
[0098] In step 450, the method 400 uses the regular expression generator 130 to form at least one regular expression based on an analysis of the hyperlinks of the email corresponding to the generated hash. For example, the analysis may include an indication of whether the email contains unwanted content. Figure 1 The principles for forming regular expressions have been described.
[0099] In step 460, the method 400 generates at least one heuristic rule for identifying emails containing unwanted content by combining at least one hash with a corresponding regular expression using the heuristic rule generator 150, wherein the at least one hash is derived from a sequence of fields in the header of the email. Each heuristic rule is generated by combining at least one hash with a regular expression corresponding to the hash (the regular expression has been previously formed). The correspondence is determined by means of statistics used to form the hash and the regular expression. Therefore, one heuristic rule should be formed for each regular expression previously created. Then, the generated heuristic rules are evaluated to identify at least one heuristic rule that brings together the maximum number of emails corresponding to the cluster for which the heuristic rule is formed. The evaluation is performed by comparing the generated heuristic rules with each other. Therefore, one heuristic rule will be retained, which most completely describes the cluster. The method then proceeds to step 470 to end the process.
[0100] In one aspect, the method further comprises, for each hash, performing a preliminary check on a hash set containing only hashes of legitimate emails (i.e., non-spam emails), whereby if there is a match between the formed hash and a hash from the hash set, the formed hash is removed from further processing and is not used to form the heuristic rule. If the formed hash does not match any hash from the hash set, the method continues to generate the heuristic rule.
[0101] In one aspect, the generated heuristic rules may also contain additional conditions that are also formed based on the acquired statistical data 160, considering the primary conditions that have been created.
[0102] In one aspect, for a set of legitimate emails, the generated heuristic rules are validated and if the generated heuristic rules do not match any emails, the heuristic rules are sent to the clients 180A, 180B, ..., 180N via the system 100 .
[0103] In one aspect, the method further comprises, prior to generating the heuristic rules, performing, for each hash, a preliminary check on a set of hashes containing only hashes for non-spam emails, wherein the hash is used in generating the heuristic rules only if the preliminary check does not find a match with any hash in the set of hashes.
[0104] In one aspect, each heuristic rule is generated by combining at least one hash with a regular expression to which the hash corresponds (the regular expression having been previously formed).
[0105] In one aspect, each cluster contains emails grouped by data type, wherein one of the data types used to form at least one cluster is a data type of a field of a header to which each email corresponds.
[0106] In one aspect, the most common combination corresponds to the set of fields that brings together the largest number of emails out of all emails analyzed.
[0107] In one aspect, the most common combinations are identified based on at least one of: a threshold established for the number of emails analyzed to obtain statistics, or exceeding or reaching a threshold for grouping together a predetermined percentage of the emails from which statistics are obtained.
[0108] In one aspect, the statistical data is obtained based on lexical analysis of the emails.
[0109] In one aspect, the regular expression is formed based on analyzing the hyperlinks in the following manner: dividing each identified hyperlink into multiple segments according to established delimiters; for each hyperlink, calculating the number of segments obtained by the division; comparing hyperlinks having the same number of segments segment by segment; and combining the hyperlinks into the regular expression according to the length of the hyperlink, wherein if the segments are the same, the segments remain unchanged; if the segments are different, the segments are converted into a form in which only the same values are retained.
[0110] In one aspect, the hash comprises at least an MD5 hash or a flexible hash of the most common group combinations.
[0111] Figure 5 A block diagram of a computer system 20 is shown according to an exemplary aspect on which aspects of the system and method for forming rules for identifying emails including unwanted content may be implemented. It should be noted that the computer system 20 may correspond to, for example, the detection device described previously. The computer system 20 may be in the form of multiple computing devices, or in the form of a single computing device, such as a desktop computer, a laptop computer, a handheld computer, a mobile computing device, a smart phone, a tablet computer, a server, a mainframe, an embedded device, and other forms of computing devices.
[0112] As shown, computer system 20 includes a central processing unit (CPU) 21, a system memory 22, and a system bus 23 connecting various system components, including a memory associated with the central processing unit 21. The system bus 23 may include a bus memory or a bus memory controller, a peripheral bus, and a local bus capable of interacting with any other bus architecture. Examples of buses may include PCI, ISA, PCI-Express, HyperTransport, and the like. TM (HyperTransport TM ), unlimited bandwidth TM (InfiniBand TM ), Serial ATA, I2C, and other suitable interconnections. The central processing unit 21 (also referred to as a processor) may include a single group or multiple groups of processors having a single core or multiple cores. The processor 21 may execute one or more computer executable codes that implement the technology of the present invention. The system memory 22 may be any memory for storing data used herein and / or computer programs executable by the processor 21. The system memory 22 may include volatile memory (such as random access memory (Random Access Memory, RAM) 25) and non-volatile memory (such as read-only memory (Read-Only Memory, ROM) 24, flash memory, etc.) or any combination thereof. The basic input / output system (Basic Input / Output System, BIOS) 26 may store basic programs for transmitting information between elements of the computer system 20, such as those basic programs when the operating system is loaded using ROM 24.
[0113] The computer system 20 may include one or more storage devices, such as one or more removable storage devices 27, one or more non-removable storage devices 28, or a combination thereof. The one or more removable storage devices 27 and the one or more non-removable storage devices 28 are connected to the system bus 23 via a memory interface 32. In one aspect, the storage devices and corresponding computer-readable storage media are power-independent modules for storing computer instructions, data structures, program modules, and other data for the computer system 20. The system memory 22, the removable storage devices 27, and the non-removable storage devices 28 can use a variety of computer-readable storage media. Examples of computer-readable storage media include: machine memory, such as cache, SRAM, DRAM, zero-capacitor RAM, two-transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other storage technology, such as in a solid state drive (SSD) or a flash drive; magnetic cassettes, tapes, and disk storage, such as in a hard drive or floppy disk; optical storage, such as in a compact disk (CD-ROM) or a digital versatile disk (DVD); and any other medium that can be used to store the desired data and can be accessed by the computer system 20.
[0114] The system memory 22, removable storage device 27 and non-removable storage device 28 of computer system 20 can be used to store operating system 35, additional program applications 37, other program modules 38 and program data 39. Computer system 20 may include a peripheral interface 46 for transmitting data from input device 40, such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices, such as a printer or scanner via one or more I / O ports, such as a serial port, parallel port, Universal Serial Bus (USB), or other peripheral interfaces. Display device 47 (such as one or more monitors, projectors or integrated displays) can also be connected to system bus 23 via output interface 48 (such as a video adapter). In addition to display device 47, computer system 20 can also be equipped with other peripheral output devices (not shown), such as speakers and other audio-visual devices.
[0115] The computer system 20 can work in a network environment using a network connection with one or more remote computers 49. The one or more remote computers 49 can be local computer workstations or servers, which include most or all of the elements described above when describing the properties of the computer system 20. Other devices can also be present in the computer network, such as but not limited to routers, websites, peer devices or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with the remote computers 49 via one or more networks, such as a local area computer network (LAN) 50, a wide area computer network (WAN), an intranet, and the Internet. Examples of the network interface 51 may include an Ethernet interface, a frame relay interface, a synchronous optical network (SONET) interface, and a wireless interface.
[0116] Various aspects of the present invention may be systems, methods and / or computer program products. The computer program products may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform various aspects of the present invention.
[0117] Computer readable storage medium can be a tangible device, and this tangible device can keep and store the program code of the form of instruction or data structure, and this program code can be accessed by the processor of computing device (such as computing system 20).Computer readable storage medium can be electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or its any suitable combination.By way of example, this kind of computer readable storage medium can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), flash memory, hard disk, portable computer disk, memory stick, floppy disk or even mechanical coding device, such as punch card or convex structure in the groove of recording instruction thereon.As used in this article, computer readable storage medium should not be regarded as temporary signal itself, temporary signal such as radio wave or other free propagation electromagnetic wave, electromagnetic wave propagated by waveguide or transmission medium or electric signal transmitted by wire.
[0118] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network interface in each computing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing device.
[0119] The computer-readable program instructions for performing the operation of the present invention can be assembler instructions, instruction set architecture (Instruction-Set-Architecture, ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages (including object-oriented programming languages and traditional programmed programming languages).Computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, partly on the user's computer and partly on a remote computer or completely on a remote computer or server as an independent software package.In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including LAN or WAN), or can be connected to an external computer (for example, through the Internet).In some embodiments, electronic circuit (including for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (Programmable Logic Array, PLA)) can perform computer-readable program instructions by utilizing the state information of computer-readable program instructions to personalize the electronic circuit, thereby performing various aspects of the present invention.
[0120] In various aspects, the systems and methods described in the present invention can be processed in modules. The term "module" as used herein refers to, for example, a real-world device, component, or arrangement of components implemented using hardware (e.g., via an application-specific integrated circuit (ASIC) or FPGA), or to a combination of hardware and software, such as a microprocessor system and an instruction set that implements the module's functionality (which, when executed, converts the microprocessor system into a dedicated device). A module can also be implemented as a combination of two modules, in which certain functions are facilitated by hardware alone and other functions are facilitated by a combination of hardware and software. In some implementations, at least a portion of a module (and in some cases, all of a module) can be implemented in a computer system (e.g., as described above in Figure 5Therefore, each module can be implemented in various suitable configurations and should not be limited to any specific implementation illustrated in this article.
[0121] For the sake of clarity, not all routine features of various aspects are disclosed herein. It should be appreciated that in the development of any practical implementation of the present invention, many specific implementation decisions must be made in order to achieve the developer's specific goals, and these specific goals will vary for different implementations and different developers. It should be understood that such development efforts can be complex and time-consuming, but are still routine tasks of engineering for those of ordinary skill in the art who understand the advantages of the present invention.
[0122] In addition, it should be understood that the words or terms used herein are for the purpose of description rather than limitation, so the terms or terms of this specification should be interpreted by those skilled in the art according to the teachings and guidance provided herein in combination with the knowledge of (one or more) relevant technical personnel. In addition, it is not intended to attribute any term in this specification or claims to an uncommon or special meaning unless explicitly stated as such.
[0123] The various aspects disclosed herein include the present and future known equivalents of the known modules cited herein in an illustrative manner. In addition, although various aspects and applications have been shown and described, it will be apparent to those skilled in the art who understand the advantages of the present invention that more modifications than those mentioned above are feasible without departing from the inventive concept disclosed herein.
Claims
1. A method for generating heuristic rules for identifying spam emails, the method include: collect statistics about the content of multiple emails through the processor; analyzing, by the processor, the collected statistical data to identify different types of content of the email, the different types of content comprising one or more of a title and a hyperlink of the email; grouping, by the processor, the plurality of emails into one or more clusters based on the identified different types of the content in the emails, wherein at least one cluster includes one or more groups of fields in headers of the emails; selecting, by the processor, at least one most common group combination of data in each cluster; generating, by the processor, a hash from the selected at least one most common group combination by converting the at least one most common group combination into a hash; forming, by the processor, at least one regular expression based on analysis of a hyperlink of an email corresponding to the generated hash; and At least one heuristic rule for identifying spam emails is generated by the processor by combining at least one hash with a corresponding regular expression, wherein the at least one hash is derived from a sequence of fields of a header of the email.
2. The method according to claim 1, in, The regular expression is formed as follows: dividing each identified hyperlink into a plurality of segments according to the established delimiters; For each hyperlink, calculating the number of segments obtained by the division; Hyperlinks with the same number of segments are compared segment by segment; as well as combining the hyperlinks into a regular expression according to the length of the hyperlinks, wherein if segments are the same, the segments are kept unchanged; If the segments are not identical, the segment is converted to a form that retains only identical values.
3. The method according to claim 1, further comprising: include: Prior to generating the heuristic rules, for each hash, a preliminary check is performed on a set of hashes containing only hashes of non-spam emails, wherein only if the preliminary check does not find a match with any hash in the set of hashes is the hash used in generating the heuristic rules.
4. The method according to claim 3, in, The hash set at least includes a set of hashes corresponding to legitimate emails and a set of hashes corresponding to emails containing spam information.
5. The method according to claim 1, in, The most common group combination is identified based on at least one of: a threshold established for a number of emails analyzed to obtain the statistical data, or exceeding or reaching a threshold for grouping together a predetermined percentage of the emails from which the statistical data is obtained.
6. The method according to claim 1, in, The statistical data is obtained based on lexical analysis of the email.
7. The method according to claim 1, further comprising: include: validating the generated heuristic rules on a collection of legitimate emails; as well as When the verified heuristic rule does not match any email, the heuristic rule is sent to the user.
8. The method according to claim 1, in, The hashing comprises at least MD5.
9. A system for generating heuristic rules for identifying spam emails, include: at least one processor, the at least one processor being configured to: collect statistics about the content of multiple emails; analyzing the collected statistical data to identify different types of content of the emails, the different types of content comprising one or more of a title and a hyperlink of the emails; grouping the plurality of emails into one or more clusters based on the identified different types of the content in the emails, wherein at least one cluster includes one or more groups of fields in headers of the emails; selecting at least one most common group combination of data in each cluster; generating a hash from the selected at least one most common group combination by converting the at least one most common group combination into a hash; forming at least one regular expression based on analysis of a hyperlink of an email corresponding to the generated hash; and At least one heuristic rule for identifying spam emails is generated by combining at least one hash derived from a sequence of fields of a header of the email with a corresponding regular expression.
10. The system according to claim 9, in, The configuration for forming the regular expression includes configuration for: dividing each identified hyperlink into a plurality of segments according to the established delimiters; For each hyperlink, calculating the number of segments obtained by the division; Hyperlinks with the same number of segments are compared segment by segment; as well as combining the hyperlinks into a regular expression according to the length of the hyperlinks, wherein if segments are the same, the segments are kept unchanged; If the segments are not identical, the segment is converted to a form that retains only identical values.
11. The system according to claim 9, in, The processor is further configured to: Prior to generating the heuristic rules, for each hash, a preliminary check is performed on a set of hashes containing only hashes of non-spam emails, wherein only if the preliminary check does not find a match with any hash in the set of hashes is the hash used in generating the heuristic rules.
12. The system according to claim 11, in, The hash set at least includes a set of hashes corresponding to legitimate emails and a set of hashes corresponding to emails containing spam information.
13. The system according to claim 9, in, The most common group combination is identified based on at least one of: a threshold established for a number of emails analyzed to obtain the statistical data, or exceeding or reaching a threshold for grouping together a predetermined percentage of the emails from which the statistical data is obtained.
14. The system according to claim 9, in, The statistical data is obtained based on lexical analysis of the email.
15. The system according to claim 9, in, The processor is further configured to: validating the generated heuristic rules on a collection of legitimate emails; and When the verified heuristic rule does not match any email, the heuristic rule is sent to the user.
16. The system according to claim 9, in, The hashing comprises at least MD5.
17. A non-transitory computer-readable medium having stored thereon computer-executable instructions for generating heuristic rules for identifying spam emails, the computer-executable instructions comprising instructions for: collect statistics about the content of multiple emails; analyzing the collected statistical data to identify different types of content of the emails, the different types of content comprising one or more of a title and a hyperlink of the emails; The plurality of emails are grouped into one or more clusters based on the different types of the contents in the identified emails, wherein: at least one cluster comprising one or more groups of fields in a header of the email; selecting at least one most common group combination of data in each cluster; generating a hash from the selected at least one most common group combination by converting the at least one most common group combination into a hash; forming at least one regular expression based on analysis of a hyperlink of an email corresponding to the generated hash; as well as At least one heuristic rule for identifying spam emails is generated by combining at least one hash derived from a sequence of fields of a header of the email with a corresponding regular expression.
18. The non-transitory computer readable medium of claim 17, in, The instructions for forming the regular expression include instructions for: dividing each identified hyperlink into a plurality of segments according to the established delimiters; For each hyperlink, calculating the number of segments obtained by the division; Hyperlinks with the same number of segments are compared segment by segment; as well as combining the hyperlinks into a regular expression according to the length of the hyperlinks, wherein if segments are the same, the segments are kept unchanged; If the segments are not identical, the segment is converted to a form that retains only identical values.
19. The non-transitory computer-readable medium of claim 17, the computer-executable instructions further comprising instructions for: Before generating the heuristic rules, for each hash, a preliminary check is performed on a set of hashes containing only hashes of non-spam emails, in, Only if the preliminary check does not find a match with any hash in the set of hashes is that hash used in generating the heuristic rules.
20. The non-transitory computer readable medium of claim 19, in, The hash set at least includes a set of hashes corresponding to legitimate emails and a set of hashes corresponding to emails containing spam information.
