Intelligent classification of text-based content

Abstract

A method of classifying text-based content is described herein. For example, a classification system performs operations including receiving text-based content including a plurality of characters, generating a plurality of character category sequences using the plurality of characters and based on a plurality of predefined character categories, calculating a frequency distribution of the plurality of character category sequences, and classifying the plurality of character category sequences based on the frequency distribution. And classifying the text-based content based on the calculated frequency distribution. The classification operation uses a machine learning model that has been trained using multiple examples of text-based content. In response to the classification, the system may take appropriate actions. For example, in response to classifying the text-based content as unsolicited, the system may limit distribution of the text-based content or generate an alert for the text-based content.

Description

[0001] Copyright Notice

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Background Art

[0003] In the age of the Internet and online communications, unsolicited text-based content (e.g., text-based messages) is ubiquitous. Certain types of text-based content (e.g., unsolicited emails, unsolicited SMS text messages, etc.) are unwanted, sometimes fraudulent content that is typically sent in bulk and that the recipient does not request or expect to receive. Unsolicited text-based content is not only a nuisance that can clog the recipient's inbox, but can also be malicious and pose a security threat to the recipient's computing device and / or network. Although various methods have been developed to detect and filter unsolicited text-based content, ample opportunity still exists to improve detection techniques for unsolicited text-based content. Summary of the Invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Certain aspects of the present disclosure relate to one or more computer-readable media having computer-executable instructions encoded thereon that cause one or more processors to perform a method for classifying text-based content. The method may receive text-based content comprising a plurality of characters and generate a plurality of character category sequences for the text-based content using the plurality of characters. Each of the plurality of character category sequences includes a plurality of character category identifiers for a plurality of characters of different character sequences in the text-based content. Each of the plurality of character category identifiers identifies one of a plurality of predefined character categories. The method may calculate a frequency distribution of the plurality of character category sequences and classify the text-based content based on the calculated frequency distribution of the plurality of character category sequences. The frequency distribution represents the relative frequency of occurrence of unique character category sequences within a total number of the plurality of character category sequences. Compared to an input based on the frequency distribution of the character sequences of the text-based content, the dimensionality of the input to the machine learning model is reduced, which may reduce computational complexity and reduce memory utilization of the classification operation without adversely affecting the accuracy of the classification results.

[0006] Certain aspects of the present disclosure also relate to a computing device comprising a memory, one or more hardware processors coupled to the memory, and one or more computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations for classifying text-based content. The operations may include receiving text-based content comprising a plurality of characters, and generating a plurality of character category sequences for the text-based content using the plurality of characters. Each of the plurality of character category sequences includes a plurality of character category identifiers for the plurality of characters, respectively, for different character sequences in the text-based content. Each of the plurality of character category identifiers identifies one of a plurality of predefined character categories. The operations also include calculating a frequency distribution for the plurality of character category sequences, and classifying the text-based content based on the calculated frequency distribution for the plurality of character category sequences. The frequency distribution represents the relative frequency of occurrence of unique character category sequences within a total number of the plurality of character category sequences. The dimensionality of the input to a machine learning model is reduced compared to an input based on the frequency distribution of the character sequences of the text-based content, which can reduce computational complexity and memory utilization of the classification operation without adversely affecting the accuracy of the classification results.

[0007] Certain aspects of the present disclosure also relate to a computer-implemented method for training a machine learning model to classify text-based content. The method includes: receiving multiple examples of text-based content, each example having a known classification; and performing a training operation using the multiple examples of text-based content. For each of the multiple examples of text-based content, the training operation includes generating multiple character category sequences for the example of text-based content using multiple characters of the example of text-based content. Each character category sequence in the multiple character category sequences of the example of text-based content includes multiple character category identifiers for multiple characters of different character sequences in the example of text-based content. Each character category identifier in the multiple character category identifiers identifies one of a plurality of predefined character categories. The training operation also includes calculating a frequency distribution of the multiple character category sequences of the example of text-based content, and training the machine learning model based on the calculated frequency distribution of the example of text-based content and the known classification of the example of text-based content. The frequency distribution represents the relative frequency of occurrence of unique character category sequences in a total number of the multiple character category sequences. The dimensionality of the input to the machine learning model is reduced compared to the input of the frequency distribution of character sequences based on examples of text-based content, which can reduce computational complexity and reduce memory utilization of training operations without adversely affecting the results of the training.

[0008] As described herein, various other features and advantages may be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram depicting an example computing system for intelligent classification of text-based content.

[0010] Figure 2 is a flow chart of an example overall method for intelligent classification of text-based content.

[0011] Figure 3 is a table that gives a partial list of some predefined Unicode categories.

[0012] Figure 4 is example software code for computing the frequency distribution of character class sequences for text-based messages.

[0013] Figure 5 Depicted are example pie charts showing frequency distributions of character category sequences obtained from two different messages.

[0014] Figure 6 is a block diagram of an example system for training a machine learning model for intelligent classification of text-based content.

[0015] Figure 7 is a flowchart of an example overall method for training a machine learning model for intelligent classification of text-based content.

[0016] Figure 8 Schematically depicted is a two-dimensional space defining feature vectors used as input in a machine learning model for intelligent classification of text-based content.

[0017] Figure 9 is a diagram of an example computing system in which some of the described intelligent classification techniques may be implemented.

[0018] Figure 10 is an example mobile device that can be used in conjunction with the intelligent classification techniques described herein.

[0019] Figure 11 is an example cloud-enabled environment that can be used in conjunction with the intelligent classification techniques described in this article. DETAILED DESCRIPTION

[0020] Overview of Detecting Unsolicited Messages

[0021] Unsolicited text-based content is text-based content that the recipient has not requested or expected to receive. This unsolicited text-based content can be sent through various forms of electronic communication, such as email, messaging apps, and social media. Unsolicited text-based content often contains fraudulent or malicious content, such as phishing links, malware, or scamware, and can pose a significant threat to user privacy and security. To combat the distribution of this unsolicited text-based content, various technologies, such as machine learning algorithms, natural language processing, and blacklists, can be used to identify and filter out unsolicited text-based content. For example, machine learning algorithms can analyze large datasets of unsolicited text-based content to identify common patterns and train models to automatically identify and classify unsolicited text-based content. Natural language processing can analyze the linguistic features of text-based content to detect unusual or suspicious patterns. Blacklists can maintain lists of known senders of unsolicited text-based content or malicious IP addresses and block unsolicited text-based content from these sources.

[0022] However, existing techniques for detecting unsolicited text-based content are not foolproof and may have limitations, such as (a) failing to detect new or complex techniques for generating unsolicited text-based content, (b) generating false positives, or (c) incorrectly marking legitimate text-based content as unsolicited. Furthermore, senders of unsolicited text-based content employ various techniques to circumvent detection, such as using random or misspelled words, embedding grammatical errors, using unusual capitalization, and / or replacing letters in the text with special characters or symbols. Such text manipulation methods can make it more difficult to automatically detect unsolicited text-based content.

[0023] Furthermore, short text-based content (e.g., text messages or social media posts) presents particular challenges for detecting unsolicited text-based content due to its limited length and informal language. Unlike longer text-based content (e.g., emails) that may contain more contextual information, short text-based content often relies on context and shared knowledge between the sender and recipient, making it more difficult for automated systems to determine the intent of the text-based content. Additionally, senders of unsolicited text-based content may use techniques such as URL shortening or code obfuscation to hide their true intent or circumvent detection. Furthermore, short text-based content may use slang, abbreviations, or non-standard syntax, making it difficult for existing technologies to distinguish between legitimate and unsolicited text-based content.

[0024] As described below, the intelligent text-based content classification technology described herein can improve the accuracy and efficiency of detecting unsolicited text-based messages by extracting hidden information embedded in the text-based content (e.g., statistical properties of character sequences) and providing such information as input to a machine learning model for detecting unsolicited text-based content. For illustrative purposes, text-based messages (or simply "messages") are used as an example to illustrate the disclosed technology below, but it should be understood that the disclosed technology can be used for the classification of any text-based content.

[0025] Example computing system for intelligent classification of text-based content

[0026] Figure 1 Shown is a block diagram of an example computing system 100 configured for intelligent classification of text-based content (eg, messages) in accordance with the techniques described herein.

[0027] The system 100 includes a message classifier 110 configured to process the incoming message 102. Figure 1 As shown, the message classifier 110 may include an application programming interface (API) 112 , a sequencer 114 , a converter 116 , a frequency analyzer 118 , and a machine learning (ML) model 120 .

[0028] The API 112 may receive the message 102 and optionally perform some pre-processing of the message 102 (eg, verify that the message 102 is sent from a legitimate source, check whether the message 102 contains a virus, etc.).

[0029] The received message may be sent to a sequencer 114 , which is configured to parse the message and generate a corresponding sequence of characters corresponding to the message, as further described below.

[0030] The character sequence for the corresponding message generated by the sequencer 114 may be converted into a corresponding character category sequence for the corresponding message based on a plurality of predefined character categories stored in the category storage 140 .

[0031] Alternatively, as described below, the order of the sequencer and converter can be switched, where the converter converts each character of the message into a series of corresponding categories (with category identifiers) and the sequencer generates the character category sequence of the message by scanning the series of corresponding categories.

[0032] Alternatively, as described below, the operations of the sequencer and converter can be performed simultaneously, where characters of a message are scanned character by character (e.g., from the first character to the last character of the message), corresponding character category identifiers are determined as characters are scanned, and a character category sequence is generated as character category identifiers are determined for a new sequence of characters scanned in the message.

[0033] The frequency analyzer 118 may analyze the converted character category sequences of the corresponding messages. Specifically, for each message, the frequency analyzer 118 may calculate the frequency distribution of the converted character category sequences.

[0034] For each message, the calculated frequency distribution of the converted character category sequences can be fed into the input of the ML model 120, which can classify the message 102 into one of a plurality of message categories or classes. For example, the ML model 120 can classify the message 102 as a normal message or an unsolicited message.

[0035] For each incoming message 102 , a classification result 104 (eg, a normal flag or an unsolicited flag) may be generated as an output of the message classifier 110 (eg, via the API 112 ).

[0036] For each incoming message 102, the message processor 160 can take appropriate action on the message based on the corresponding classification result 104. For example, if the message 102 is classified as a normal message, the message 102 can be routed to the intended recipient. On the other hand, if the message 102 is classified as an unsolicited message, the message processor 160 can block, quarantine, or otherwise restrict the message 102 from being distributed to the intended recipient. In some examples, the message processor 160 can automatically generate an alert or warning message, such as in a header or other text portion attached to the text of the message 102, or in a separate message, indicating that the incoming message 102 is classified as an unsolicited message.

[0037] The ML model 120 can be trained a priori, for example, by a training unit 130. The training unit 130 can train the ML model 120 using a plurality of training examples or messages (not messages 102, but representative of such messages), each of which has a known classification (e.g., a normal message or an unsolicited message). Specifically, for each training message, a plurality of character sequences can be generated (e.g., by a sequencer 114), which can be converted into a plurality of character category sequences (e.g., by a converter 116), based on which a frequency distribution can be calculated (e.g., by a frequency analyzer 118). The calculated frequency distribution and the known classification of each training message can be stored in a training data repository 150, which can be used by the training unit 130 to train the ML model 120.

[0038] In some examples, metadata of incoming messages (e.g., date and time, sender's name and address, subject line, message length, etc.) and classification results 104 (and optionally, some intermediate results, such as frequency distribution, etc.) may also be stored in data log 108.

[0039] In some examples, an administrator can interact with message classifier 110 (e.g., via a user interface) through administrator console 106 to monitor and / or control various aspects of message classification. For example, through administrator console 106, an administrator can view the status of each stage of message classification, configure parameters for message classification, etc. An administrator can also view data log 108 through administrator console 106.

[0040] In some examples, once the classification results 104 of the incoming messages 102 are independently verified, for example, by an administrator, the calculated frequency distribution and the verified classification results 104 may also be saved in the training data repository 150. When a sufficient amount of new data has been added to the training data repository 150, the training unit 130 may retrain or update the ML model 120.

[0041] Although the ML model 120 is shown as including Figure 1 , but it should be understood that in some examples, the ML model 120 can be external to the message classifier 110 and communicate with the message classifier 110.

[0042] In some examples, one or more components of system 100 (e.g., message classifier 110, training unit 130, message processor 160, training data repository 150, etc.) can reside in a cloud environment (e.g., as described further below). Figure 11 In some examples, any component of system 100 can be part of a local computing device or system.

[0043] In practice, the systems and subsystems shown herein (e.g., system 100) may vary in complexity, having additional functionality, more complex components, etc. For example, additional functionality may exist within message classifier 110. Additional components may be included to implement security, redundancy, load balancing, reporting, etc.

[0044] The described computing systems can be networked via wired or wireless network connections (including the Internet). Alternatively, the systems can be connected via intranet connections (e.g., in corporate environments, government environments, etc.) or via buses or other interconnection mechanisms in local computing devices or systems.

[0045] System 100 and any other systems / subsystems described herein can be implemented in conjunction with any hardware components described herein, such as the computing systems described below (e.g., processing units, memory, etc.). In any of the examples herein, messages, character sequences, character class sequences, character classes, frequency distributions, etc. can be stored in one or more computer-readable storage media or computer-readable storage devices. The techniques described herein are general to operating system or hardware details and can be applied in any of a variety of environments to utilize the described features.

[0046] Example Overall Approach for Intelligent Classification of Text-Based Content

[0047] Figure 2 It can be done, for example, by Figure 1 Flowchart of an example overall method 200 for intelligent classification of text-based content performed by the system 100.

[0048] At 210 , text-based content (eg, message 102 ) comprising a plurality of characters may be received (eg, by API 112 of message classifier 110 ).

[0049] A plurality of character class sequences are generated for the text-based content using a plurality of characters and based on a plurality of predefined character classes. For example, at 220, a plurality of character sequences can be generated (e.g., by sequencer 114) for the text-based content using a plurality of characters. Each character sequence has a count of characters that depends on the implementation. For example, the character count of each sequence can be two, three, or some other number of characters. The plurality of character sequences can be generated, for example, by scanning from the first character to the last character of the text-based content. In this way, the plurality of character sequences can be sequentially sorted such that adjacent character sequences in the plurality of character sequences are offset by one character.

[0050] At 230, the generated plurality of character sequences of the text-based content may be converted (e.g., by converter 116) into a plurality of character category sequences of the text-based content. This conversion may be based on a plurality of predefined character categories, which may be stored on a computer-readable medium (e.g., category storage 140). Figure 3 Examples of predefined character categories are described. Alternatively, different predefined character categories can be used. To convert a plurality of character sequences into a plurality of character category sequences, an operation can be performed for each character sequence in the plurality of character sequences. For example, characters in the character sequence can be converted into corresponding character category identifiers in a corresponding character category sequence, where each character category identifier identifies one of the plurality of predefined character categories.

[0051] Alternatively, multiple character class sequences can be generated for text-based content using operations with different timing. For example, a series of character class identifiers can be generated by converting corresponding characters of the text-based content into corresponding character class identifiers, where each character class identifier identifies one of a plurality of predefined character classes. The series of character class identifiers can then be scanned from a first character class identifier to a last character class identifier in the series of character class identifiers to generate a plurality of character class sequences. In this way, the plurality of character class sequences can be sequentially ordered such that adjacent character class sequences in the plurality of character class sequences are offset by one character class identifier.

[0052] Alternatively, as another example of the timing of operations when generating multiple character category sequences, multiple text characters of the text-based content can be scanned character by character (e.g., from the first character to the last character of the text-based content). As each character is scanned, a corresponding character category identifier is determined, where each character category identifier identifies one of a plurality of predefined character categories. For each character sequence in the multiple character sequences of the text-based content (e.g., for each pair of characters, or for each triple of characters, etc.), the character category identifiers of the characters of the character sequence can be used to determine a corresponding character category sequence. In this way, multiple character category sequences can be generated as the characters of the text-based content are scanned, where one character category sequence is generated at a time. Figure 4 An example of operation with this timing is shown for an example embodiment.

[0053] At 240, a frequency distribution of multiple character category sequences for the text-based content may be calculated (e.g., by the frequency analyzer 118). As used herein, the term "frequency distribution" indicates any representation of the pattern of occurrences of possible character category sequences in multiple category sequences for the text-based content. For example, the frequency distribution of the text-based content may be a count of the number of occurrences of possible character category sequences in multiple category sequences for the text-based content, respectively. Or, as another example, the frequency distribution of the text-based content may be a percentage value representing the likelihood of occurrence of possible character category sequences in multiple category sequences for the text-based content, respectively. Or, as another example, the frequency distribution of the text-based content may be any other measure representing the frequency or likelihood of occurrence of possible character category sequences in multiple category sequences for the text-based content, respectively. Depending on the embodiment, the frequency distribution may be exact or approximate, due to rounding or quantization processing.

[0054] Intermediate values ​​used to calculate the frequency distribution of the text-based content can be updated synchronously with the generation of the plurality of character class sequences for the text-based content. For example, even if the final value of the frequency distribution is not calculated until all of the plurality of character class sequences have been generated, a histogram tracking the number of occurrences of different possible character class sequences can still be updated while the plurality of character class sequences are being generated.

[0055] Then, at 250 , the text-based content can be classified (e.g., by the trained ML model 120 ) as normal or unsolicited based on the calculated frequency distribution (e.g., by providing the calculated frequency distribution of the plurality of character category sequences as input to the trained ML model 120 ).

[0056] Optionally, at 260, action may be taken with respect to the text-based content based on the classification results of step 250. For example, in response to classifying the text-based content as unsolicited, distribution of the text-based content may be restricted, or an alert for the text-based content may be generated (e.g., by message processor 160). Alternatively, in response to classifying the text-based content as normal, distribution of the text-based content using conventional procedures may occur.

[0057] Method 200 and any other methods described herein can be performed by computer-executable instructions (e.g., causing a computing system to perform the method) stored in one or more computer-readable media (e.g., storage devices or other tangible media) or stored in one or more computer-readable storage devices. Such methods can be implemented in software, firmware, hardware, or a combination thereof. Such methods can be performed, at least in part, by a computing system (e.g., one or more computing devices).

[0058] The actions shown can be described from alternative perspectives while still implementing these techniques. For example, "receiving" can also be described as "sending" from a different perspective.

[0059] Example character sequences

[0060] As described above, a sequencer (eg, 114 ) may parse text-based content (eg, a message) to generate a plurality of character sequences.

[0061] In some examples, multiple character sequences can be generated based on the text-based content by scanning the text-based content character by character from the first character to the last character, so that the multiple character sequences are sequentially ordered and every two adjacent character sequences are offset by one character. Alternatively, the characters of the text-based content can be scanned in a different order, for example, from the last character to the first character.

[0062] As described herein, each character sequence may have a predetermined length (L). Thus, if the text-based content has N characters (assuming N>L), a total of (N-L+1) character sequences may be generated based on the text-based content. In a specific example, each character sequence may include two characters (L=2). In this case, the number of character sequences generated based on the text-based content is N-1. In other examples, each character sequence may have more than two characters (e.g., three, four, five, six, or more characters). Unless otherwise described, two character sequences are used in the following examples, but it should be understood that the technology disclosed herein may also be used when each character sequence has more than two characters.

[0063] As an example, assume that a message (classified as a normal message) contains the following single sentence enclosed in a pair of double quotes:

[0064] “Let’s meet at 6:30!” This message has nineteen characters including spaces, from which the following eighteen character sequences can be generated (each character sequence is enclosed in a pair of square brackets): [Le], [et], [t’], [s’], [s], [m], [me], [ee], [et], [t], [a], [at], [t], [6], [6:], [:3],

[30] , and [0! ].

[0065] As another example, assume that another message (classified as an unsolicited message) contains the following sentence enclosed in a pair of double quotes: "LET$m33t@6:30". This message has seventeen characters including spaces (double space marks occur), based on which the following sixteen character sequences can be generated (each character sequence is enclosed in a pair of square brackets): [LE], [ET], [T$], [$],

[0066] [], [m], [m3],

[33] , [3t], [t], [@], [@], [6], [6:], [:3], and

[30] .

[0067] The examples in this section show two representative messages, a normal message and an unsolicited message.In practice, text-based content can have any combination of characters.

[0068] The examples in this section are English messages, but the methods described herein can be applied to text-based content in a variety of other languages. In some cases, the term "character" or "text character" refers to any symbol used in a writing system, rather than a series of such symbols for a word. For example, a character can be a letter of the alphabet, a syllabic mark in a syllabic script, a number, a grammatical mark, punctuation or other typographic symbol, or a "special character" such as a mathematical symbol, a currency symbol, and an emoticon. In some cases, the term "character" or "text character" refers to any symbol used in a writing system, but does not include logographic symbols used in ideographic scripts (such as Chinese characters, etc.).

[0069] Example character class sequence

[0070] As described above, each character sequence may be converted to a corresponding character category sequence based on a plurality of predefined character categories (eg, by converter 116). The predefined character categories may depend on the implementation.

[0071] In some examples, the predefined character categories can be Unicode categories. Unicode, maintained by the Unicode Consortium, is a standard for encoding, representing, and processing text in various writing systems. For those writing systems, it provides a unique numerical value (also called a code point) for each character used in writing, which is used to represent the character in digital form. Unicode supports a wide range of scripts, including Latin, Greek, Cyrillic, Arabic, Hebrew, Chinese, Japanese, Korean, and others. It also includes a large number of special characters, such as mathematical symbols, currency symbols, and emojis.

[0072] Unicode categories are a way of categorizing Unicode characters based on their properties. Figure 3 In an example embodiment, a table 300 is shown that lists some Unicode categories. For each Unicode category, table 300 shows the category name (e.g., "uppercase"), the category number or identifier (ID), and a brief description of the category, including the corresponding two-letter Unicode designation (e.g., "Lu"). In an example embodiment, the number of Unicode categories can be 31 or 30, depending on whether "Reserved, noncharacter" and "Reserved, reserved" are treated as two separate Unicode categories or combined into one Unicode category.

[0073] In some examples, the predefined character categories can be different from the existing Unicode categories. For example, a user can

[0074] Unicode categories are combined into one user-defined character category (e.g., a user can combine the Connector Punctuation (Pc) Unicode category and the Dash Punctuation (Pd) Unicode category into one user-defined character category). As another example, a user can arbitrarily define character categories based on need and / or experience. In some examples, the predefined character categories can be different for different alphabets and / or scripts (e.g., Latin, Greek, Cyrillic, Arabic, Hebrew, Chinese, Japanese, Korean, etc.).

[0075] As described herein, converting a plurality of character sequences of a message into a plurality of character class sequences of the message includes converting characters in each character sequence of the message into corresponding character class identifiers in a corresponding character class sequence. Each character class identifier can uniquely identify one of a plurality of predefined character classes. Figure 3 , the character category identifier can be a category name (e.g., the name in the first column of Table 300), a category ID (e.g., a number in the second column of Table 300), a Unicode designated symbol (e.g., a two-letter string listed in the third column of Table 300), or any other identifier that can uniquely identify different character categories.

[0076] Therefore, the number of characters in the character sequence is equal to the number of character class identifiers in the converted character class sequence. For example, assuming that Unicode designators are used as character class identifiers, the character sequence [Le] can be converted to the character class sequence Lu-Ll (indicating an uppercase letter followed by a lowercase letter), the character sequence [$] can be converted to the character class sequence Sc-Zs (indicating a currency symbol followed by a space separator), and so on.

[0077] By converting character sequences into corresponding character class sequences, the number of unique character class sequences is significantly reduced compared to the number of unique character sequences. For example, for a character sequence containing two characters selected from U possible characters, while there may be millions of permutations (UxU) of combined unique characters (hundreds or thousands), the total number of unique character class sequences is limited to MxM, where M is the number of predefined character classes (e.g., if M=30, the total number of unique character class sequences is 900, while if U=1,000, the total number of unique character sequences is 1,000,000).

[0078] Example frequency distribution of character class sequences

[0079] As described above, for each message, a frequency distribution of the converted character class sequences can be calculated (e.g., by the frequency analyzer 118). Specifically, the frequency distribution measures the relative frequency of occurrence of each unique character class sequence among the total number of converted character class sequences. For example, the example message

[0080] "Let's meet at 6:30!" can be converted into a total of eighteen character class sequences. Using Unicode designation symbols as character class identifiers, the character class sequence Lu-LI (indicating an uppercase letter followed by a lowercase letter) occurs only once (i.e., [Le]). Therefore, its corresponding frequency is 1 / 18 = 5.6%. The character class sequence LI-LI (indicating a lowercase letter followed by a lowercase letter) occurs five times (i.e., [et], [me], [ee], [et], and [at]). Therefore, its corresponding frequency is 5 / 18 = 27.8%; and so on.

[0081] As an example, Figure 4 Example software code 400 (written in C#) is shown that can be used to parse a message (input "text") containing multiple text characters, generate multiple character category sequences (each containing two character category identifiers) for the message using the multiple text characters and based on multiple predefined Unicode categories, and calculate a frequency distribution of the multiple character category sequences.

[0082] Figure 5 Two pie charts 510 and 520 are depicted, which respectively show the calculated frequency distribution of character category sequences for the following two example messages: "Let's meet at 6:30!" and "LET$m33t@6:30". For simplicity, each unique character category sequence has an abbreviated sequence number, for example, S1 indicates a lowercase letter followed by a lowercase letter, S2 indicates an uppercase letter followed by a lowercase letter, etc. As shown, the message "Let's meet at 6:30!" has ten different character category sequences (S1-S10), while the message "LET$m33t@6:30" has fourteen different character category sequences (S5-S18).

[0083] Different patterns in the frequency distribution of character class sequences can be used to classify text-based content, for example, to distinguish normal messages from unsolicited messages. Figure 5In the example shown, four character class sequences (S1-S4) are unique to the message "Let's meet at 6:30!" and therefore indicate a normal message (in the context of this simple example). In contrast, eight character class sequences (S11-S18) are unique to the message "LET$m33t@6:30" and therefore indicate an unsolicited message (again, in the context of this simple example). Therefore, assuming that the message "Let's meet at 6:30!" is a normal message and the message "LET$m33t@6:30" is an unsolicited message, some combinations of character class sequences (e.g., S11-S18) can represent a signature pattern of an unsolicited message, or at least indicate that the underlying message has a higher probability of containing language that is characteristic of an unsolicited message.

[0084] It is worth noting that some character class sequences may be common in both normal and unsolicited messages (e.g., Figure 5 In the example shown, S5-S10 appear in two messages.) In addition, some character category sequences, although appearing less frequently in normal messages (e.g., Lu-Lu, which indicates an uppercase letter followed by an uppercase letter), do not necessarily mean that the message is an unsolicited message. Conversely, some character category sequences, although appearing frequently in normal messages (e.g., Ll-Ll, which indicates a lowercase letter followed by a lowercase letter), do not rule out the possibility that the message is an unsolicited message. In some examples, the trained ML model can be used to more accurately classify normal and unsolicited messages based on the frequency distribution of character category sequences, as described further below.

[0085] Example machine learning model for intelligent classification of text-based content

[0086] As described above, an ML model (e.g., 120) can be used to classify a message as a normal message or an unsolicited message based on a calculated frequency distribution of character class sequences. The ML model can be previously trained (e.g., by training unit 130) using multiple examples (e.g., training messages) of text-based content with known normal / unsolicited classifications.

[0087] Figure 6 An example system 600 is depicted for training an ML model 620 for intelligent classification of text-based content. The ML model 620 may be an embodiment of the ML model 120. As shown, stored training data 650 (similar to the data stored in the Figure 1 The training data storage repository 150 is used to train the ML model 620. Figure 2As described for the incoming messages, for each training message, a plurality of character category sequences may be generated, based on which a frequency distribution may be calculated.

[0088] Inputs to the ML model 620 may include feature vectors configured to receive the computed frequency distribution 610 of the training messages, respectively, and outputs of the ML model 620 may be configured to receive the normal / unsolicited indicators 630 of the training messages, respectively.

[0089] In some examples, the feature vector of the ML model 620 may include the calculated frequencies for all unique character category sequences. For example, if M is the number of predefined character categories, the feature vector of the ML model 620 may include the calculated frequencies for M×M unique character category sequences, even if the calculated frequencies for some unique character category sequences are zero.

[0090] In other examples, the feature vector of the ML model 620 may include only the calculated frequencies of a subset of all unique character category sequences that are most influential in classifying a message. Which character category sequences are used in the feature vector of the ML model 620 may be determined by the feature selector 640. Thus, the dimensionality of the feature vector may be reduced, which may increase the speed and efficiency of training the ML model 620 without compromising classification accuracy.

[0091] For example, if it is determined that certain character class sequences never appear in any training messages, the feature selector 640 may remove the calculated frequencies of these character class sequences (which are zero for all training messages) from the feature vector of the ML model 620. In this scenario, the dimension of the feature vector of the ML model 620 may be equal to the count of unique character class sequences that appear at least once in the training messages.

[0092] As another example, if it is determined that certain character category sequences never appear or their appearance is extremely rare (e.g., below a predefined threshold), the calculated frequencies of these character category sequences can be added to generate an aggregate frequency of new character category sequences for each training message. That is, all of these low-frequency character category sequences can be grouped together as "miscellaneous" or "other" character category sequences. In this case, the feature selector 640 can replace the calculated frequencies of these low-frequency character category sequences with the aggregate frequencies representing the "miscellaneous" character category sequences. If the number of these low-frequency character category sequences grouped together is K, the dimension of the feature vector can be reduced by K-1 by replacing the calculated frequencies of these low-frequency character category sequences with the aggregate frequencies representing the "miscellaneous" character category sequences.

[0093] In some examples, the feature vector of the ML model 620 may include additional metadata for the training messages in addition to the calculated frequencies of unique character class sequences. For example, the feature vector may include the character counts for each training message. As another example, the feature vector may include the language of each training message. Or, as another example, the feature vector may include the message type (e.g., email, text message, etc.) of each training message. Other relevant information about the training messages (e.g., date and time, sender's name and / or address, etc.) may also be included in the feature vector.

[0094] Alternatively, ML models 620 may be trained separately for different types of training messages. For example, ML models 620 may be trained separately for messages in different languages. Alternatively, ML models 620 may be trained separately for different message types (e.g., email, text message, etc.). Subsequently, based on the pre-classification of the incoming message (e.g., identifying the language of the incoming message, or identifying the message type of the incoming message), the appropriate trained ML model 620 is used.

[0095] The ML model 620 can be configured to implement a supervised machine learning algorithm to derive or estimate optimal model parameters that minimize a particular objective function or loss function based on the training data. Specifically, the supervised machine learning algorithm can generate a predicted normal / unsolicited classification based on the input frequency distribution 610, and compare the predicted classification with a known normal / unsolicited indicator 630, and the difference between the predicted classification and the true label (known indicator 630) can be used to calculate a loss function. The supervised machine learning algorithm can iteratively adjust the model parameters until the loss function is minimized to an acceptable level or until a stopping criterion is met. Once training is complete, the model parameters can be fixed, and the ML model 620 can be used to make a normal / unsolicited classification based on a new set of frequency distributions of character class sequences of incoming messages.

[0096] In some examples, ML model 620 can include a neural network, such as a deep neural network, having an input layer, one or more hidden layers for which model parameters are calculated, and an output layer (which produces a classification). In some examples, ML model 620 can be configured to implement other classification algorithms, such as decision trees, Bayesian networks, support vector machines, logistic regression, K-nearest neighbors, hidden Markov models, and the like.

[0097] Although in the depicted example, each example of text-based content is classified by ML model 620 as either normal or unsolicited, it should be understood that ML model 620 can be trained to classify text-based content into more than two categories or classes. For example, the unsolicited classification can be divided into two or more subclassifications (e.g., likely unsolicited, likely unsolicited, etc.) based on certain predictions and / or statistical metrics calculated by ML model 620.

[0098] Example overall approach for training a machine learning model for intelligent classification of text-based content

[0099] Figure 7 is a flow chart of an example overall method 700 for training an ML model for intelligent classification of text-based content and may be, for example, Figure 1 The system 100 is executed.

[0100] A plurality of examples of text-based content (eg, training messages) can be received at 710. Each example of text-based content has a known classification (eg, normal message or unsolicited message).

[0101] The training operation is then performed on examples of text-based content. Figure 7 , training operations are shown as being performed simultaneously for all examples (e.g., generating character category sequences for all examples and then computing frequency distributions for all examples). In practice, all training operations may be performed on a given example of text-based content, followed by all training operations for the next example of text-based content, and so on, on an example-by-example basis.

[0102] A plurality of character class sequences may be generated for each example of the text-based content using a plurality of text characters in the example of the text-based content and based on a plurality of predefined character classes. For example, at 720, a plurality of character sequences may be generated (e.g., by sequencer 114) for each example of the text-based content, as described with reference to FIG. Figure 2 Then, at 730, for each example of text-based content, the plurality of character sequences may be converted (e.g., by converter 116) into a plurality of character category sequences based on a plurality of predefined character categories, as described in reference to Figure 2 In some examples, the predefined character categories can be Unicode categories, as described above.

[0103] Alternatively, multiple character class sequences may be generated for each example of text-based content using operations with different timings, as described above with reference to Figure 2 described.

[0104] At 740, a frequency distribution of a plurality of character class sequences for each example of the text-based content may be calculated (eg, by frequency analyzer 118), as described above with reference to Figure 2 As stated.

[0105] At 750, an ML model (e.g., ML model 120 or 620) may be trained based on the calculated frequency distribution and the known classification of each example of the text-based content. For example, the calculated frequency distribution may be used in a training operation as described above with reference to Figure 6 As stated.

[0106] Example methods for dimensionality reduction

[0107] In some examples, after training the ML model, the dimensionality of the feature vector used by the ML model can be further reduced (e.g., to 120 or 620). In other words, of all the unique character category sequences used in training the machine learning model, it is likely that only a subset of them are actually useful for classifying incoming text-based content.

[0108] In some examples, the calculated frequencies of some character class sequences may not have predictive power for the ML model (e.g., the frequencies of certain character class sequences do not characterize whether the underlying message is a normal message or an unsolicited message). In such cases, the frequencies of these character class sequences can be removed from the feature vector, thereby reducing the dimensionality of the feature vector for the ML model.

[0109] On the other hand, even if the calculated frequencies of some character class sequences are very small (indicating that these character class sequences are very rare), such small frequencies may have relatively large predictive power (for example, such rare character class sequences may be indicative of unsolicited messages). In such cases, the frequencies of these character class sequences should be retained in the feature vector.

[0110] Various dimensionality reduction techniques (which use unsupervised learning algorithms) can be used to reduce the number of features (e.g., the frequency of unique character class sequences) while retaining relevant information for classification of text-based content. In some examples, Figure 6 The feature selector 640 can be configured to implement one or more dimensionality reduction algorithms. Example dimensionality reduction algorithms include principal component analysis (PCA), independent component analysis (ICA), linear discriminant analysis (LDA), t-distributed stochastic neighbor embedding (t-SNE), autoencoders, etc.

[0111] Reducing the dimensionality of an ML model’s feature vectors not only reduces computational complexity and improves the efficiency of the ML model, but also reduces the risk of overfitting (e.g., when an ML model becomes too complex and begins to fit the noise in the data from the training samples).

[0112] To ensure that dimensionality reduction of the ML model does not remove important information from the data (which could result in a loss of classification accuracy and / or performance), the implementation of dimensionality reduction can be guided by measuring one or more metrics of the performance of the ML model when classifying training messages. Example performance metrics include explained variance ratio, reconstruction error, classification accuracy, etc.

[0113] For illustration purposes, Figure 8 Schematically depicts a two-dimensional space defining feature vectors used in ML models for intelligent classification of text-based content (for character sequences containing three or more characters, a three-dimensional or higher space can be constructed to define feature vectors). For simplicity, assume that there are ten predefined character categories and each character category sequence has two category identifiers (ranging from 1 to 10). The horizontal axis 810 indicates the first category identifier and the vertical axis 820 indicates the second category identifier. Therefore, the total number of unique character category sequences is 10x10=100, as shown by the intersection of the ten horizontal dashed lines and the ten vertical dashed lines.

[0114] In some examples, the frequencies of all 100 unique character class sequences can be included in the feature vector of the ML model.

[0115] In other examples, as described above, the feature vector may be initially reduced by excluding character class sequences that never appear in the training messages. Figure 8 In the example shown, the remaining character category sequences (occurring at least once in the training message) after removing the zero-frequency character category sequences are shown as twelve hollow circles 840 and eight solid circles 830. Therefore, the dimension of the feature vector can be reduced from 100 to 20.

[0116] In some examples, one or more of the dimensionality reduction techniques described above can be used to reduce the dimensionality of the feature vector. This dimensionality reduction can be performed on the initial full set of 100 character class sequences or on a reduced set of 20 character class sequences (i.e., after initially excluding zero-frequency character class sequences). Figure 8 In the example depicted in , such dimensionality reduction may result in a reduced feature vector that includes frequencies of only eight character category sequences represented by solid circles 830 (i.e., frequencies of character category sequences represented by twelve hollow circles 840 are removed from the feature vector).

[0117] Example Benefits

[0118] Several advantages can be achieved via the techniques described herein.

[0119] One of the technical problems solved by the disclosed technology is the inaccurate classification of text-based content. For example, existing technologies for detecting unsolicited messages generally have difficulty in accurately distinguishing between normal messages and unsolicited messages, especially for short messages. The technical solution disclosed herein includes constructing a character category sequence based on text-based content, and identifying that the character category sequences of text-based content of different classifications (such as normal messages and unsolicited messages) have different sequence patterns, even in short examples of text-based content. The technical effects achieved by such a technical solution include extracting hidden information embedded in text-based content, resulting in improved accuracy of classification of text-based content (for example, more accurate detection of unsolicited messages). Utilizing such classification information, text-based content can be processed more effectively and efficiently by automated processing systems and / or end users.

[0120] Another technical problem solved by the disclosed technology is the high utilization of computing resources for the classification of text-based content. For example, some methods for classifying text-based content use complex natural language processing techniques that are computationally expensive. In addition, some methods for classifying text-based content use complex machine learning models that require large memory spaces in order to process the many possible character sequences that may be present in the text-based content. The technical solution disclosed herein includes converting a character sequence into a character category sequence for the purpose of classification. The technical effects achieved by such a technical solution include significantly reducing the dimensionality of the input data to the ML model (for example, from millions of unique character sequences to hundreds of unique character category sequences) while still capturing the underlying character sequence information, thereby not only making it computationally feasible to perform the classification of text-based content in real time or with minimal processing delay, but also significantly reducing the memory usage and computational complexity of the ML model. Therefore, the disclosed technology can perform classification operations in a more efficient manner.

[0121] The technical solutions disclosed herein also include calculating the frequency distribution of character category sequences (to extract the statistical properties of the underlying character sequences), and using the frequency distribution of character category sequences instead of the frequency distribution of character sequences to train the ML model. The technical effects achieved by such a technical solution include more efficient training of the ML model in terms of speed and use of computing resources (e.g., memory, CPU time, etc.) because the data used to train the ML model can be transformed from a high-dimensional space (defined by the unique character sequences that appear or their relative frequency distribution) to a lower-dimensional space (defined by the unique character category sequences that appear or their relative frequency distribution). In addition, compared to existing technologies that use natural language processing or other text-based classification, the disclosed technology has the technical effect of resulting in more accurate message classification because the frequency distribution of character category sequences can contain more predictive features than the original character sequence.

[0122] In addition, the technical solution disclosed in this article includes reducing the dimension of the feature vector used by the ML model, thereby having the technical effect of further reducing the computational complexity of training (and classification) and reducing the risk of overfitting without compromising the performance (e.g., accuracy) of the classification of text-based content.

[0123] Example computing system

[0124] Figure 9 A generalized example of a suitable computing system 900 is depicted in which the described techniques may be implemented. The computing system 900 is not intended to suggest any limitation as to scope of use or functionality, as these techniques may be implemented in various general-purpose or special-purpose computing systems.

[0125] refer to Figure 9 , the computing system 900 includes one or more processing units 910, 915 and memories 920, 925. Figure 9 , the basic configuration 930 is included within the dashed line. Processing units 910, 915 can execute computer-executable instructions, for example, to implement the methods described in the examples herein. A processing unit can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. A processing unit can also include multiple processors. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 9 A central processing unit 910 and a graphics processing unit or co-processing unit 915 are shown. Tangible memory 920, 925 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two accessible by the processing units. The memories 920, 925 store software 980 that implements one or more of the techniques described herein in the form of computer-executable instructions suitable for execution by the processing units.

[0126] The computing system may have additional features. For example, the computing system 900 includes storage 940, one or more input devices 950, one or more output devices 960, and one or more communication connections 970. An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of the computing system 900. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 900 and coordinates the activities of the components of the computing system 900.

[0127] Tangible storage 940 may be removable or non-removable and include magnetic disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information and that can be accessed within computing system 900. Storage 940 stores instructions for software 980 implementing one or more techniques described herein.

[0128] The input device 950 may be a touch input device such as a keyboard, a mouse, a pen, or a trackball, a voice input device, a scanning device, or another device that provides input to the computing system 900. The output device 960 may be a display, a printer, a speaker, a CD writer, or another device that provides output from the computing system 900.

[0129] Communication connection 970 enables communication with another computing entity via a communication medium. The communication medium transmits information, such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the communication medium may employ electrical, optical, RF, or other carrier waves.

[0130] The techniques may be described in the general context of computer-executable instructions, such as those contained in program modules, executed in a computing system on a target real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as needed. The computer-executable instructions for program modules may be executed within a local or distributed computing system.

[0131] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed and may include any combination of specialized and / or general-purpose hardware and software that implements the functionality described herein.

[0132] For presentation purposes, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms indicate operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0133] Sample mobile device

[0134] Figure 101 is a system diagram depicting an example mobile device 1000 including various optional hardware and software components (shown generally at 1002) in which the described embodiments, techniques, and technologies may be implemented. For ease of illustration, any component 1002 in the mobile device may communicate with any other component, although not all connections are shown. The mobile device may be any of a variety of computing devices (e.g., a cellular phone, a smartphone, a handheld computer, a personal digital assistant (PDA), etc.) and may enable wireless, two-way communication with one or more mobile communication networks 1004 (e.g., cellular, satellite, or other networks).

[0135] The illustrated mobile device 1000 may include a controller or processor 1010 (e.g., a signal processor, microprocessor, ASIC, or other control and processing logic circuit) for performing tasks such as signal encoding, data processing, input / output processing, power control, and / or other functions. An operating system 1012 may control the allocation and use of component 1002 and support one or more applications 1014. The applications may include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) or any other computing applications. A function 1013 for accessing an application store may also be used to obtain and update applications 1014. Applications 1014 may also include applications related to text-based message processing. Specifically, one or more of applications 1014 may be configured to implement the intelligent proactive message detection technology described herein.

[0136] The illustrated mobile device 1000 may include memory 1020. Memory 1020 may include non-removable memory 1022 and / or removable memory 1024. Non-removable memory 1022 may include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. Removable memory 1024 may include flash memory or a subscriber identity module (SIM) card, which is well known in GSM communication systems, or other well-known memory storage technologies, such as "smart cards." Memory 1020 may be used to store data and / or code for running operating system 1012 and applications 1014. Example data may include web pages, text, images, sound files, video data, or other data sets sent to and / or received from one or more network servers or other devices via one or more wired or wireless networks. Memory 1020 may be used to store subscriber identifiers such as an International Mobile Subscriber Identity (IMSI) and device identifiers such as an International Mobile Equipment Identifier (IMEI). Such identifiers may be sent to a network server to identify the user and device.

[0137] The mobile device 1000 may support one or more input devices 1030, such as a touch screen 1032, a microphone 1034, a camera 1036, a physical keyboard 1038, and / or a trackball 1040, and one or more output devices 1050, such as a speaker 1052 and a display 1054. Other possible output devices (not shown) may include piezoelectric or other tactile output devices. Some devices may provide more than one input / output function. For example, the touch screen 1032 and the display 1054 may be combined in a single input / output device.

[0138] The input device 1030 may include a natural user interface (NUI). A NUI is any interface technology that enables a user to interact with a device in a "natural" way, without the artificial constraints imposed by input devices such as a mouse, keyboard, or remote control. Examples of NUI methods include those that rely on voice recognition, touch and stylus recognition, gesture recognition on and near the screen, mid-air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence. Other examples of NUIs include motion gesture detection using accelerometers / gyroscopes, facial recognition, 3D displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide more natural interfaces, as well as technologies for sensing brain activity using electric field sensing electrodes (e.g., EEG and related methods). Thus, in one specific example, the operating system 1012 or the application 1014 may include voice recognition software as part of a voice user interface that allows the user to operate the device 1000 via voice commands. In addition, the device 1000 may include input devices and software that allow user interaction via the user's spatial gestures, such as detecting and interpreting gestures to provide input to a gaming application.

[0139] The wireless modem 1060 can be coupled to an antenna (not shown) and can support two-way communication between the processor 1010 and external devices, as is well known in the art. The modem 1060 is shown generally and can include a cellular modem for communicating with the mobile communication network 1004 and / or other radio-based modems (e.g., Bluetooth 1064 or Wi-Fi 1062). The wireless modem 1060 is typically configured to communicate with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between a mobile device and a public switched telephone network (PSTN).

[0140] The mobile device may also include at least one input / output port 1080, a power source 1082, a satellite navigation system receiver 1084 (e.g., a global positioning system (GPS) receiver), an accelerometer 1086, and / or a physical connector 1090, which may be a USB port, an IEEE 1394 (FireWire) port, and / or an RS-232 port. The components 1002 shown are not required or all-inclusive, as any component may be deleted and other components may be added.

[0141] Sample cloud support environment

[0142] Figure 11 A generalized example of a suitable cloud support environment 1100 in which the described embodiments, techniques, and technologies may be implemented is shown. In the example environment 1100, the cloud 1110 provides various types of services (e.g., computing services). For example, the cloud 1110 may include a collection of computing devices that may be centrally located or distributed, which provide cloud-based services to various types of users and devices connected via a network such as the Internet. The implementation environment 1100 may be used in different ways to complete computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on a local computing device (e.g., connected devices 1130, 1140, 1150), while other tasks (e.g., storage of data to be used in subsequent processing) may be performed in the cloud 1110.

[0143] In example environment 1100, cloud 1110 provides services for connected devices 1130, 1140, and 1150 with various screen capabilities. Connected device 1130 represents a device with a computer screen 1135 (e.g., a medium-sized screen). For example, connected device 1130 may be a personal computer, such as a desktop computer, laptop computer, notebook, netbook, etc. Connected device 1140 represents a device with a mobile device screen 1145 (e.g., a small-sized screen). For example, connected device 1140 may be a mobile phone, smartphone, personal digital assistant, tablet computer, etc. Connected device 1150 represents a device with a large screen 1155. For example, connected device 1150 may be a television screen (e.g., a smart TV) or another device connected to a television (e.g., a set-top box or game console). One or more of connected devices 1130, 1140, and 1150 may include touchscreen capabilities. Touchscreens can accept input in different ways. For example, a capacitive touchscreen detects touch input when an object (e.g., a fingertip or stylus) distorts or interrupts the current flowing across the surface. As another example, a touch screen can use an optical sensor to detect touch input when a light beam from the optical sensor is interrupted. Physical contact with the screen surface is not necessary for input to be detected by some touch screens. Devices without screen capabilities can also be used in example environment 1100. For example, cloud 1110 can provide services for one or more computers (e.g., server computers) that do not have displays.

[0144] Services can be provided by cloud 1110 through service provider 1120 or through other providers of online services (not shown). For example, cloud services can be customized for the screen size, display capabilities, and / or touch screen capabilities of specific connected devices (e.g., connected devices 1130, 1140, 1150).

[0145] In example environment 1100, cloud 1110 provides the techniques and solutions described herein to various connected devices 1130, 1140, 1150, at least in part using service provider 1120. For example, service provider 1120 can provide a centralized solution for various cloud-based services. Service provider 1120 can manage service subscriptions for users and / or devices (e.g., connected devices 1130, 1140, 1150 and / or their respective users).

[0146] Example Implementations

[0147] Although the operations of some disclosed methods are described in a particular order for ease of presentation, it should be understood that this description includes rearrangement unless the specific language set forth below requires a specific ordering. For example, in some cases, operations described sequentially may be rearranged or performed simultaneously. Furthermore, for simplicity, the accompanying figures may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods.

[0148] Any of the disclosed methods may be implemented as computer-executable instructions or computer program products stored on one or more computer-readable storage media and executed on a computing device (i.e., any available computing device, including smartphones or other mobile devices that include computing hardware). A computer-readable storage medium is a tangible medium that can be accessed within a computing environment (one or more optical media disks, such as DVDs or CDs, volatile memory (such as DRAM or SRAM), or non-volatile memory (such as flash memory or hard drives). As an example and with reference to Figure 9 , computer-readable storage media include memories 920 and 925 and storage 940. The term computer-readable storage medium does not include signals and carrier waves (ie, non-transitory). Furthermore, the term computer-readable storage medium does not include communication connections, such as 970.

[0149] Any computer-executable instructions for implementing the disclosed technology and any data created and used during implementation of the disclosed embodiments can be stored on one or more computer-readable storage media. The computer-executable instructions can be, for example, a dedicated software application or a part of a software application accessed or downloaded via a web browser or other software application (e.g., a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (e.g., a cloud computing network), or other such networks) using one or more network computers.

[0150] For the sake of clarity, only certain selected aspects of a software-based embodiment are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, or any other suitable programming language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be elaborated in this disclosure.

[0151] Furthermore, any software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via any suitable communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0152] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and subcombinations with each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems to be solved.

[0153] The technology from any example can be combined with the technology described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be taken as limiting the scope of the disclosed technology.

[0154] As used in this application and in the claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Additionally, the term "including" means "comprising." Furthermore, "and / or" means "and" or "or" as well as "and" and "or."

[0155] Example Embodiments

[0156] Any of the following example embodiments may be implemented.

[0157] Example 1. One or more computer-readable media having computer-executable instructions encoded thereon, the computer-executable instructions being configured to cause one or more processors to perform operations when programmed thereby, the operations comprising: receiving text-based content comprising a plurality of characters; generating a plurality of character class sequences for the text-based content using the plurality of characters, wherein each of the plurality of character class sequences comprises a plurality of character class identifiers for a plurality of characters of different character sequences in the text-based content, and wherein each of the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution of the plurality of character class sequences, wherein the frequency distribution represents a relative frequency of occurrence of unique character class sequences within a total number of the plurality of character class sequences; and classifying the text-based content based on the calculated frequency distribution of the plurality of character class sequences. The dimensionality of the input to the machine learning model is reduced compared to an input based on the frequency distribution of the character sequences of the text-based content, which can reduce computational complexity and reduce memory utilization of the classification operation without adversely affecting the accuracy of the classification results.

[0158] Example 2. One or more computer-readable media as described in Example 1, wherein classifying the text-based content uses a machine learning model that has been trained using a plurality of examples of text-based content, each of the plurality of examples of text-based content having a known classification, and wherein classifying the text-based content comprises: providing the calculated frequency distribution of the plurality of character category sequences as input to the machine learning model; and determining a classification of the text-based content based on an output of the machine learning model.

[0159] Example 3. One or more computer-readable media as described in Example 2, wherein the machine learning model has been trained by a training operation, the training operation comprising, for each of a plurality of examples of the text-based content: generating a plurality of character category sequences for the example of the text-based content using a plurality of characters in the example of the text-based content and based on the plurality of predefined character categories; calculating a frequency distribution of the plurality of character category sequences for the example of the text-based content; and providing the calculated frequency distribution of the example of the text-based content and the known classification of the example of the text-based content as input to the machine learning model, wherein the input to the machine learning model comprises a feature vector configured to receive the calculated frequency distribution of the example of the text-based content.

[0160] Example 4. One or more computer-readable media of Example 3, wherein the feature vector has a dimension equal to a count of unique character class sequences that appear at least once in multiple examples of the text-based content; or equal to a count of possible unique character class sequences.

[0161] Example 5. The one or more computer-readable media of any of Examples 3-4, wherein the training operation further comprises reducing the dimensionality of the feature vector based on one or more metrics measuring the performance of the machine learning model in classifying multiple examples of text-based content.

[0162] Example 6. One or more computer-readable media as described in any of Examples 1-5, wherein generating the plurality of character category sequences for the text-based content comprises: generating a plurality of character sequences for the text-based content using the plurality of characters; and converting the plurality of character sequences into the plurality of character category sequences based on the plurality of predefined character categories.

[0163] Example 7. The one or more computer-readable media of Example 6, wherein generating the plurality of character sequences comprises scanning from a first character to a last character of the text-based content such that the plurality of character sequences are sequentially ordered and such that adjacent character sequences in the plurality of character sequences are offset by one character.

[0164] Example 8. The one or more computer-readable media of Example 6, wherein each character sequence in the plurality of character sequences has two or more characters.

[0165] Example 9. One or more computer-readable media according to any of Examples 6-8, wherein converting the plurality of character sequences into the plurality of character class sequences comprises: for each character sequence in the plurality of character sequences, converting a character in the character sequence into a corresponding character class identifier in a corresponding character class sequence in the character class sequences.

[0166] Example 10. One or more computer-readable media according to any one of Examples 1-5, wherein generating the plurality of character category sequences for the text-based content comprises: generating a series of character category identifiers by converting corresponding characters of the text-based content into corresponding character category identifiers; and generating the plurality of character category sequences by scanning the series of character category identifiers from a first character category identifier to a last character category identifier in the series of character category identifiers, such that the plurality of character category sequences are sequentially ordered and such that adjacent character category sequences in the plurality of character category sequences are offset by one character category identifier.

[0167] Example 11. One or more computer-readable media according to any one of Examples 1-5, wherein the plurality of character class sequences are generated for the text-based content by scanning the plurality of characters of the text-based content character by character, and for each character sequence in a plurality of character sequences for the text-based content, determining a corresponding character class sequence in the plurality of character class sequences.

[0168] Example 12. The one or more computer-readable media of any of Examples 1-11, wherein the predefined character category is a Unicode category.

[0169] Example 13. The one or more computer-readable media of any of Examples 1-12, wherein the operations further comprise: in response to classifying the text-based content as unsolicited, limiting distribution of the text-based content or generating an alert for the text-based content.

[0170] Example 14. A computing device comprising: a memory; one or more hardware processors coupled to the memory; and one or more computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations, the operations comprising: receiving text-based content comprising a plurality of characters; generating a plurality of character class sequences for the text-based content using the plurality of characters, wherein each of the plurality of character class sequences comprises a plurality of character class identifiers for a plurality of characters of a different character sequence in the text-based content, and wherein each of the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution for the plurality of character class sequences, wherein the frequency distribution represents a relative frequency of occurrence of unique character class sequences within a total number of the plurality of character class sequences; and classifying the text-based content based on the calculated frequency distribution of the plurality of character class sequences. The dimensionality of an input to a machine learning model is reduced compared to an input based on the frequency distribution of character sequences of the text-based content, which can reduce computational complexity and memory utilization of a classification operation without adversely affecting the accuracy of the classification results.

[0171] Example 15. The computing device of Example 14, wherein classifying the text-based content uses a machine learning model that has been trained using multiple examples of text-based content, each example of the text-based content having a known classification, and wherein classifying the text-based content comprises: providing the calculated frequency distribution of the multiple character category sequences as input to the machine learning model; and determining the classification of the text-based content based on an output of the machine learning model.

[0172] Example 16. The computing device of Example 15, wherein the machine learning model has been trained by a training operation comprising, for each of a plurality of examples of the text-based content: generating a plurality of character category sequences for the example of the text-based content using a plurality of characters in the example of the text-based content and based on the plurality of predefined character categories; calculating a frequency distribution of the plurality of character category sequences for the example of the text-based content; and providing the calculated frequency distribution of the example of the text-based content and the known classification of the example of the text-based content as input to the machine learning model, wherein the input to the machine learning model comprises a feature vector configured to receive the calculated frequency distribution of the example of the text-based content.

[0173] Example 17. The computing device of Example 16, wherein the training operation further comprises reducing the dimensionality of the feature vector based on one or more metrics measuring the performance of the machine learning model in classifying the plurality of examples of the text-based content.

[0174] Example 18. The computing device of any of Examples 14 to 17, wherein the operations further comprise: in response to classifying the text-based content as unsolicited, limiting distribution of the text-based content or generating an alert for the text-based content.

[0175] Example 19. A computer-implemented method comprises: receiving a plurality of examples of text-based content, each example having a known classification; and for each of the plurality of examples of the text-based content: generating a plurality of character class sequences for the example of the text-based content using a plurality of characters of the example of the text-based content, wherein each of the plurality of character class sequences for the example of the text-based content includes a plurality of character class identifiers for a plurality of characters of a different character sequence in the example of the text-based content, and wherein each of the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution of the plurality of character class sequences for the example of the text-based content, wherein the frequency distribution represents a relative frequency of occurrence of unique character class sequences within a total number of the plurality of character class sequences for the example of the text-based content; and training a machine learning model based on the calculated frequency distribution of the example of the text-based content and the known classification of the example of the text-based content. The dimensionality of the input to the machine learning model is reduced compared to an input based on the frequency distribution of the character sequences of the example of the text-based content, which can reduce computational complexity and memory utilization of training operations without adversely affecting training results.

[0176] Example 20. The computer-implemented method of Example 19 further includes: receiving new text-based content comprising a plurality of characters; generating a plurality of character category sequences for the new text-based content using the plurality of characters of the new text-based content, wherein each character category sequence of the plurality of character category sequences of the new text-based content comprises a plurality of character category identifiers for the plurality of characters of different character sequences in the new text-based content, respectively, and wherein each character category identifier of the plurality of character category identifiers of the new text-based content identifies one of a plurality of predefined character categories; calculating a frequency distribution of the plurality of character category sequences of the new text-based content; providing the calculated frequency distribution of the new text-based content as input to a machine learning model; and classifying the new text-based content using the machine learning model.

[0177] Example Alternatives

[0178] The techniques from any example can be combined with the techniques described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. Instead, the scope of the disclosed technology includes that covered by the scope and spirit of the appended claims.

Claims

1. One or more computer-readable media (920, 925, 940) having encoded thereon computer-executable instructions for causing one or more processors, when programmed thereby, to perform operations comprising: receiving text-based content comprising a plurality of characters (210); generating a plurality of character class sequences for the text-based content using the plurality of characters (220, 230), wherein each character class sequence in the plurality of character class sequences includes a plurality of character class identifiers for a plurality of characters of different character sequences in the text-based content, respectively, and wherein each character class identifier in the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution of the plurality of character class sequences (240), wherein the frequency distribution represents a relative frequency of occurrence of unique character class sequences within a total number of the plurality of character class sequences; and The text-based content is classified based on the calculated frequency distribution of the plurality of character class sequences (250).

2. One or more computer-readable media according to claim 1, wherein: Classifying the text-based content uses a machine learning model that has been trained using a plurality of examples of text-based content, each of the plurality of examples of text-based content having a known classification, and wherein classifying the text-based content comprises: providing the calculated frequency distribution of the plurality of character category sequences as input to the machine learning model; and A classification of the text-based content is determined based on an output of the machine learning model.

3. One or more computer-readable media according to claim 2, wherein: The machine learning model has been trained by a training operation comprising, for each of a plurality of examples of the text-based content: generating a plurality of character category sequences for the example of text-based content using a plurality of characters in the example of text-based content and based on the plurality of predefined character categories; calculating a frequency distribution of the plurality of character class sequences for examples of the text-based content; as well as providing as input to the machine learning model the calculated frequency distribution of examples of the text-based content and the known classifications of examples of the text-based content, Wherein, the input to the machine learning model comprises a feature vector configured to receive a calculated frequency distribution of examples of the text-based content.

4. One or more computer-readable media according to claim 3, wherein: The feature vector has the following dimensions: is equal to a count of unique character class sequences that occur at least once in a plurality of instances of the text-based content; or Equal to the count of possible unique character class sequences.

5. One or more computer-readable media according to claim 3, wherein: The training operation also includes reducing the dimensionality of the feature vector based on one or more metrics measuring the performance of the machine learning model in classifying multiple examples of the text-based content.

6. One or more computer-readable media according to claim 1, wherein: Generating the plurality of character category sequences for the text-based content includes: generating a plurality of character sequences for the text-based content using the plurality of characters; and The plurality of character sequences are converted into the plurality of character category sequences based on the plurality of predefined character categories.

7. One or more computer-readable media according to claim 6, wherein: Generating the plurality of character sequences includes scanning from a first character to a last character of the text-based content so that the plurality of character sequences are sequentially ordered, and shifting adjacent character sequences in the plurality of character sequences by one character.

8. One or more computer-readable media according to claim 6, wherein: Each character sequence of the plurality of character sequences has two or more characters.

9. One or more computer-readable media according to claim 6, wherein: Converting the plurality of character sequences into the plurality of character class sequences includes: for each character sequence in the plurality of character sequences, converting characters in the character sequence into corresponding character class identifiers in a corresponding character class sequence in the character class sequences.

10. The one or more computer-readable media of claim 1, wherein: Generating the plurality of character category sequences for the text-based content includes: generating a series of character class identifiers by converting respective characters of the text-based content into corresponding character class identifiers; and The multiple character category sequences are generated by scanning the series of character category identifiers from the first character category identifier to the last character category identifier in the series of character category identifiers, so that the multiple character category sequences are sorted sequentially and adjacent character category sequences in the multiple character category sequences are offset by one character category identifier.

11. One or more computer-readable media according to claim 1, wherein: Generating the plurality of character category sequences for the text-based content includes: The text-based content is scanned character by character, and for each character sequence in a plurality of character sequences of the text-based content, a corresponding character class sequence in the plurality of character class sequences is determined.

12. One or more computer-readable media according to claim 1, wherein: The predefined character categories are Unicode categories.

13. One or more computer-readable media according to claim 1, wherein: The operations further include: responsive to classifying the text-based content as unsolicited, restricting distribution of the text-based content; In response to classifying the text-based content as unsolicited, generating an alert for the text-based content; or In response to classifying the text-based content as normal, distribution of the text-based content is permitted.

14. A computing device comprising: Memory (920, 925); one or more hardware processors (910, 915) coupled to the memory; as well as One or more computer-readable storage media (940) storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations comprising: receiving text-based content comprising a plurality of characters (210); generating a plurality of character class sequences for the text-based content using the plurality of characters (220, 230), wherein each character class sequence in the plurality of character class sequences includes a plurality of character class identifiers for a plurality of characters of different character sequences in the text-based content, respectively, and wherein each character class identifier in the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution of the plurality of character class sequences (240), wherein the frequency distribution represents relative frequencies of occurrence of unique character class sequences within a total number of the plurality of character class sequences; and The text-based content is classified based on the calculated frequency distribution of the plurality of character category sequences (250), wherein the dimensionality of the input to the machine learning model is reduced compared to the input according to the frequency distribution of the character sequences of the text-based content.

15. The computing device of claim 14, wherein: Classifying the text-based content uses a machine learning model that has been trained using a plurality of examples of text-based content, each of the plurality of examples of text-based content having a known classification, and wherein classifying the text-based content comprises: providing the calculated frequency distribution of the plurality of character category sequences as input to the machine learning model; and A classification of the text-based content is determined based on an output of the machine learning model.

16. The computing device of claim 15, wherein: The machine learning model has been trained by a training operation comprising, for each of a plurality of examples of the text-based content: generating a plurality of character category sequences for the example of text-based content using a plurality of characters in the example of text-based content and based on the plurality of predefined character categories; calculating a frequency distribution of the plurality of character class sequences for examples of the text-based content; as well as providing as input to the machine learning model the calculated frequency distribution of examples of the text-based content and the known classifications of examples of the text-based content, Wherein, the input to the machine learning model comprises a feature vector configured to receive a calculated frequency distribution of examples of the text-based content.

17. The computing device of claim 16, wherein: The training operation also includes reducing the dimensionality of the feature vector based on one or more metrics measuring the performance of the machine learning model in classifying the plurality of examples of the text-based content.

18. The computing device of claim 14, wherein: The operations further include: responsive to classifying the text-based content as unsolicited, restricting distribution of the text-based content; In response to classifying the text-based content as unsolicited, generating an alert for the text-based content; or In response to classifying the text-based content as normal, distribution of the text-based content is permitted.

19. A computer-implemented method comprising: receiving a plurality of examples of text-based content, each example of the plurality of examples of text-based content having a known classification (710); as well as For each of the multiple examples of text-based content: generating a plurality of character class sequences for an example of text-based content using a plurality of characters of the example of text-based content (720, 730), wherein each character class sequence of the plurality of character class sequences for the example of text-based content includes a plurality of character class identifiers for a plurality of characters of a different character sequence in the example of text-based content, respectively, and wherein each character class identifier of the plurality of character class identifiers identifies one of a plurality of predefined character classes; calculating a frequency distribution of the plurality of character class sequences for the examples of the text-based content (740), wherein the frequency distribution represents the relative frequencies of occurrence of unique character class sequences for the examples of the text-based content within the plurality of character class sequences for the total number of examples of the text-based content; and A machine learning model (750) is trained based on the calculated frequency distribution of the multiple character category sequences of the examples of the text-based content and the known classification of the examples of the text-based content, wherein the dimensionality of the input to the machine learning model is reduced compared to the input of the frequency distribution of the character sequence of the examples of the text-based content.

20. The computer-implemented method of claim 19, further comprising: receiving new text-based content comprising a plurality of characters; generating a plurality of character class sequences for the new text-based content using the plurality of characters of the new text-based content, wherein each character class sequence of the plurality of character class sequences of the new text-based content includes a plurality of character class identifiers for a plurality of characters of different character sequences in the new text-based content, respectively, and wherein each character class identifier of the plurality of character class identifiers for the new text-based content identifies one of the plurality of predefined character classes; calculating a frequency distribution of the plurality of character category sequences for the new text-based content; providing the calculated frequency distribution of the new text-based content as input to the machine learning model; and The new text-based content is classified using the machine learning model.

Images