Method, device, electronic device and storage medium for classifying spam text messages

Abstract

The present invention relates to a method, device, electronic device and storage medium for classifying spam text messages. The method for classifying spam text messages includes the following steps: S1, pre-processing text messages, downloading a text message data set from a communication network as a basic data set for generating word vectors, and using a vector space model to form a word-text matrix from the word set; S2, generating text message feature vectors including a topic feature vector and a context feature vector, the topic feature vector is obtained by performing a singular value decomposition of the word-text matrix, and the context feature vector is used to predict the probability of the context based on the central word; S3, neural network fusion and training, a training task for a classifier for processing text messages, the classifier is used to distinguish whether the text message is a spam text message; S4, text message classification and discrimination. According to the spam text message classification method of the present invention, the topic feature and the context feature can make up for each other's shortcomings, realize multi-feature fusion, and thus realize accurate classification of spam text messages.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a method, device, electronic device and storage medium for classifying spam text messages based on the fusion of topic features and context features. Background Art

[0002] Spam text messages are unwanted messages sent to users without their consent, or messages that users cannot opt ​​out of. These messages primarily include: commercial or advertising messages sent without their consent; and other messages that violate industry self-regulatory regulations. The prevalence of spam text messages has seriously impacted people's lives, the image of carriers, and even social stability.

[0003] Current solutions for spam message control can be broadly categorized into two technical areas: keyword filtering and artificial intelligence (AI) algorithms. Keyword filtering is a relatively traditional approach. It involves accumulating a large number of spam message samples, extracting keywords from them, and then using software system rules to determine whether a message is spam based on the presence of these keywords. Some systems even require manual judgment. In recent years, with the increasing maturity of big data technology, machine learning, and AI algorithms, AI-based algorithms have become increasingly adopted by the industry. Spam message classification, within the AI ​​field, falls under the category of text classification. Currently, mainstream text classification models fall into two categories: probabilistic and non-probabilistic models. Probabilistic models include the Naive Bayes classification algorithm, while non-probabilistic models are further categorized into semantic models that use word vectors to represent text and convolutional neural network models. Both types of algorithms are widely used in scenarios such as spam email and spam message classification.

[0004] However, current methods based on keyword filtering and probability statistics or non-probability statistics all have certain limitations, which are mainly manifested as follows:

[0005] Regarding the first method: using "keyword filtering" technology to block spam text messages, since such text messages are often expressed in euphemistic sentences and the words in the sent content are constantly changing, it is very difficult for machines to identify spam text messages, and often a large amount of manual intervention and intervention is necessary.

[0006] The second approach, using probabilistic statistics for text classification, has its flaws: For example, using the Naive Bayes algorithm for spam text message classification, the Naive Bayes method uses the distribution of each category in the training samples and the distribution of each characteristic element within the category to calculate the posterior probability and use the maximum likelihood method to determine the category to which the test sample belongs. The limitations of using this algorithm for text classification are that the proportions of each category in the training set cannot differ too much; categories with larger proportions have a higher probability of classification. Furthermore, the algorithm assumes independence between words and shares weights, ignoring their relevance and facing the problem of coreference resolution (different representations of the same entity), which reduces the algorithm's accuracy.

[0007] Regarding the flaws of the second method of text classification using non-probabilistic statistical models, for example, using word vectors to represent the semantic content of text, this approach uses a metric in the word vector space to represent the semantic similarity between texts. However, Chinese words can have polysemy (the same word can have different meanings, such as the word "apple," which can mean either a fruit or an iPhone) and synonyms (different words with the same meaning, such as "trip" and "journey"). Furthermore, Chinese sentences can use metaphors to express meaning. Relying solely on word vector weights to determine text similarity cannot achieve an accurate understanding of semantics. This is because the weights of word-text vectors only reflect the frequency of word occurrence and do not consider the order in which words appear in the text or the different meanings of the same word within the context, that is, the position of the word within the context. This also reduces the accuracy of such algorithms.

[0008] Conventional text classification relies on single features or complex "memory" network structures, such as LSTMs, to complete the classification task. However, SMS classification has its own unique characteristics and difficulties compared to conventional text classification, primarily manifested in the following aspects: sparsity. Short texts are short, typically containing only a few to a dozen meaningful words, making it difficult to extract effective feature words; real-time nature. Short texts are updated rapidly and easily spread; massive volume. Short texts are prevalent in our daily lives. Due to their timely updates and rapid dissemination, a vast amount of short text data has accumulated on the internet, requiring high-speed processing and computational speed for short texts; and irregularity. Short texts are concise, but abbreviations, irregular terms, and internet buzzwords are widely used, resulting in high text noise. Compared to conventional text classification, SMS classification has its own unique characteristics and difficulties. Conventional text classification is less effective in classifying relatively short texts. Because these algorithms use simple statistical methods to determine the weights of individual words as feature values, they ignore the semantic relationships between different feature words and their impact on SMS text representation. This can easily lead to semantic differences between the text representation of the SMS message and the original text.

[0009] Traditional algorithm models based on single features cannot cope with the problem of polysemy and synonymy of words in Chinese natural language, or the loss of deep semantic information due to different contexts, which affects the effectiveness of SMS text classification. Feature extraction for short texts such as SMS cannot extract deep semantics. Summary of the Invention

[0010] The technical problem to be solved by this invention is to address the problem in spam text message classification scenarios where traditional algorithm models based on single features are unable to cope with the loss of deep semantic information caused by the polysemy and monosemy of words in Chinese natural language, or due to different contexts. In particular, the problem of feature extraction from short texts such as text messages is that deep semantics cannot be extracted. In short, this invention solves the problem of how to enable machines to identify the true semantics expressed in a text message, thereby achieving accurate text message classification.

[0011] To solve the above technical problem, according to one aspect of the present invention, a method for classifying spam text messages is provided, which comprises the following steps:

[0012] S1. SMS text preprocessing: Download SMS dataset from the communication network as the basic data set for word vector generation. The SMS dataset includes classification labels and SMS content. Classification labels include normal, commercial advertising, harassment, fraud, invoice processing, loan processing, and real estate. Considering each SMS as a text, the set of all SMS constitutes the text set D = {d1, d2, ..., d n}, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m};

[0013] Use the vector space model to form a word-text matrix from the word set:

[0014]

[0015] S2. Generate SMS feature vectors, where the SMS feature vectors include: topic feature vectors and context feature vectors. The topic feature vectors are used to identify semantic expressions including synonyms and polysemous words, and are obtained by performing singular value decomposition of the word-text matrix in step S1. The context feature vectors are used to predict the probability of the context based on the central word.

[0016] S3. Neural network fusion and training. The neural network is a multilayer perceptron, consisting of an input layer, a hidden layer, and an output layer. The neural network is used to train a text message classifier, which is used to distinguish whether a text message is spam or not. Each neuron in the input layer of the multilayer perceptron is a binary data tuple consisting of a topic feature vector and a context feature vector. The binary inputs are linearly combined. The output predicts the classification. After multiple rounds of iteration, a stable classifier is finally obtained.

[0017] S4. SMS classification and discrimination: A text message of unknown classification is converted into a corresponding topic feature matrix and context feature matrix, which are input into the classifier trained by the multi-layer perceptron to obtain the classification result of the text message. Based on business needs, corresponding processing is performed, such as interception, release, and sending of the text message.

[0018] According to an embodiment of the present invention, TF-IDF frequency calculation may be performed after step S1. The TF-IDF frequency is used to calculate the scalar value of each element in the matrix, which is expressed as:

[0019]

[0020] Among them, tf ij For word w i Appears in text d j The frequency in tf·j is text d j The sum of the frequencies of all words that appear in df i It contains the word w i The number of texts in , df is the total number of texts in the text set D;

[0021] The logic expressed by this formula is: the more frequently a word appears in a text, the more important it is in the text; the fewer texts a word appears in the entire text set, the more representative it is of the text in which it appears; therefore, the TFIDF eigenvalue reflects a comprehensive feature of the importance of a word in a text and the degree to which the word represents the characteristics of the text. The higher the eigenvalue, the more important and representative the word is in the text. The actual meaning of the word vector space is that each column vector x j Represents a text d j

[0022]

[0023] The inner product or normalized inner product (cosine) of two word vectors represents the semantic similarity between the corresponding texts; the similarity calculation formula is:

[0024]

[0025] where · represents the inner product of vectors, ||x i || represents the norm of the vector, that is, the length of the vector; intuitively speaking, the more words that appear in two texts, the closer their semantic content is; at this time, the larger the inner product of the corresponding word vectors; the similarity of word vectors constructed with word-frequency features is used as the basis for measuring text similarity, which is based on the independence assumption between words, but the large number of polysemy and multiple meanings of a word in natural language makes it impossible for the model to recognize features at the semantic level, thereby leading to classification errors; therefore, the present invention, based on the word-frequency feature vector model, further extracts features from this word vector to form a deeper semantic feature - topic feature vector, so as to solve the problems of polysemy and multiple meanings in text classification.

[0026] According to an embodiment of the present invention, in step S2, topic feature vectors may be extracted by performing singular value decomposition on the word-text vector space matrix to obtain the topic feature vectors;

[0027] The singular value decomposition process of the word-matrix is ​​to decompose the original word-text matrix into three matrices: topic feature matrix, singular value matrix, and document matrix. The multiplication of the three matrices is equal to the original matrix.

[0028] For a word-frequency feature matrix A containing m words and n documents m×n The singular value decomposition of can be expressed as: A=U∑V T , where U is an m-order orthogonal matrix, each of whose column vectors is a left singular vector of A; V is an n-order orthogonal matrix, each of whose columns is a right singular vector of A; ∑ is an m×n-order diagonal matrix, the matrix has values ​​in the oblique part and the rest are 0. The elements with values ​​are arranged in descending order, and the diagonal elements are called singular values ​​of A, which are arranged in descending order;

[0029]

[0030] The U matrix on the right side of the equation is the topic feature matrix. It should be noted here that the term "topic" is an abstraction of "latent semantics" in this invention. After decomposing a text matrix expressing word-frequency features, from a geometric perspective, the word-feature vector is mapped to another low-dimensional vector space through a linear transformation.

[0031] According to an embodiment of the present invention, in step S2, the skip-gram model of word2vec may be used to generate a context feature vector. Generating the context feature vector includes the following steps:

[0032] S21, after the SMS text is segmented, a word set W is formed = {w1, w2, ..., w v};

[0033] S22. Perform one-hot encoding on each word in the word set to form a one-hot vector matrix X of the word, where the dimension of each column of the word vector is V;

[0034] S23. Each column of word vector x[k] is passed to the hidden layer through the neurons of the skip-gram model input layer. The hidden layer performs the dot product operation between the weight vector W[|v|,m] and the input vector x[k]. The value H[1,N] of the hidden layer neurons is directly passed to the output layer. The output layer performs the dot product operation between H[1,N] and W'[N,|v|] and gives the vector U. The probability of each vector is obtained. Using the softmax function, the output vector U is obtained at each iteration. The word with the largest probability is the final result. If the word is within the specified upper and lower bounds, the word with the largest probability is the final result. The predicted word in the text position is wrong, and the backpropagation algorithm can be used to correct the weight vectors W and W'; the above steps are performed for each word w(t) in the dictionary; and each word w(t) will be passed K times; the forward propagation algorithm will be executed |v|*k times in each period of time; the softmax activation function is used to obtain the multi-classification probability of the context of the central word, which is compared with the true value. After multiple rounds of training, the W parameter matrix is ​​continuously corrected to minimize the loss function. Finally, when the model converges, W is the parameter matrix at this time, and the word vector matrix with context features is obtained.

[0035] According to an embodiment of the present invention, in step S3, the topic features and context features can be fused through a neural network, and the input of the two-tuple is linearly combined into: WX+UY, where W is the weight of each column vector of the topic feature matrix X, and U is the weight of each column vector of the context feature matrix Y. The values ​​of W and U range from 0 to 1. Any real number, wherein the initial value is defined as 0.5, so that before the training starts, the model judges the importance of the topic features and the context features to be 0.5;

[0036] The neurons in the hidden layer use [xy] vector and [wy] T The dot product of the vectors is calculated, and the calculated values ​​of each neuron are added together. The calculation method for each neuron in the output layer is to find the joint probability of the values ​​of the neurons in the hidden layer, and then pass it through the softmax function to obtain a multi-classification probability value. The category to which the neuron with the largest probability belongs is the predicted category of the input text message. The error between the predicted classification and the true classification is backpropagated to the hidden layer, and the weight ratio of w and u is corrected. After multiple rounds of iteration, a stable classifier is finally obtained.

[0037] According to an embodiment of the present invention, the method for classifying spam text messages may further include: S5, inputting the intercepted or released text messages into a text message training corpus.

[0038] According to a second aspect of the present invention, there is provided an apparatus for classifying spam text messages, comprising:

[0039] The preprocessing module is used for SMS text preprocessing. The preprocessing module downloads SMS datasets from the communication network as the basic data set for generating word vectors. The SMS datasets include classification labels and SMS content. Classification labels include normal, commercial advertising, harassment, fraud, invoice processing, loan processing, and real estate. If each SMS is regarded as a text, then the set of all SMS constitutes the text set D = {d1, d2, ..., d n}, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m};

[0040] Use the vector space model to form a word-text matrix from the word set:

[0041]

[0042] The feature engineering module is used to generate SMS feature vectors. The SMS feature vectors include topic feature vectors and context feature vectors. The topic feature vectors are used to identify semantic expressions including synonyms and polysemous words. They are obtained by performing singular value decomposition on the word-text matrix in step S1. The context feature vectors are used to predict the probability of the context based on the central word.

[0043] The neural network module is used to build and train neural network fusion. The neural network is a multi-layer perceptron, which consists of an input layer, a hidden layer, and an output layer. The neural network is used to train a text message classifier, which is used to distinguish whether a text message is spam or not. Each neuron in the input layer of the multi-layer perceptron is a binary data tuple, which includes a topic feature vector and a context feature vector. The binary inputs are linearly combined. The output predicts the classification. After multiple rounds of iteration, a stable classifier is finally obtained.

[0044] The SMS judgment and classification module is used to classify and identify spam SMS messages. When an SMS message of unknown classification is input, the SMS judgment and classification module forms the corresponding topic feature matrix and context feature matrix, and inputs them into the classifier trained by the multi-layer perceptron to obtain the classification result of the SMS message. According to business needs, corresponding processing is made, such as interception, release, and delivery of the SMS message.

[0045] According to an embodiment of the present invention, the topic feature vector can be obtained by performing singular value decomposition on the word-text vector space matrix; the context feature vector can be generated using the skip-gram model of word2vec.

[0046] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a spam message classification program stored in the memory and executable on the processor, wherein the spam message classification program implements the steps of the above-mentioned spam message classification method when executed by the processor.

[0047] According to a fourth aspect of the present invention, a computer storage medium is provided, wherein a spam message classification program is stored on the computer storage medium, and when the spam message classification program is executed by a processor, the steps of the above-mentioned spam message classification method are implemented.

[0048] Compared with the prior art, the technical solution provided by the embodiments of the present invention can achieve at least the following beneficial effects:

[0049] This paper constructs a multi-layer perceptron neural network structure to fuse the topic features and context features of words in text messages, two different vector spaces, and performs supervised training on the multi-layer perceptron model to obtain a simple and efficient classifier. This method is simple and easy to implement. Compared with other neural network models such as LSTM, it has a simple structure, low computational complexity, and is as accurate as LSTM models.

[0050] The present invention has two main highlights: First, the use of two different features, topic features and context features. These two features each have their own advantages and can compensate for each other's shortcomings: the former can solve the problem of multiple meanings and multiple synonyms, but cannot solve the semantic differences caused by different contexts; the latter can solve the problem of different semantics caused by different contexts, but cannot solve the problem of multiple meanings and multiple synonyms. Second, a method for fusing different feature vector spaces is designed, because the topic feature vector space and the context feature vector space are two completely different vector spaces and cannot be expressed by each other through simple linear transformation. Among the many current text classification methods, the vast majority are based on the same feature vector space for modeling. There are also very few text classification methods that mention the idea of ​​multi-feature fusion, but there is no specific implementation method. The present invention proposes to achieve multi-feature fusion by designing a simple neural network structure - a multi-layer perceptron, and to fuse two different features through supervised training, thereby achieving accurate classification of spam text messages.

[0051] The present invention provides a model that can fuse multiple different dimensional features, has relatively low computational complexity and high accuracy, to achieve the classification of text messages. The advantages of the present invention are: First, two different feature vectors are fused together through a neural network model-a multi-layer perceptron model, so that the two feature vectors can play their respective strengths and make up for each other's shortcomings. On the basis of word vector-frequency feature representation, the word vector space is linearly transformed (dimensionality reduction, feature extraction) to obtain a topic-feature vector space. Technically, through singular value decomposition processing, the word-feature vector space is mapped to the topic-feature vector space, thereby eliminating the influence of polysemy and synonymy of multiple words. Second, by designing a neural network structure of a multi-layer perceptron, the skip-gram model is used to obtain the context feature vector of the word, which is fused with the topic feature vector, so that the classification of spam text messages can be done more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0053] Figure 1 The flowchart of the junk message classification method according to the embodiment of the present invention is shown.

[0054] Figure 2 is a graph showing a singular value decomposition according to an embodiment of the present invention.

[0055] Figure 3 3 is a schematic diagram showing how the representation of text in word vector space is converted into the representation in topic space through linear transformation according to an embodiment of the present invention.

[0056] Figure 4a FIG. 4 is a schematic diagram showing a Skip-gram model according to an embodiment of the present invention.

[0057] Figure 4b Schematic diagram of the Skip-gram network architecture according to an embodiment of the present invention

[0058] Figure 5 2 is a schematic diagram showing the structure of a multi-layer perceptron that integrates two features according to an embodiment of the present invention.

[0059] Figure 6 FIG. 4 is a flowchart showing the SMS classification and interception process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in the present patent application specification and claims do not denote any order, quantity, or importance, but are merely used to distinguish different components. Similarly, terms such as "a" or "an" do not denote a limitation of quantity, but rather denote the presence of at least one.

[0062] Figure 1 The flowchart of the junk message classification method according to the embodiment of the present invention is shown.

[0063] like Figure 1 As shown, the method for classifying spam text messages includes the following steps:

[0064] S1. SMS text preprocessing: Download SMS dataset from the communication network as the basic data set for word vector generation. The SMS dataset includes classification labels and SMS content. Classification labels include normal, commercial advertising, harassment, fraud, invoice processing, loan processing, and real estate. Considering each SMS as a text, the set of all SMS constitutes the text set D = {d1, d2, ..., d n}, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m};

[0065] Use the vector space model to form a word-text matrix from the word set:

[0066]

[0067] S2. SMS feature vector generation, where the SMS feature vector includes: a topic feature vector and a context feature vector. The topic feature vector is used to identify semantic expressions including synonyms and polysemous words, and is obtained by performing singular value decomposition of the word-text matrix in step S1. The context feature vector is used to predict the probability of the context based on the central word.

[0068] S3. Neural network fusion and training. The neural network is a multi-layer perceptron, which includes an input layer, a hidden layer, and an output layer. The neural network is used to train a text message classifier, which is used to distinguish whether a text message is spam. Each neuron in the input layer of the multi-layer perceptron is a binary data. The binary data content includes a topic feature vector and a context feature vector. The binary input is linearly combined. The output predicts the classification. After multiple rounds of iteration, a stable classifier is finally obtained.

[0069] S4. SMS classification and discrimination: A text message of unknown classification is converted into a corresponding topic feature matrix and context feature matrix, which are input into the classifier trained by the multi-layer perceptron to obtain the classification result of the text message. Based on business needs, corresponding processing is performed, such as interception, release, and sending of the text message.

[0070] This paper constructs a multi-layer perceptron neural network structure to fuse the topic features and context features of words in text messages, two different vector spaces, and performs supervised training on the multi-layer perceptron model to obtain a simple and efficient classifier. This method is simple and easy to implement. Compared with other neural network models such as LSTM, it has a simple structure, low computational complexity, and is as accurate as LSTM models.

[0071] According to one or some embodiments of the present invention, after step S1, TF-IDF frequency calculation is performed. The TF-IDF frequency is used to calculate the scalar value of each element in the matrix, which is expressed as:

[0072]

[0073] Among them, tf ij For word w i Appears in text d j The frequency in tf ·j is text d j The sum of the frequencies of all words that appear in df i It contains the word w i df is the total number of texts in the text set D.

[0074] The logic expressed by this formula is: the more frequently a word appears in a text, the more important it is in the text; the fewer texts a word appears in the entire text set, the more representative it is of the text in which it appears; therefore, the TFIDF eigenvalue reflects a comprehensive feature of the importance of a word in a text and the degree to which the word represents the characteristics of the text. The higher the eigenvalue, the more important and representative the word is in the text. The actual meaning of the word vector space is that each column vector x j Represents a text d j

[0075]

[0076] The inner product or normalized inner product (cosine) of two word vectors represents the semantic similarity between the corresponding texts. The similarity calculation formula is:

[0077]

[0078] where · represents the inner product of vectors, ||x i || represents the norm of the vector, that is, the length of the vector; intuitively speaking, the more words that appear in two texts, the closer their semantic content is; at this time, the larger the inner product of the corresponding word vectors; the similarity of word vectors constructed with word-frequency features is used as the basis for measuring text similarity, which is based on the independence assumption between words, but the large number of polysemy and multiple meanings of a word in natural language makes it impossible for the model to recognize features at the semantic level, thereby leading to classification errors; therefore, the present invention, based on the word-frequency feature vector model, further extracts features from this word vector to form a deeper semantic feature - topic feature vector, so as to solve the problems of polysemy and multiple meanings in text classification.

[0079] Figure 2 is a diagram illustrating a singular value decomposition according to an embodiment of the present invention; Figure 3 3 is a schematic diagram showing how the representation of text in word vector space is converted into the representation in topic space through linear transformation according to an embodiment of the present invention.

[0080] like Figure 2 As shown, in step S2, the topic feature vector is extracted, and the topic feature vector can be obtained by performing singular value decomposition on the word-text vector space matrix.

[0081] Latent semantic analysis (LSA) is an unsupervised learning method mainly used for topic analysis of text. Its characteristic is to discover the topic-based semantic relationship between text and words through matrix decomposition. Topic feature vector based on words: Topic refers to a number of semantically related words in the text. A topic consists of at least one word. In the present invention, a topic consists of a word in the text of a short message. In the present invention, although the topic and the word are essentially the same thing, the feature vector of the topic and the feature vector of the word are two different vectors in the scenario of polysemy and synonymy of multiple words, and the space they are in is also different (the multiplication and decomposition of the matrix is ​​a spatial transformation of the vector in a geometric sense: stretching or rotation). Because of this, the topic feature solves the semantic expression problem of synonyms and polysemy. The topic feature of a word is obtained by performing a singular value decomposition of the word-text matrix described in (1).

[0082] The singular value decomposition process of the word-matrix is ​​to decompose the original word-text matrix into three matrices: topic feature matrix, singular value matrix, and document matrix. The product of the three matrices is equal to the original matrix.

[0083] like Figure 3 As shown, for a word-frequency feature matrix A containing m words and n documents m×n The singular value decomposition of can be expressed as: A=U∑V T , where U is an m-order orthogonal matrix, each of whose column vectors is a left singular vector of A; V is an n-order orthogonal matrix, each of whose columns is a right singular vector of A; ∑ is an m×n-order diagonal matrix, which has values ​​in the diagonal part and the rest are 0. The elements with values ​​are arranged in descending order, and the diagonal elements are called singular values ​​of A, which are arranged in descending order.

[0084]

[0085] The U matrix on the right side of the equation is the topic feature matrix. It should be noted here that the term "topic" is an abstraction of "latent semantics" in this invention. After decomposing a text matrix expressing word-frequency features, from a geometric perspective, the word-feature vector is mapped to another low-dimensional vector space through a linear transformation.

[0086] This low-dimensional vector space reflects a potential semantic feature of the word. After the linear transformation of the word-frequency matrix, each vector x in the word vector space is j It can be expressed as y j A linear combination of j The coefficient of is the topic vector:

[0087] xj ≈y 1j t1+y 2j t2+…+y kj t k , j=1,2…,n (6)

[0088] It's important to emphasize that, in this context, a topic is a single word, but the two have different feature vectors. After singular value decomposition, the word frequency feature vector for the word "apple" is different from the topic feature vector for the word "apple." The word feature vectors for "airplane" and "aircraft" are different, but their topic feature vectors may be the same. Because of this, topic feature vectors can address the issues of polysemy and multiple meanings.

[0089] Figure 4a 2 is a schematic diagram showing a Skip-gram model according to an embodiment of the present invention; Figure 4b FIG. 4 is a schematic diagram showing a Skip-gram network architecture according to an embodiment of the present invention.

[0090] like Figure 4a and 4b As shown, in step S2, the word2vec skip-gram model is used to generate a contextual feature vector. This model's original task is to predict the probability of the occurrence of surrounding context words based on the central word. After the skip-gram model training converges, a byproduct is obtained—the word embedding matrix, which is the vector containing the contextual features we will use. The following details the process of generating this word embedding matrix.

[0091] Generating a context feature vector includes the following steps:

[0092] S21, after the SMS text is segmented, a word set W is formed = {w1, w2, ..., w v};

[0093] S22. Perform one-hot encoding on each word in the word set to form a one-hot vector matrix X of the word, where the dimension of each column of the word vector is V;

[0094] S23. Each column of word vector x[k] is passed to the hidden layer through the neurons of the skip-gram model input layer. The hidden layer performs the dot product operation between the weight vector W[|v|,m] and the input vector x[k]. The value H[1,N] of the hidden layer neurons is directly passed to the output layer. The output layer performs the dot product operation between H[1,N] and W'[N,|v|] and gives the vector U. The probability of each vector is obtained. Using the softmax function, the output vector U is obtained at each iteration. The word with the largest probability is the final result. If the word is within the specified upper and lower bounds, the word with the largest probability is the final result. The predicted word in the text position is wrong, and the backpropagation algorithm can be used to correct the weight vectors W and W'; the above steps are performed for each word w(t) in the dictionary; and each word w(t) will be passed K times; the forward propagation algorithm will be executed |v|*k times in each period of time; the softmax activation function is used to obtain the multi-classification probability of the context of the central word, which is compared with the true value. After multiple rounds of training, the W parameter matrix is ​​continuously corrected to minimize the loss function. Finally, when the model converges, W is the parameter matrix at this time, and the word vector matrix with context features is obtained.

[0095] Contextual features of words: Contextual features are a by-product (also called word embedding matrix) generated during the training process of the skip-gram model in Word2Vec. The original task of the skip-gram model is not text classification, but to predict the probability of the n words before and after a central word. That is, the probability of predicting the context based on the central word. During the training process, when the model converges, a by-product is obtained, which is the contextual feature matrix of the words in the text. The by-product generated by this model - the contextual feature matrix, is the second feature required by the present invention.

[0096] Figure 5 2 is a schematic diagram showing the structure of a multi-layer perceptron that integrates two features according to an embodiment of the present invention.

[0097] like Figure 5 As shown, in step S3, the topic features and context features can be fused through a neural network, and the input of the binary group can be linearly combined into: WX+UY, where W is the weight of each column vector of the topic feature matrix X, and U is the weight of each column vector of the context feature matrix Y. The value range of W and U is any real number between 0 and 1, and the initial value is defined as 0.5, so that before the training starts, the model judges the importance of topic features and context features to be 0.5.

[0098] The neurons in the hidden layer use [xy] vector and [wy] TThe dot product of the vectors is calculated, and the calculated values ​​of each neuron are added together. The calculation method for each neuron in the output layer is to find the joint probability of the values ​​of the neurons in the hidden layer, and then pass it through the softmax function to obtain a multi-classification probability value. The category to which the neuron with the largest probability belongs is the predicted category of the input text message. The error between the predicted classification and the true classification is backpropagated to the hidden layer, and the weight ratio of w and u is corrected. After multiple rounds of iteration, a stable classifier is finally obtained.

[0099] Figure 6 FIG. 4 is a flowchart showing the SMS classification and interception process according to an embodiment of the present invention.

[0100] like Figure 6 As shown, the method for classifying spam text messages may further include: S5, inputting the intercepted or released text messages into a text message training corpus.

[0101] The present invention has two main highlights: First, the use of two different features, topic features and context features. These two features each have their own advantages and can compensate for each other's shortcomings: the former can solve the problem of multiple meanings and multiple synonyms, but cannot solve the semantic differences caused by different contexts; the latter can solve the problem of different semantics caused by different contexts, but cannot solve the problem of multiple meanings and multiple synonyms. Second, a method for fusing different feature vector spaces is designed, because the topic feature vector space and the context feature vector space are two completely different vector spaces and cannot be expressed by each other through simple linear transformation. Among the many current text classification methods, the vast majority are based on the same feature vector space for modeling. There are also very few text classification methods that mention the idea of ​​multi-feature fusion, but there is no specific implementation method. The present invention proposes to achieve multi-feature fusion by designing a simple neural network structure - a multi-layer perceptron, and to fuse two different features through supervised training, thereby achieving accurate classification of spam text messages.

[0102] According to a second aspect of the present invention, a device for classifying spam text messages is provided, which includes: a preprocessing module, a feature engineering module, a neural network module and a text message judgment and classification module.

[0103] The preprocessing module is used for SMS text preprocessing. The preprocessing module downloads SMS datasets from the communication network as the basic data set for generating word vectors. The SMS datasets include classification labels and SMS content. Classification labels include normal, commercial advertising, harassment, fraud, invoice processing, loan processing, and real estate. Considering each SMS as a text, the set of all SMS constitutes the text set D = {d1, d2, ..., d n}, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m}.

[0104] Use the vector space model to form a word-text matrix from the word set:

[0105]

[0106] The feature engineering module is used to generate SMS feature vectors, which include topic feature vectors and context feature vectors. Topic feature vectors are used to identify semantic expressions including synonyms and polysemous words. They are obtained by performing singular value decomposition of the word-text matrix in step S1. Context feature vectors are used to predict the probability of context based on the central word.

[0107] The neural network module is used to build the fusion and training of neural networks. The neural network is a multi-layer perceptron, which includes an input layer, a hidden layer, and an output layer. The neural network is used to process the training task of the SMS text classifier, which is used to distinguish whether the SMS is spam. Each neuron in the input layer of the multi-layer perceptron is a binary data. The binary data content includes the topic feature vector and the context feature vector. The binary input is linearly combined. The output predicts the classification. After multiple rounds of iteration, a stable classifier is finally obtained.

[0108] The SMS judgment and classification module is used to classify and identify spam SMS messages. When an SMS message of unknown classification is input, the SMS judgment and classification module forms the corresponding topic feature matrix and context feature matrix, and inputs them into the classifier trained by the multi-layer perceptron to obtain the classification result of the SMS message. According to business needs, corresponding processing is made, such as interception, release, and delivery of the SMS message.

[0109] According to one or some embodiments of the present invention, the topic feature vector is obtained by performing singular value decomposition on the word-text vector space matrix; the context feature vector is generated using the skip-gram model of word2vec.

[0110] The present invention provides a model that can fuse multiple different dimensional features, has relatively low computational complexity and high accuracy, to achieve the classification of text messages. The advantages of the present invention are: First, two different feature vectors are fused together through a neural network model-a multi-layer perceptron model, so that the two feature vectors can play their respective strengths and make up for each other's shortcomings. On the basis of word vector-frequency feature representation, the word vector space is linearly transformed (dimensionality reduction, feature extraction) to obtain a topic-feature vector space. Technically, through singular value decomposition processing, the word-feature vector space is mapped to the topic-feature vector space, thereby eliminating the influence of polysemy and synonymy of multiple words. Second, by designing a neural network structure of a multi-layer perceptron, the skip-gram model is used to obtain the context feature vector of the word, which is fused with the topic feature vector, so that the classification of spam text messages can be done more accurately.

[0111] According to another aspect of the present invention, a device for classifying spam messages is provided, comprising: a memory, a processor, and a spam message classification program stored in the memory and executable on the processor, wherein the spam message classification program implements the steps of the above-mentioned spam message classification method when executed by the processor.

[0112] According to the present invention, a computer storage medium is also provided.

[0113] The computer storage medium stores a junk text message classification program, which implements the steps of the junk text message classification method when executed by the processor.

[0114] The method implemented when the spam message classification program running on the processor is executed can refer to the various embodiments of the spam message classification method of the present invention, and will not be described in detail here.

[0115] The present invention also provides a computer program product.

[0116] The computer program product of the present invention includes a spam message classification program, which implements the steps of the above spam message classification method when executed by a processor.

[0117] The method implemented when the spam message classification program running on the processor is executed can refer to the various embodiments of the spam message classification method of the present invention, and will not be described in detail here.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0119] The above are merely exemplary embodiments of the present invention and are not intended to limit the scope of protection of the present invention. The scope of protection of the present invention is determined by the appended claims.

Claims

1. A method for classifying spam text messages, comprising the following steps: S1. SMS text preprocessing: Download SMS dataset from the communication network as the basic data set for word vector generation. The SMS dataset includes classification labels and SMS content. The classification labels include normal, commercial advertisement, harassment, fraud, invoice processing, loan processing, and real estate. Considering each SMS as a text, the set of all SMS constitutes the text set D = {d1, d2, ..., d n }, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m }; Use the vector space model to form a word-text matrix from the word set: S2, SMS feature vector generation, where The SMS feature vector includes: a topic feature vector and a context feature vector. The topic feature vector is used to identify semantic expressions including synonyms and polysemous words, and is obtained by performing singular value decomposition of the word-text matrix in step S1. The context feature vector is used to predict the probability of the context based on the central word. In step S2, topic feature vectors are extracted by performing singular value decomposition on the word-text vector space matrix to obtain topic feature vectors. The singular value decomposition process of the word-matrix is ​​to decompose the original word-text matrix into three matrices: topic feature matrix, singular value matrix, and document matrix. The multiplication of the three matrices is equal to the original matrix. For a word-frequency feature matrix A containing m words and n documents m×n The singular value decomposition of can be expressed as: A=U∑V T , where U is an m-order orthogonal matrix, each of whose column vectors is a left singular vector of A; V is an n-order orthogonal matrix, each of whose columns is a right singular vector of A; Σ is an m×n-order diagonal matrix, which has values ​​in the oblique part and the rest are 0. The elements with values ​​are arranged in descending order, and the diagonal elements are called singular values ​​of A, which are arranged in descending order. The U matrix on the right side of the equation is the topic feature matrix; S3. Neural network fusion and training. The neural network is a multilayer perceptron, comprising an input layer, a hidden layer, and an output layer. The neural network is used to train a text message classifier, which is used to distinguish whether a text message is spam. Each neuron in the input layer of the multilayer perceptron is a binary data set, comprising a topic feature vector and a context feature vector. The binary inputs are linearly combined. The output predicts the classification, and after multiple rounds of iteration, a stable classifier is ultimately obtained. S4. SMS classification and discrimination: A text message of unknown classification is converted into a corresponding topic feature matrix and context feature matrix, which are input into the classifier trained by the multi-layer perceptron to obtain the classification result of the text message. Based on business needs, corresponding processing is performed, such as interception, release, and sending of the text message.

2. The method according to claim 1, wherein After step S1, the TF-IDF frequency calculation is performed. The TF-IDF frequency is used to calculate the scalar value of each element in the matrix, which is expressed as: Among them, tf ij For word w i Appears in text d j The frequency in t f·j is text d j The sum of the frequencies of all words that appear in df i It contains the word w i The number of texts in , df is the total number of texts in the text set D; The TFIDF eigenvalue reflects the importance of a word in a text and the degree to which the word represents the characteristics of the text. The higher the eigenvalue, the more important and representative the word is in the text. The actual expression of the word vector space is that each column vector x j Represents a text d j The inner product or normalized inner product (cosine) of two word vectors represents the semantic similarity between the corresponding texts; the similarity calculation formula is: where · represents the inner product of vectors, ||x i || represents the norm of the vector, that is, the length of the vector. Intuitively, the more words that appear in two texts, the closer their semantic content is. In this case, the larger the inner product of the corresponding word vectors is. The similarity of word vectors constructed based on word-frequency features is used as the basis for measuring text similarity.

3. The method according to claim 2, wherein: In step S2, the skip-gram model of word2vec is used to generate a context feature vector, and the generating of the context feature vector includes the following steps: S21, after the SMS text is segmented, a word set W is formed = {w1, w2, ..., w v }; S22. Perform one-hot encoding on each word in the word set to form a one-hot vector matrix X of the word, where the dimension of each column of the word vector is V; S23. Each column of word vector x[k] is passed to the hidden layer through the neurons of the skip-gram model input layer. The hidden layer performs the dot product operation between the weight vector W[|v|,m] and the input vector x[k]. The value H[1,N] of the hidden layer neurons is directly passed to the output layer. The output layer performs the dot product operation between H[1,N] and W'[N,|v|] and gives the vector U. The probability of each vector is obtained. Using the softmax function, the output vector U is obtained at each iteration. The word with the largest probability is the final result. If the word is within the specified upper and lower bounds, the word with the largest probability is the final result. The predicted word in the text position is wrong, and the backpropagation algorithm can be used to correct the weight vectors W and W'; the above steps are performed for each word w(t) in the dictionary; and each word w(t) will be passed K times; the forward propagation algorithm will be executed |v|*k times in each period of time; the softmax activation function is used to obtain the multi-classification probability of the context of the central word, which is compared with the true value. After multiple rounds of training, the W parameter matrix is ​​continuously corrected to minimize the loss function. Finally, when the model converges, W is the parameter matrix at this time, and the word vector matrix with context features is obtained.

4. The method according to claim 3, wherein: In step S3, the topic features and context features are fused through a neural network, and the input of the two-tuple is linearly combined into: WX+UY, where W is the weight of each column vector of the topic feature matrix X, and U is the weight of each column vector of the context feature matrix Y. The values ​​of W and U range from 0 to 1. The initial value is defined as 0.5, so that before the training starts, the model judges the importance of topic features and context features to be 0.5; The neurons in the hidden layer use [xy] vector and [wy] T The dot product of the vectors is calculated, and the calculated values ​​of each neuron are added together. The calculation method for each neuron in the output layer is to find the joint probability of the values ​​of the neurons in the hidden layer, and then pass it through the softmax function to obtain a multi-classification probability value. The category to which the neuron with the largest probability belongs is the predicted category of the input text message. The error between the predicted classification and the true classification is backpropagated to the hidden layer, and the weight ratio of w and u is corrected. After multiple rounds of iteration, a stable classifier is finally obtained.

5. The method of claim 1 , further comprising: S5. Input the intercepted or released SMS into the SMS training corpus.

6. A device for classifying spam text messages, the device implementing the method for classifying spam text messages according to claim 1, the device comprising: The preprocessing module is used for SMS text preprocessing. The preprocessing module downloads SMS datasets from the communication network as the basic data set for word vector generation. The SMS datasets include classification labels and SMS content. The classification labels include normal, commercial advertising, harassment, fraud, invoice processing, loan processing, and real estate. Each SMS is regarded as a text. The set of all SMS constitutes the text set D = {d1, d2, ..., d n }, there are n texts in the text set; each text message is segmented using the precise mode, stop words are filtered out, and a word set is formed. The words that appear in all texts form a word set W = {w1, w2, ..., w m }; Use the vector space model to form a word-text matrix from the word set: A feature engineering module is used to generate a text message feature vector. The text message feature vector includes a topic feature vector and a context feature vector. The topic feature vector is used to identify semantic expressions including synonyms and polysemous words and is obtained by performing a singular value decomposition of the word-text matrix in step S1. The context feature vector is used to predict the probability of the context based on the central word. A neural network module is used to build and train a neural network. The neural network is a multi-layer perceptron, which includes an input layer, a hidden layer, and an output layer. The neural network is used to train a text message classifier, which is used to distinguish whether a text message is spam. Each neuron in the input layer of the multi-layer perceptron is a binary data tuple, which includes a topic feature vector and a context feature vector. The binary inputs are linearly combined. The output predicts the classification, and after multiple rounds of iteration, a stable classifier is ultimately obtained. The SMS judgment and classification module is used to classify and identify spam SMS. When an SMS message of unknown classification is input, the SMS judgment and classification module forms a corresponding topic feature matrix and context feature matrix, which are input into the classifier trained by the multi-layer perceptron to obtain the classification result of the SMS message. According to business needs, corresponding processing is performed, such as interception, release, and sending of the SMS message.

7. The device according to claim 6, wherein The topic feature vector is obtained by performing singular value decomposition on the word-text vector space matrix; the context feature vector is generated using the skip-gram model of word2vec.

8. An electronic device comprising: A memory, a processor, and a spam message classification program stored in the memory and executable on the processor, wherein the spam message classification program, when executed by the processor, implements the steps of the spam message classification method according to any one of claims 1 to 5.

9. A computer storage medium, wherein: The computer storage medium stores a spam message classification program, which, when executed by a processor, implements the steps of the spam message classification method according to any one of claims 1 to 5.

Images