Text classification method, storage medium and computing device

Abstract

The present disclosure provides a method, storage medium and computing device for classifying text. A method for calculating a dictionary for bad text is provided, and the dictionary is used for classifying text, comprising: performing word segmentation on the bad text, and obtaining a first text composed of words based on the word segmentation result; replacing at least part of the words in the first text with a synonym set of the word to obtain a second text including the synonym set; calculating the feature value of each synonym set in the second text; dividing the second text into multiple windows, and calculating the feature value of each window; performing weighted averaging on the feature value of each synonym set and the mean of the feature value of the window containing the synonym set to obtain a corrected feature value of each synonym set; and determining a dictionary, wherein the dictionary includes the synonym set in the second text and the corrected feature value of each synonym set.

Description

Technical Field

[0001] The present disclosure relates to the field of text detection, and in particular to a method, storage medium and computing device for detecting bad text. Background Art

[0002] The detection of bad websites based on text is generally done by extracting text, segmenting it, extracting keywords from the segmented text using the TF-IDF algorithm, and obtaining keyword weights to obtain text features. After that, a machine learning classification model is used to determine whether the text is good or bad.

[0003] The TF-IDF algorithm calculates scores for similar words separately. Since the frequency of each word is low, the representation of the mined features is not accurate enough. In addition, the large number of words in the dictionary will result in high-dimensional sparse features. Therefore, the standard TF-IDF keyword method of representing text has natural high-dimensional and sparse characteristics. When there are fewer training texts, this method may have the problem of insufficient feature coverage. If a large amount of training text is used to cover a more complete range of words, it will result in more sparse features.

[0004] When judging whether a text is bad text based on features, the contribution of words similar to the features to the classification of the article cannot be obtained.

[0005] Moreover, the standard TFIDF algorithm relies on the frequency of word occurrence and lacks sufficient use of the contextual information of the text.

[0006] Therefore, there is a need to improve the standard TFIDF algorithm to overcome the above defects and make it more suitable for the scenario of bad website detection. Summary of the invention

[0007] According to a first aspect of the present disclosure, a method for calculating a dictionary for bad text is provided, the dictionary being used for text classification, comprising: performing word segmentation on the bad text, and obtaining a first text consisting of words based on the word segmentation result; replacing at least part of the words in the first text with a synonym set of the word to obtain a second text including the synonym set; calculating a feature value of each synonym set in the second text; dividing the second text into a plurality of windows, and calculating a feature value of each window; performing weighted averaging on the feature value of each synonym set and the mean of the feature value of the window containing the synonym set to obtain a corrected feature value of each synonym set; and determining a dictionary, wherein the dictionary includes the synonym set in the second text and the corrected feature value of each synonym set.

[0008] According to a second aspect of the present disclosure, a method for training a classification prediction model for text is provided, comprising: obtaining a plurality of samples, wherein the samples are bad texts; obtaining feature vectors of the plurality of samples using a dictionary calculated according to the method of the first aspect of the present disclosure; and training a classifier using the feature vectors of the plurality of samples to obtain a classification prediction model; wherein obtaining the feature vectors of the plurality of samples comprises performing the following steps on each of the plurality of samples: constructing a feature vector of the sample using the dictionary, wherein the dimension of the feature vector of the sample is equal to the number of synonym sets included in the dictionary, and the feature vector of the sample comprises the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; for each synonym set in the dictionary, determining whether the sample includes words of the synonym set; if the sample includes words of the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is the feature value corresponding to the synonym set in the dictionary; if the sample does not include words of the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is zero.

[0009] According to a third aspect of the present disclosure, a method for classifying text is provided, comprising: obtaining a feature vector of a text to be predicted using a dictionary calculated according to the method described in the first aspect of the present disclosure; and predicting the probability that the text to be predicted is a bad text based on the feature vector of the text to be predicted using a classification prediction model obtained according to the method described in the second aspect; wherein obtaining the feature vector of the text to be predicted comprises the following steps: constructing a feature vector of the text to be predicted using a dictionary, wherein the dimension of the feature vector of the text to be predicted is equal to the number of synonym sets included in the dictionary, and the feature vector of the text to be predicted comprises the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; for each synonym set in the dictionary, determining whether the text to be predicted includes words in the synonym set; if the text to be predicted includes words in the synonym set, the feature values ​​corresponding to the synonym set in the feature vector of the text to be predicted are the feature values ​​corresponding to the synonym set in the dictionary; if the text to be predicted does not include words in the synonym set, the feature values ​​corresponding to the synonym set in the feature vector of the text to be predicted are zero.

[0010] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium for calculating a dictionary for bad text is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the first aspect of the present disclosure.

[0011] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium for training a classification prediction model for text is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the second aspect of the present disclosure.

[0012] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium for text prediction is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the third aspect of the present disclosure.

[0013] According to a seventh aspect of the present disclosure, there is provided a computing device for computing a dictionary for bad texts, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, and a program is stored in the memory, which, when executed by the processor, enables the processor to execute the method according to the first aspect of the present disclosure.

[0014] According to an eighth aspect of the present disclosure, there is provided a computing device for training a classification prediction model for text, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, and a program is stored in the memory, which, when executed by the processor, enables the processor to execute the method according to the second aspect of the present disclosure.

[0015] According to a ninth aspect of the present disclosure, there is provided a computing device for predicting text, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to the third aspect of the present disclosure.

[0016] The present disclosure can more fully mine the features of the text, and use the synonym set and context to modify the features of the text to achieve a more accurate representation of the text, which is beneficial to the classification of the text.

[0017] Other features and advantages of the present disclosure will become more apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0020] Figure 1 is a schematic diagram of a method for calculating a dictionary for bad text according to an embodiment of the present disclosure.

[0021] Figure 2is a flowchart of a method for calculating a dictionary for objectionable text according to an embodiment of the present disclosure.

[0022] Figure 3 It is a flowchart of a method for training a classification prediction model for text according to an embodiment of the present disclosure.

[0023] Figure 4 is a schematic diagram of an exemplary computing device that can be used to implement methods according to embodiments of the present disclosure.

[0024] For ease of understanding, the position, size, range, etc. of each structure shown in the drawings and the like may not represent the actual position, size, range, etc. Therefore, the present disclosure is not limited to the position, size, range, etc. disclosed in the drawings and the like. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0026] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0027] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0028] The detection of bad websites can be achieved by detecting the text content of the website. First, you need to obtain the text content of the website. For example, you can use selenium to obtain the source code of the web page, and then you can extract the text from the source code of the web page. After obtaining the text of the website, you can process the text to obtain features related to bad content.

[0029] Figure 1 is a schematic diagram of a method for calculating a dictionary for bad text according to an embodiment of the present disclosure. The dictionary for bad text can be used to obtain features of bad text, and thus can be used for text classification. Figure 1As shown, obtaining a dictionary may include tokenizing the text, replacing the words in the text with sets of synonyms, calculating the eigenvalue of the set of synonyms, correcting the eigenvalue of the set of synonyms using a sliding window, etc. The following will be described in detail with reference to the flowchart shown in Figure 2 the flowchart shown in

[0030] Figure 2 is a flowchart of a method for calculating a dictionary for bad text according to an embodiment of the present disclosure.

[0031] At S201, the bad text can be tokenized, and a first text composed of words can be obtained based on the tokenization result.

[0032] As Figure 1 shown in p , the text is tokenized to obtain a first text composed of words K1, K2, …, K

[0033] etc. In an embodiment according to the present disclosure, after tokenization, stop words can be removed from the tokenization result, and the remaining words can be retained as valid words so that the first text only includes valid words. Stop words can include words such as "de" (of), "le" (particle indicating completion) that are useless for analyzing the characteristics of the text.

[0034] The set of synonyms of a word can refer to words that are semantically the same and / or similar to the word, and can be不限于 the words included in the first text. For example, a set of synonyms can be constructed using FastText, etc. As Figure 1 shown in p , the words K1, K2, …, K j etc. in the first text can be replaced with the corresponding sets of synonyms A1, A2, …, A j etc. to obtain a second text including the sets of synonyms A1, A2, …, A

[0035] etc. In an embodiment according to the present disclosure, it is not necessary to replace all the words included in the first text with the corresponding sets of synonyms, but only the words related to bad information in the first text are replaced with the sets of synonyms of the words. For example, before replacement, it can be determined whether the word in the first text is a word related to bad information based on a known bad information sample. If the word is related to bad information, the word is replaced with the corresponding set of synonyms, otherwise no replacement of the set of synonyms is performed.

[0036] The TF-IDF algorithm is a statistical analysis method for keywords. TF-IDF can indicate the importance of a word in a corpus. The standard TF-IDF algorithm for a word is as follows:

[0037] tfidf (i，j) =tf (i，j) *idf j ,

[0038] Among them, tfidf (i，j) is the TF-IDF value of word i in text j; tf (i，j) is the TF value of word i in text j, which can represent word frequency; idf j is the IDF value of word i, which can represent the inverse document frequency.

[0039]

[0040] Among them, n (i，j) is the number of times word i appears in text j, is the sum of the number of occurrences of all valid words in text j.

[0041]

[0042] Where |D| is the number of texts in the corpus, |{j:t i ∈d j}| is the number of texts in the corpus that contain word i, and ∈ is a small value.

[0043] In an embodiment according to the present disclosure, the standard TF-IDF algorithm may be improved to calculate the TF-IDF value of a synonym set. The TF-IDF value of a synonym set is calculated as follows:

[0044]

[0045] in, A is the synonym set in text j l The TF-IDF value of A is the synonym set in text j l TF value, A is a synonym set l IDF value of

[0046]

[0047] Among them, n (i，j) is the number of times word i appears in text j, is the sum of the occurrences of valid words in text j, A is a synonym set lThe sum of the TF values ​​of the words in ;

[0048]

[0049] Where |D| is the number of texts in the corpus, |{j:t i ∈d j}| is the number of texts in the corpus that contain word i, A is a synonym set in the corpus l The sum of the number of words in the text, ∈ is a small value.

[0050] By using the TF-IDF algorithm for the synonym set as described above, we can get the following: Figure 1 The vector representing the features of synonyms shown in {(A1, V A1 ), (A2, V A2 ),...,(A m , V Am )}, where V Ak For the synonym set A k The corresponding TF-IDF value. This vector includes the synonym set in the text and the TF-IDF value corresponding to the synonym set.

[0051] At S204 , the second text may be divided into a plurality of windows, and a feature value of each window may be calculated.

[0052] The length of the window can be selected as needed. In an embodiment according to the present disclosure, the length of the window can be fixed, for example, Figure 1 As shown in , every two synonym sets are divided into a window. In other embodiments according to the present disclosure, the length of the window may not be fixed.

[0053] The feature value of each window can be the TF-IDF value of each window. The algorithm is similar to the standard TF-IDF algorithm described above. Each window can be regarded as a word. The TF-IDF value calculated by the standard TF-IDF algorithm is the TF-IDF value of the window. Thus, we can get Figure 1 The vector {(S1, V S1 ), (S2, V S2 ),...,(S t , V St )}, where V Sk For the window S k The corresponding TF-IDF value. This vector includes the window in the text and the TF-IDF value corresponding to the window.

[0054] At S205, the feature value of each synonym set and the mean of the feature value of the window containing the synonym set may be weighted averaged to obtain a modified feature value of each synonym set. l The algorithm for the corrected TF-IDF value is as follows:

[0055]

[0056] in, A is the synonym set in text j l The corrected TF-IDF value of A is the synonym set in text j l TF-IDF value of A is a synonym set contained in text j l The mean of the TF-IDF values ​​of the window; α is a correction coefficient, and its range is 0<α≤1. For example, α=0.8 can be taken.

[0057] At S206 , a dictionary may be determined, the dictionary including the synonym sets in the second text and the corrected feature value of each synonym set.

[0058] In an embodiment according to the present disclosure, the corrected feature values ​​of the synonym set in the second text can be sorted from high to low, and the dictionary only includes the synonym set whose feature values ​​are ranked in the top n positions and the corrected statistical feature values ​​of the synonym set, where n is an integer not less than 2.

[0059] For example, Figure 1 As shown in , we can intercept the synonym set whose feature values ​​are ranked in the first l places, and get the dictionary {(A1, V' A1 ), (A2, V' A2 ),...,(A l , V' Al )}, where V' Ak For the synonym set A k The corresponding corrected TF-IDF value.

[0060] Figure 3 It is a flowchart of a method for training a classification prediction model for text according to an embodiment of the present disclosure.

[0061] At S301 , a plurality of samples may be obtained, wherein the samples are bad texts.

[0062] At S302, the dictionary as described above may be used to obtain feature vectors of multiple samples. Obtaining the feature vectors of multiple samples may include performing the following steps on each of the multiple samples:

[0063] Using the dictionary, constructing a feature vector of the sample, wherein the dimension of the feature vector of the sample is equal to the number of synonym sets included in the dictionary, and the feature vector of the sample includes the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets;

[0064] For each synonym set in the dictionary, determining whether the sample includes a word in the synonym set;

[0065] If the sample includes words in the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is the feature value corresponding to the synonym set in the dictionary;

[0066] If the sample does not include words in the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is zero.

[0067] For example, you can use Figure 1 The dictionary in to obtain the feature vectors of multiple samples. Figure 1 The dictionary in {(A1,V' A1 ),(A2,V' A2 ),…,(A l ,V' Al )} is l, then the feature vector of the bad text is an l-dimensional vector. For example, for a sample x in multiple samples, we can construct an l-dimensional vector {(A1,X1),(A2,X2),…,(A l ,X l )}. If the sample x includes the synonym set A k In the word, then X k =V' Ak Otherwise, X k = 0. By analogy, we can obtain the eigenvector of sample x {(A1,X1),(A2,X2),…,(A l ,X l )}.

[0068] At S303, the feature vectors of multiple samples can be used to train a classifier to obtain a classification prediction model. In an embodiment according to the present disclosure, the classifier can be an SVM classifier. In other embodiments according to the present disclosure, other classifiers can also be used, such as XGBoost, feedforward neural networks, etc.

[0069] After obtaining the trained classification prediction model for text, the text to be predicted can be classified to determine whether the text to be predicted is a bad text.

[0070] The dictionary described above can be used to obtain the feature vector of the text to be predicted. The specific steps can be similar to the steps of constructing the feature vector of the sample described above. For example, Figure 1 The dictionary in is used to obtain the feature vector of the text to be predicted. Figure 1 The dictionary in {(A1,V' A1 ),(A2,V' A2 ),…,(A l ,V' Al )} is l, then the feature vector of the text to be predicted is a l-dimensional vector. For example, for the text y to be predicted, we can construct an l-dimensional vector {(A1,Y1),(A2,Y2),…,(A l ,Y l )}. If the text to be predicted y includes the synonym set A k If the word in k =V' Ak Otherwise, Y k =0. By analogy, we can obtain the feature vector {(A1,Y1),(A2,Y2),…,(A l ,Y l )}.

[0071] The classification prediction model trained as described above can be used to calculate the probability that the text to be predicted is a bad text based on the feature vector of the text to be predicted.

[0072] After calculating the probability that the text to be predicted is a bad text, the text to be predicted can be classified. In an embodiment according to the present disclosure, it can be determined whether the probability that the text to be predicted is a bad text is greater than a threshold; if the probability that the text to be predicted is a bad text is greater than the threshold, the text to be predicted is determined to be a bad text. For example, the threshold can be 0.8, and if the probability that the text to be predicted is a bad text is greater than 0.8, the text to be predicted is determined to be a bad text.

[0073] Figure 4 An exemplary computing device that can be used to implement the method according to an embodiment of the present disclosure is shown. Figure 4 As shown, computing device 400 may include one or more processors 402. Processor(s) 402 may be any kind of processor and may include, but are not limited to, one or more general purpose processors or special purpose processors (such as a dedicated processing chip).

[0074] The computing device 400 may also include or be connected to a non-volatile storage device 404, which may be any storage device that is non-volatile and can enable data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a compact disk or any other optical medium, a cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code.

[0075] The processor 402 and / or the storage device 404 may be connected or communicated with the bus 406 via one or more interfaces. The bus 406 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, etc.

[0076] I / O device 408 may be any device that enables a user to interact with computing device 400. Examples of I / O device 408 may include, but are not limited to, a keyboard, a trackpad, a mouse, a joystick or other pointing device, a microphone, a speaker, and a display.

[0077] The computing device 400 may also include a network interface 410. The network interface 410 may be any type of device or system capable of enabling communication with an external device and / or a network, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset (such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication facility, etc.).

[0078] I / O devices 408 and / or network interface 410 may also be communicatively coupled with processor 402 and / or storage device 404 via bus 406 .

[0079] Various aspects, embodiments, implementations or features of the foregoing embodiments may be used alone or in any combination. Various aspects of the foregoing embodiments may be implemented by software, hardware or a combination of hardware and software.

[0080] For example, the foregoing embodiments may be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer readable media include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard drive, solid state drive, and optical data storage device. The computer readable medium may also be distributed among network coupled computer systems so that the computer readable code is stored and executed in a distributed manner.

[0081] For example, the aforementioned embodiments may be in the form of hardware circuits. The hardware circuits may include any combination of combinational logic circuits, clock storage devices (such as floppy disks, flip-flops, latches, etc.), finite state machines, memories such as static random access memories or embedded dynamic random access memories, custom designed circuits, programmable logic arrays, etc.

[0082] In one embodiment, hardware circuits according to the present disclosure may be implemented by coding and designing one or more integrated circuits using a hardware description language (HDL) such as Verilog or VHDL, or in combination using discrete circuits.

[0083] The present disclosure can achieve feature dimensionality reduction while covering more text information and utilizing context information, so as to more accurately understand and characterize the bad information text; and the improved features can be used to train a classifier model, so as to achieve more accurate classification of the predicted text.

[0084] In summary, according to the first aspect of the present disclosure, a method for calculating a dictionary for bad text is provided, and the dictionary is used for text classification, including: performing word segmentation on the bad text, and obtaining a first text composed of words based on the word segmentation result; replacing at least part of the words in the first text with a synonym set of the word to obtain a second text including a synonym set; calculating a feature value of each synonym set in the second text; dividing the second text into multiple windows, and calculating a feature value of each window; performing weighted averaging on the feature value of each synonym set and the mean of the feature value of the window containing the synonym set to obtain a corrected feature value of each synonym set; and determining a dictionary, wherein the dictionary includes the synonym set in the second text and the corrected feature value of each synonym set.

[0085] In some embodiments, obtaining the first text composed of words based on the word segmentation result further includes: removing stop words from the word segmentation result and retaining the remaining words as valid words, so that the first text only includes valid words.

[0086] In some embodiments, the feature value is a TF-IDF value; wherein the TF-IDF value of the synonym set is calculated as follows:

[0087]

[0088] in, is the TF-IDF value of synonym set Al in text j, is the TF value of synonym set Al in text j, is the IDF value of synonym set Al;

[0089]

[0090] Where n(i, j) is the number of times word i appears in text j. is the sum of the occurrences of valid words in text j, is the sum of the TF values ​​of the words in the synonym set Al;

[0091]

[0092] Where |D| is the number of texts in the corpus, |{j:t i ∈d j}| is the number of texts in the corpus that contain word i, It is the sum of the number of texts in the corpus that contain words in the synonym set Al, and ∈ is a small value.

[0093] In some embodiments, replacing at least part of the words in the first text with a set of synonyms for the words includes: replacing only the words in the first text that are related to bad information with a set of synonyms for the words, wherein before the replacement, it is determined whether the words in the first text are words related to bad information based on known bad information samples.

[0094] In some embodiments, the method further includes: sorting the corrected feature values ​​of the synonym set in the second text from high to low, and the dictionary only includes the synonym set whose feature values ​​are ranked in the top n positions and the corrected statistical feature values ​​of the synonym set, where n is an integer not less than 2.

[0095] According to a second aspect of the present disclosure, a method for training a classification prediction model for text is provided, comprising: obtaining a plurality of samples, wherein the samples are bad texts; obtaining feature vectors of the plurality of samples using a dictionary calculated according to the method of the first aspect of the present disclosure; and training a classifier using the feature vectors of the plurality of samples to obtain a classification prediction model; wherein obtaining the feature vectors of the plurality of samples comprises performing the following steps on each of the plurality of samples: constructing a feature vector of the sample using the dictionary, wherein the dimension of the feature vector of the sample is equal to the number of synonym sets included in the dictionary, and the feature vector of the sample comprises the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; for each synonym set in the dictionary, determining whether the sample includes words of the synonym set; if the sample includes words of the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is the feature value corresponding to the synonym set in the dictionary; if the sample does not include words of the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is zero.

[0096] In some embodiments, the classifier is a svm classifier.

[0097] According to a third aspect of the present disclosure, a method for classifying text is provided, comprising: obtaining a feature vector of a text to be predicted using a dictionary calculated according to the method described in the first aspect of the present disclosure; and predicting the probability that the text to be predicted is a bad text based on the feature vector of the text to be predicted using a classification prediction model obtained according to the method described in the second aspect; wherein obtaining the feature vector of the text to be predicted comprises the following steps: constructing a feature vector of the text to be predicted using a dictionary, wherein the dimension of the feature vector of the text to be predicted is equal to the number of synonym sets included in the dictionary, and the feature vector of the text to be predicted comprises the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; for each synonym set in the dictionary, determining whether the text to be predicted includes words in the synonym set; if the text to be predicted includes words in the synonym set, the feature values ​​corresponding to the synonym set in the feature vector of the text to be predicted are the feature values ​​corresponding to the synonym set in the dictionary; if the text to be predicted does not include words in the synonym set, the feature values ​​corresponding to the synonym set in the feature vector of the text to be predicted are zero.

[0098] In some embodiments, the method further includes: determining whether the probability that the text to be predicted is bad text is greater than a threshold; if the probability that the text to be predicted is bad text is greater than the threshold, determining that the text to be predicted is bad text.

[0099] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium for calculating a dictionary for bad text is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the first aspect of the present disclosure.

[0100] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium for training a classification prediction model for text is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the second aspect of the present disclosure.

[0101] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium for text prediction is provided, on which a program is stored, and when the program is executed by a computer, the computer executes the method according to the third aspect of the present disclosure.

[0102] According to a seventh aspect of the present disclosure, there is provided a computing device for computing a dictionary for bad texts, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, and a program is stored in the memory, which, when executed by the processor, enables the processor to execute the method according to the first aspect of the present disclosure.

[0103] According to an eighth aspect of the present disclosure, there is provided a computing device for training a classification prediction model for text, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, and a program is stored in the memory, which, when executed by the processor, enables the processor to execute the method according to the second aspect of the present disclosure.

[0104] According to a ninth aspect of the present disclosure, there is provided a computing device for predicting text, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to the third aspect of the present disclosure.

[0105] Although some specific embodiments of the present disclosure have been shown in detail by examples, it should be understood by those skilled in the art that the above examples are intended to be illustrative only and do not limit the scope of the present disclosure. It should be appreciated that some steps in the aforementioned method are not necessarily performed in the order shown in the figure, but they can be performed simultaneously, in different orders or in an overlapping manner. In addition, those skilled in the art can add some steps or omit some steps as needed. Some components in the aforementioned system are not necessarily arranged according to the diagram, and those skilled in the art can add some components or omit some components as needed. It should be understood by those skilled in the art that the above embodiments can be modified without departing from the scope and essence of the present disclosure. The scope of the present disclosure is defined by the attached claims.

Claims

1. A method for calculating a dictionary for bad text, wherein the dictionary is used for text classification, comprising: Segmenting the bad text into words, and obtaining a first text composed of words based on the segmentation result; Replacing at least part of the words in the first text with a synonym set of the words to obtain a second text including the synonym set; Calculate the feature value of each synonym set in the second text; Divide the second text into a plurality of windows, and calculate a feature value of each window; Taking a weighted average of the feature value of each synonym set and the mean of the feature value of the window containing the synonym set to obtain a modified feature value of each synonym set; determining a dictionary, wherein the dictionary includes a synonym set in the second text and a modified feature value of each synonym set; Among them, the feature value is the TF-IDF value.

2. The method according to claim 1, wherein: The first text composed of words obtained based on the word segmentation result also includes: Stop words are removed from the word segmentation results and the remaining words are retained as valid words, so that the first text only includes valid words.

3. The method according to claim 2, wherein: in, The TF-IDF value of a synonym set is calculated as follows: in, A is the synonym set in text j l The TF-IDF value of A is the synonym set in text j l TF value, A is a synonym set l IDF value of Among them, n (i,j) is the number of times word i appears in text j, is the sum of the occurrences of valid words in text j, A is a synonym set l The sum of the TF values ​​of the words in ; Where |D| is the number of texts in the corpus, |{j:t i ∈d j }| is the number of texts in the corpus that contain word i, A is a synonym set in the corpus l The sum of the number of words in the text, ∈ is a small value.

4. The method according to claim 1, wherein: Replacing at least part of the words in the first text with a synonym set of the word includes: Only replace the words related to bad information in the first text with the synonym set of the word, Before the replacement, it is determined whether the words in the first text are words related to the bad information based on the known bad information samples.

5. The method according to claim 1, wherein: The method further includes: sorting the corrected feature values ​​of the synonym sets in the second text from high to low, The dictionary only includes the synonym sets whose feature values ​​are ranked in the top n and the corrected statistical feature values ​​of the synonym sets. Here, n is an integer not less than 2.

6. A method for training a classification prediction model for text, comprising: Acquire multiple samples, wherein the samples are bad texts; Obtaining feature vectors of the plurality of samples using a dictionary calculated according to the method described in any one of claims 1 to 5; and Using the feature vectors of the multiple samples to train a classifier to obtain a classification prediction model; Wherein, obtaining the feature vectors of the multiple samples comprises performing the following steps on each of the multiple samples: Using the dictionary, constructing a feature vector of a sample, wherein the dimension of the feature vector of the sample is equal to the number of synonym sets included in the dictionary, and the feature vector of the sample includes the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; For each synonym set in the dictionary, determining whether the sample includes a word in the synonym set; If the sample includes words of the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is the feature value corresponding to the synonym set in the dictionary; If the sample does not include words in the synonym set, the feature value corresponding to the synonym set in the feature vector of the sample is zero.

7. The method according to claim 6, wherein: The classifier is a svm classifier.

8. A method for classifying text, comprising: Obtaining a feature vector of the text to be predicted using the dictionary calculated according to the method described in any one of claims 1 to 5; as well as Using the classification prediction model obtained by the method according to any one of claims 6-7, predicting the probability that the text to be predicted is a bad text based on the feature vector of the text to be predicted; Wherein, obtaining the feature vector of the text to be predicted includes the following steps: Using the dictionary, constructing a feature vector of the text to be predicted, wherein the dimension of the feature vector of the text to be predicted is equal to the number of synonym sets included in the dictionary, and the feature vector of the text to be predicted includes the synonym sets in the dictionary and feature values ​​corresponding to the synonym sets; For each synonym set in the dictionary, determining whether the text to be predicted includes a word in the synonym set; If the text to be predicted includes words in the synonym set, the feature value corresponding to the synonym set in the feature vector of the text to be predicted is the feature value corresponding to the synonym set in the dictionary; If the text to be predicted does not include words in the synonym set, the feature value corresponding to the synonym set in the feature vector of the text to be predicted is zero.

9. The method according to claim 8, further comprising: Determining whether the probability that the text to be predicted is bad text is greater than a threshold; If the probability that the text to be predicted is bad text is greater than a threshold, the text to be predicted is determined to be bad text.

10. A non-transitory computer-readable storage medium for calculating a dictionary for objectionable texts, on which a program is stored, which, when executed by a computer, causes the computer to execute the method according to any one of claims 1 to 5.

11. A non-transitory computer-readable storage medium for training a classification prediction model for text, having a program stored thereon, which, when executed by a computer, causes the computer to execute the method according to claim 6 or 7.

12. A non-transitory computer-readable storage medium for prediction of text, having a program stored thereon, which, when executed by a computer, causes the computer to execute the method according to claim 8 or 9.

13. A computing device for calculating a dictionary for objectionable texts, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to any one of claims 1 to 5.

14. A computing device for training a classification prediction model for text, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, and a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to claim 6 or 7.

15. A computing device for text prediction, comprising a memory and a processor, wherein the memory is communicatively coupled to the processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the method according to claim 8 or 9.

Images