Method and device for determining abnormal short text and electronic equipment

Abstract

The invention provides a method for determining an abnormal short text, and relates to the technical field of information processing, in particular to the technical field of text recognition, abnormal short text detection and abnormal user name mining. According to the specific implementation scheme, word vectors of all short texts are determined respectively; obtaining a common weight of each short text based on the word vector of each short text; for each short text, determining a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type; and according to each target short text and the corresponding common weight, performing clustering to obtain a plurality of short text clusters, and determining an abnormal short text based on the plurality of short text clusters.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to the technical fields of text recognition, abnormal short text detection, and abnormal user name mining. Background Art

[0002] In some scenarios, malicious machine behavior will generate abnormal short texts in batches, interfering with network security or generating risky behaviors. For example, when machines register accounts in batches, they will randomly generate user names, which serve as a type of short text.

[0003] Since short texts are usually randomly generated and have no semantic meaning, text recognition is difficult and abnormal short texts cannot be recognized. Summary of the invention

[0004] The present disclosure provides a method, device, and electronic device for determining abnormal short texts to solve at least one of the above technical problems.

[0005] According to one aspect of the present disclosure, a method for determining abnormal short text is provided, wherein the method comprises:

[0006] Determine the word vector for each short text separately;

[0007] Based on the word vectors of each of the short texts, obtaining the commonality weight of each of the short texts;

[0008] For each of the short texts, determining a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type;

[0009] Clustering is performed according to each of the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

[0010] According to another aspect of the present disclosure, a device for determining abnormal short text is provided, wherein the device comprises:

[0011] A determination module is used to determine the word vector of each short text;

[0012] A commonality module, used for obtaining a commonality weight of each of the short texts based on the word vectors of each of the short texts;

[0013] A structure module, used for determining, for each of the short texts, a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type;

[0014] The clustering module is used to cluster the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and determine abnormal short texts based on the plurality of short text clusters.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a method for determining abnormal short texts provided by the first embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of another method for determining abnormal short texts provided by the first embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of a method for determining abnormal short texts provided by the second embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of another method for determining abnormal short texts provided by the second embodiment of the present disclosure;

[0027] Figure 5 is a structural schematic diagram of a device for determining abnormal short text provided by the third embodiment of the present disclosure;

[0028] Figure 6 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0031] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0032] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0034] The method for determining abnormal short text according to the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a display device (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method for determining abnormal short text provided by the present disclosure can be executed by a server.

[0035] Short text generally refers to text with a short length and a small number of characters. In some examples, the number of characters in a short text does not exceed 60. Short text includes, for example, one of the following: user name, title, question text, chat information, etc., which are not limited here.

[0036] Short texts usually have the following characteristics: few characteristic words, making it difficult to extract potential semantic association relationships; being non-standard, with inconsistent representation methods and structures; and having a huge data scale, making it difficult to conduct classification learning in a short time.

[0037] In network activities, some malicious machine behaviors may generate short texts in batches, interfering with network security, but it is difficult to identify them. For example, machines register accounts in batches and generate usernames for each account. These short texts of username types may have the same structure. For example, they all adopt the structure of characters at the head, Chinese in the middle, and letters at the tail, while the semantic content is inconsistent. Due to the shortness and randomness of the text, it is difficult to be detected. To solve at least the above problems, the present disclosure provides a method for determining abnormal short texts, as detailed below.

[0038] In the first embodiment of the disclosure, refer to Figure 1 , Figure 1 which shows a schematic flowchart of a method for determining abnormal short texts provided by the first embodiment of the present disclosure. The method includes:

[0039] S101. Determine the word vectors of each short text respectively.

[0040] Among them, the word vector refers to a distributed representation of a word. Its basic idea is to map a word to a vector with a fixed dimension, and the dimension is much smaller than the size of the dictionary. The vectors of these words constitute the word vector semantic space, and words with similar semantics are usually closer in space.

[0041] In S101, through various language models, the short text can be converted into word vectors. The semantic models include at least one of the following: the word to vector (word2vec) model, the Global Vectors for Word Representation (glove) model, the Embeddings from Language Models (ELMo) model, the Bidirectional Encoder Representation from Transformers (BERT) model, etc. In this specification, the example of using the word2vec model to convert the short text into word vectors is used for illustration, and it is not limited here.

[0042] In some examples, each character in the short text can also be converted into a character vector first, and then the word vectors of each word included in the short text can be obtained according to the character vectors, which is not limited here.

[0043] S102: Based on the word vectors of each short text, obtain the commonality weight of each short text.

[0044] The commonality weight of a short text represents the similarity of the characters contained in the short text. Comparing the commonality weights of different short texts can represent the similarity of different short texts. For short texts, the commonality weight can be determined based on every two adjacent characters in the short text, so that the semantic understanding of the short text can be refined to between characters, overcoming the problem that it is difficult to extract semantic features from short texts due to their short text length. Specifically, the larger the commonality weight, the greater the distance between the word vectors of the short text, the lower the similarity, and the smaller the probability of being an abnormal short text.

[0045] S103: For each short text, determine the structure type contained in the short text, and convert the short text into a target short text expressed based on the structure type.

[0046] Among them, the structural type of the short text represents the type of characters that constitute the short text. Specifically, the set of structural types includes at least one of the following: Chinese structural type, letter structural type, number structural type, symbol structural type, etc., which are not limited here. Assuming that short text a is: xxx Chinese 123, and short text b is: yes &, then short text a contains three structural types: xxx-letter structural type, Chinese-Chinese structural type, 123-number structural type; short text b contains two structural types: yes-Chinese structural type, &-symbol structural type. Because in the malicious behavior of some machines, such as batch registration behavior, since the machine generates short texts according to certain rules, the structure of the short texts generated by the same machine is usually similar. Therefore, S103 is used to parse the structural type contained in each short text in order to identify abnormal short texts.

[0047] S104 , clustering is performed according to each target short text and its corresponding commonality weight to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

[0048] In this process, the commonality weights and structural types of each short text are introduced for clustering, so that each short text cluster in the obtained multiple short text clusters indicates that at least one of the structural and semantic commonalities of the short texts is similar, and thus they may be generated by the same machine. Therefore, some short texts belonging to the same short text cluster can be regarded as abnormal short texts, thereby realizing the detection of abnormal short texts.

[0049] The short text is original short text data, and the target short text represents a target short text expressed based on a structure type, that is, a short text carrying structure type information.

[0050] In some examples, in S104, determining abnormal short texts based on multiple short text clusters includes:

[0051] Step 1: Compare the density of multiple short text clusters with the preset threshold.

[0052] Step 2: Determine the short texts contained in the short text clusters whose density is greater than a preset threshold as abnormal short texts.

[0053] The preset threshold is a threshold for determining whether the density of the short text cluster (ie, the similarity of the included short texts) is abnormal, and can be set as needed, and is not limited here.

[0054] After completing the short text clustering, multiple short text clusters are obtained. The short text cluster with a density greater than the preset threshold is screened out. It is considered that the similarity of the short texts in this short text cluster is too high, and it is likely that the short texts are generated by the batch generation behavior of the same machine. Therefore, the short texts contained in this short text cluster are regarded as abnormal short texts.

[0055] Among them, the execution order of S102 and S103 can be swapped, and there is no limitation here.

[0056] For some examples, see Figure 2 , Figure 2 A flowchart diagram of another method for determining abnormal short text is shown. After S102 and before S104, the present disclosure further includes:

[0057] S101', respectively determine the word vector of each short text.

[0058] S102', based on the word vectors of each short text, obtain the commonality weight of each short text.

[0059] S103', determining the rare characters contained in each short text, and determining the rare character weight of each short text based on the contained rare characters.

[0060] S104': for each short text, determine the structure type contained in the short text, and convert the short text into a target short text expressed based on the structure type.

[0061] S105', clustering is performed according to each target short text, its corresponding commonality weight and uncommon character weight to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

[0062] In some examples, as shown in S103', the recognition of uncommon characters can also be introduced. Specifically, an uncommon character text library can be pre-established, and based on the uncommon characters contained in the uncommon character text library, similarity search is performed with each short text to determine the uncommon characters contained in the short text, so as to obtain the uncommon character weight of each short text according to the number of uncommon characters contained. Specifically, the greater the uncommon character weight, the greater the number of uncommon characters in the short text, and the greater the probability of being an abnormal short text.

[0063] Based on the above, in S105', when clustering each target short text, the commonality weight and uncommon character weight corresponding to each target short text are introduced, and clustering calculation is performed together. The short text clustering cluster obtained is a clustering cluster based on three factors: structural similarity, commonality, and containing uncommon characters. It can accurately cluster short texts, and then determine abnormal short texts based on the accurately clustered short texts, which can ensure the accuracy of abnormal short text detection.

[0064] It should be noted that the execution order of S102'-S104' can be interchanged, which is not limited here. S105' is an implementation method of S104, and there are other implementation methods of S104, which are not limited here.

[0065] The method provided by the present invention determines the word vectors of each short text to be identified, obtains the commonality weight of each short text based on the word vectors of each short text, then determines the structural type contained in the short text, and converts the short text into a target short text expressed based on the structural type, and finally clusters the target short texts and their corresponding commonality weights to obtain multiple short text clustering clusters, and determines abnormal short texts based on the multiple short text clustering clusters, so that when clustering, the commonality weights of the word vectors of the short texts are introduced to enhance the association of the semantic commonality of the short texts; and, the short texts are divided according to the structural types. Since short texts are usually generated in batches and have similar structures, the structural similarity of the short texts can be identified in this way. Finally, clustering is performed based on the semantic commonality and structural similarity, so that the special effects of the short texts can be accurately targeted, and accurate short text clustering clusters can be obtained, and ultimately accurate detection of abnormal short texts can be achieved.

[0066] In disclosing the second embodiment, see Figure 3 , Figure 3 A flow chart of a method for determining abnormal short text according to the second embodiment of the present disclosure is shown. The method comprises:

[0067] S201, respectively determine the word vector of each short text.

[0068] Among them, word vector refers to a distributed expression of words. Its basic idea is to map words into a vector of fixed dimension, whose dimension is much smaller than the dictionary size. The vectors of these words constitute the word vector semantic space. Words with similar semantics are usually close in space.

[0069] In S101, the short text can be converted into a word vector through various language models. The semantic model includes, for example, at least one of the following: word2vec model, glove model, ELMo model, BERT model, etc. This specification uses the word2vec model to convert short text into a word vector as an example, and no limitation is made here.

[0070] In this case, each word in the short text may be converted into a word vector first, and then the word vector of each word included in the short text may be obtained according to the word vector, which is not limited here.

[0071] S202: For each short text, determine the vector distance between every two adjacent characters.

[0072] S203: Obtain the vector distance of the short text according to the vector distance between every two adjacent characters.

[0073] In some examples, S203 includes:

[0074] Add the vector distances of every two adjacent characters to get the vector distance of the short text.

[0075] The commonality weight of a short text represents the similarity of the characters contained in the short text. Comparing the commonality weights of different short texts can represent the similarity of different short texts. For short texts, the commonality weight can be determined based on every two adjacent characters in the short text, so that the semantic understanding of the short text can be refined to the level between characters, overcoming the problem that it is difficult to extract semantic features from short texts due to their shortness.

[0076] In some examples, each word in the short text may be converted into a word vector first, and then a word vector of each word included in the short text may be obtained based on the word vector, which is not limited here.

[0077] The vector distance between two adjacent characters may be, for example, a cosine distance, etc., which is not limited here.

[0078] Decompose the short text into each character, determine the vector distance between each two adjacent characters, that is, determine the semantic relevance between each two words, and then add up all the vector distances of a short text to get the commonality weight of the short text. Specifically, the greater the commonality weight, the greater the distance between the word vectors of the short text, the lower the similarity, and the lower the probability of being an abnormal short text.

[0079] It should be noted that S202 and S203 are an implementation manner of S102, and there are other implementation manners of S102, which are not limited herein.

[0080] S204. For each short text, determine the structural type included in the short text.

[0081] S205. According to the included structural type, divide the short text into multiple sub-texts of different structural types, and determine the arrangement order of each sub-text.

[0082] S206. According to the multiple sub-texts and the arrangement order of each sub-text, splice the multiple sub-texts into a target short text expressed based on the structural type.

[0083] Wherein, the short text is the original short text data, and the target short text represents a target short text expressed based on the structural type, that is, a short text carrying structural type information.

[0084] Wherein, the structural type of the short text represents the type of characters constituting the short text. Specifically, the set of structural types includes at least one of the following: Chinese structural type, letter structural type, digital structural type, symbol structural type, etc., which are not limited herein. In some malicious behaviors of machines, such as batch registration behaviors, since the machine generates short texts according to certain rules, the structures of short texts generated by the same machine are usually similar. Therefore, S204 - S206 are used to analyze the structural type included in each short text to identify abnormal short texts.

[0085] In some examples, a classification model can be used to identify the structural type included in the short text and obtain the arrangement order of each structural type in the short text. For example, assuming the content of the short text is: wer three two *, the result after S204 processing indicates that the short text includes: wer - letter structural type - arrangement order 1; three two - Chinese structural type - arrangement order 2; * - character structural type - arrangement order 3. In S205, the complete short text "wer three two *" is split into three sub-texts "wer", "three two", and "*", and the arrangement order of the three sub-texts is recorded. In S206, the short text "wer three two *" is converted into a target short text expressed based on the structural type. The target short text expressed based on the structural type represents a short text that can be recognized by a computer and carries the sub-texts of the short text and the structural types of the sub-texts. Specifically, the form of the target short text expressed based on the structural type can be various. Exemplarily, it can be expressed as "wer / three two / *", and the delimiter " / " is used to identify different structural types, and the identifier of the structural type can be carried behind each structural type, which is not limited herein.

[0086] It should be noted that S204-S206 is one implementation of S103, and there are other implementations of S103, which are not limited here.

[0087] S207 , respectively calculating the first short text similarity between every two target short texts.

[0088] Based on the clustering algorithm, the first short text similarity between any two target short texts is obtained. Since the target short texts are represented based on the structural type, the first short text similarity includes the similarity of the structural type of the short texts.

[0089] Specifically, the distance between any two target short texts can be used as the first short text similarity. The distance includes at least one of the following: Euclidean distance, Manhattan distance, Mahalanobis distance, cosine distance, Hamming distance, Chebyshev distance, Minkowski distance, etc., which are not limited here.

[0090] S208 : Calculate the second short text similarity between every two target short texts according to the first short text similarity between every two target short texts and the corresponding commonality weight.

[0091] The commonality weights of every two target short texts are introduced and combined with the first short text similarity to obtain the second short text similarity including the similarity of the structural types of the short texts and the similarity of the semantic commonality.

[0092] In some examples, S208 includes:

[0093] The second short text similarity is obtained according to the sum of the first short text similarity and the corresponding commonality weight between every two target short texts.

[0094] Specifically, the first short text similarity and the commonality weight between the two target short texts may be added together to obtain the second short text similarity.

[0095] S209 , clustering is performed according to the second short text similarity between every two target short texts to obtain a plurality of short text clusters.

[0096] The definition of clustering is: for a sample set consisting of multiple short texts, the samples are classified according to the similarity of the second short texts between the samples. Similar ones are classified into one category to obtain a short text cluster, and dissimilar ones are classified into another category. This classification is called clustering, also known as unsupervised classification.

[0097] Specifically, after obtaining the second short text similarity, the second short text similarity of the two target short texts represents the structural similarity and semantic commonality of the two target short texts. The greater the second short text similarity of the two target short texts and the closer the distance, the more likely they are to be clustered in one short text cluster.

[0098] Specifically, a variety of clustering algorithms may be used for clustering, including at least one of the following, for example: a K-means clustering algorithm, a mean shift clustering algorithm, a hierarchical clustering algorithm, a composite clustering algorithm, etc., which are not limited here.

[0099] S210: Determine abnormal short texts based on multiple short text clusters.

[0100] In some examples, in S210, determining abnormal short texts based on multiple short text clusters includes:

[0101] Step 1: Compare the density of multiple short text clusters with the preset threshold.

[0102] Step 2: Determine the short texts contained in the short text clusters whose density is greater than a preset threshold as abnormal short texts.

[0103] The preset threshold is a threshold for determining whether the density of the short text cluster (ie, the similarity of the included short texts) is abnormal, and can be set as needed, and is not limited here.

[0104] After completing the short text clustering, multiple short text clusters are obtained. The short text cluster with a density greater than the preset threshold is screened out. It is considered that the similarity of the short texts in this short text cluster is too high, and it is likely that the short texts are generated by the batch generation behavior of the same machine. Therefore, the short texts contained in this short text cluster are regarded as abnormal short texts.

[0105] It should be noted that S207-S210 is one implementation of S104, and there are other implementations of S104, which are not limited here.

[0106] See also Figure 4 , Figure 4 A flow chart of a method for determining abnormal short text according to the second embodiment of the present disclosure is shown. Based on the existence of S103' of the first embodiment of the disclosure, that is, the recognition of uncommon characters is introduced, after S206, the method includes:

[0107] S301 : Calculate the first short text similarity between every two target short texts.

[0108] S302 , calculating the second short text similarity between every two target short texts according to the first short text similarity between every two target short texts, the corresponding commonality weight and the corresponding uncommon character weight.

[0109] Specifically, a rare character text library can be established in advance, and based on the rare characters contained in the rare character text library, a similarity search can be performed with each short text to determine the rare characters contained in the short text, so as to obtain the rare character weight of each short text according to the number of rare characters contained. Specifically, the larger the rare character weight, the greater the number of rare characters contained in the short text, and the greater the probability of being an abnormal short text.

[0110] In S302, the commonality weight and uncommon word weight of every two target short texts are introduced, and combined with the first short text similarity, a second short text similarity including the similarity of the structural type of the short texts, the similarity of the semantic commonality, and the content of uncommon words is obtained.

[0111] In some examples, S302 includes:

[0112] The second short text similarity is obtained according to the sum of the first short text similarity, the corresponding commonality weight and the inverse of the corresponding uncommon character weight between every two target short texts.

[0113] Specifically, the first short text similarity, the commonality weight, and the inverse of the uncommon character weight between the two target short texts may be added together to obtain the second short text similarity.

[0114] S303 , clustering is performed according to the second short text similarity between every two target short texts to obtain a plurality of short text clusters.

[0115] The definition of clustering is: for a sample set consisting of multiple short texts, the samples are classified according to the similarity of the second short texts between the samples. Similar ones are classified into one category to obtain a short text cluster, and dissimilar ones are classified into another category. This classification is called clustering, also known as unsupervised classification.

[0116] Specifically, after obtaining the second short text similarity, the second short text similarity of the two target short texts represents the structural similarity, semantic commonality and rare character content of the two target short texts. The greater the second short text similarity of the two target short texts and the closer the distance, the more likely they are to be clustered in a short text cluster.

[0117] Specifically, a variety of clustering algorithms may be used for clustering, including, for example, at least one of the following: a K-means clustering algorithm, a mean shift clustering algorithm, a hierarchical clustering algorithm, a composite clustering algorithm, etc., which are not limited here.

[0118] S304: Determine abnormal short texts based on multiple short text clusters.

[0119] In some examples, in S304, determining abnormal short texts based on multiple short text clusters includes:

[0120] Step 1: Compare the density of multiple short text clusters with the preset threshold.

[0121] Step 2: Determine the short texts contained in the short text clusters whose density is greater than a preset threshold as abnormal short texts.

[0122] The preset threshold is a threshold for determining whether the density of the short text cluster (ie, the similarity of the included short texts) is abnormal, and can be set as needed, and is not limited here.

[0123] After completing the short text clustering, multiple short text clusters are obtained. The short text cluster with a density greater than the preset threshold is screened out. It is considered that the similarity of the short texts in this short text cluster is too high, and it is likely that the short texts are generated by the batch generation behavior of the same machine. Therefore, the short texts contained in this short text cluster are regarded as abnormal short texts.

[0124] In the third embodiment disclosed, based on Figure 1 The same principle, Figure 5 A device 50 for determining abnormal short text provided by the third embodiment of the present disclosure is shown, and the device includes:

[0125] A determination module 501 is used to determine the word vector of each short text respectively;

[0126] A commonality module 502, used to obtain a commonality weight of each short text based on the word vector of each short text;

[0127] A structure module 503 is used to determine, for each short text, a structure type contained in the short text, and convert the short text into a target short text expressed based on the structure type;

[0128] The clustering module 504 is used to cluster the target short texts and their corresponding commonality weights to obtain a plurality of short text clusters, and determine abnormal short texts based on the plurality of short text clusters.

[0129] In some examples, the common modules are specifically used to:

[0130] For each short text, determine the vector distance between every two adjacent characters;

[0131] According to the vector distance between each two adjacent characters, the vector distance of the short text is obtained.

[0132] In some examples, when the commonality module obtains the vector distance of the short text according to the vector distance of each two adjacent characters, it is used to:

[0133] Add the vector distances of every two adjacent characters to get the vector distance of the short text.

[0134] In some examples, structure modules are used to:

[0135] For each short text, determine the type of structure contained in the short text;

[0136] According to the structure types contained, the short text is divided into multiple sub-texts of different structure types, and the arrangement order of each sub-text is determined;

[0137] According to the multiple sub-texts and the arrangement order of each sub-text, the multiple sub-texts are spliced ​​into a target short text based on the structural type expression.

[0138] In some examples, the set of structure types includes at least one of the following: a Chinese structure type, an alphabetic structure type, a numeric structure type, and a symbolic structure type.

[0139] In some examples, the clustering module is used to:

[0140] Calculate the first short text similarity between every two target short texts respectively;

[0141] Calculate the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts and the corresponding commonality weight;

[0142] Clustering is performed according to the second short text similarity between every two target short texts to obtain a plurality of short text clustering clusters, and abnormal short texts are determined based on the plurality of short text clustering clusters.

[0143] In some examples, when the clustering module calculates the second short text similarity between each two target short texts based on the first short text similarity between each two target short texts and the corresponding commonality weight, it is used to:

[0144] The second short text similarity is obtained according to the sum of the first short text similarity and the corresponding commonality weight between every two target short texts.

[0145] In some examples, the apparatus further comprises:

[0146] The rare character module is used to determine the rare characters contained in each short text, and based on the rare characters contained, determine the rare character weight of each short text.

[0147] In some examples, the clustering module is used to:

[0148] Clustering is performed according to each target short text, its corresponding commonality weight and uncommon character weight to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

[0149] In some examples, the clustering module is used to:

[0150] Calculate the first short text similarity between every two target short texts respectively;

[0151] Calculate the second short text similarity between each two target short texts according to the first short text similarity, the corresponding commonality weight and the corresponding uncommon character weight between each two target short texts;

[0152] Clustering is performed according to the second short text similarity between every two target short texts to obtain a plurality of short text clustering clusters, and abnormal short texts are determined based on the plurality of short text clustering clusters.

[0153] In some examples, the clustering module is used to calculate the second short text similarity between each two target short texts according to the first short text similarity, the corresponding commonality weight and the corresponding uncommon character weight between each two target short texts:

[0154] The second short text similarity is obtained according to the sum of the first short text similarity, the corresponding commonality weight and the inverse of the corresponding uncommon character weight between every two target short texts.

[0155] In some examples, when determining an abnormal short text based on a plurality of short text clusters, the clustering module is configured to:

[0156] Compare the densities of multiple short text clusters with preset thresholds respectively;

[0157] The short texts contained in the short text clusters whose density is greater than a preset threshold are determined as abnormal short texts.

[0158] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0159] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0160] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0161] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from the storage unit 602 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0162] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0163] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a method for determining an abnormal short text. For example, in some embodiments, the method for determining an abnormal short text may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method for determining an abnormal short text described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to determine an abnormal short text in any other appropriate manner (e.g., by means of firmware).

[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining abnormal short text, wherein: The method comprises: Determine the word vector for each short text separately; Based on the word vectors of each of the short texts, obtaining the commonality weight of each of the short texts; For each of the short texts, determining a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type; Clustering is performed according to each of the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

2. The method according to claim 1, wherein: The obtaining of the commonality weight of each of the short texts based on the word vectors of each of the short texts includes: For each of the short texts, determining the vector distance between every two adjacent characters; The vector distance of the short text is obtained according to the vector distance between each two adjacent characters.

3. The method according to claim 2, wherein: The step of obtaining the vector distance of the short text according to the vector distance between each two adjacent characters includes: The vector distances of each two adjacent characters are added together to obtain the vector distance of the short text.

4. The method according to any one of claims 1 to 3, wherein: The step of determining, for each of the short texts, a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type includes: For each of the short texts, determining the structure type contained in the short text; According to the included structure types, the short text is divided into a plurality of sub-texts of different structure types, and an arrangement order of each of the sub-texts is determined; According to the multiple subtexts and the arrangement order of the subtexts, the multiple subtexts are concatenated into the target short text expressed based on the structure type.

5. The method according to claim 4, wherein: The set of structure types includes at least one of the following: Chinese structure type, letter structure type, number structure type, and symbol structure type.

6. The method according to any one of claims 1 to 5, wherein: The step of clustering the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and determining abnormal short texts based on the plurality of short text clusters includes: Calculating the first short text similarity between each two target short texts respectively; Calculating a second short text similarity between each two target short texts according to the first short text similarity between each two target short texts and the corresponding commonality weight; Clustering is performed according to the second short text similarity between every two of the target short texts to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

7. The method according to claim 6, wherein: The calculating the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts and the corresponding commonality weight comprises: The second short text similarity is obtained according to the sum of the first short text similarity and the corresponding commonality weight between every two target short texts.

8. The method according to any one of claims 1 to 7, wherein: After determining the word vectors of each short text respectively, clustering is performed according to each of the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and before determining abnormal short texts based on the plurality of short text clusters, the method further includes: Uncommon characters included in each of the short texts are determined, and based on the included uncommon characters, an uncommon character weight of each of the short texts is determined.

9. The method according to claim 8, wherein: The step of clustering the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and determining abnormal short texts based on the plurality of short text clusters includes: Clustering is performed according to each of the target short texts, the corresponding commonality weights and the uncommon character weights to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

10. The method according to claim 9, wherein: The step of clustering the target short texts, the commonality weights and the uncommon character weights thereof to obtain a plurality of short text clusters, and determining abnormal short texts based on the plurality of short text clusters includes: Calculating the first short text similarity between each two target short texts respectively; Calculate the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts, the corresponding commonality weight and the corresponding uncommon character weight; Clustering is performed according to the second short text similarity between every two of the target short texts to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

11. The method according to claim 10, wherein: The calculating the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts, the corresponding commonality weight and the corresponding uncommon character weight comprises: The second short text similarity is obtained according to the sum of the first short text similarity, the corresponding commonality weight and the inverse of the corresponding uncommon character weight between every two target short texts.

12. The method according to any one of claims 1 to 11, wherein: The determining of abnormal short texts based on the multiple short text clusters includes: Comparing the densities of the plurality of short text clusters with a preset threshold respectively; The short texts included in the short text cluster whose density is greater than the preset threshold are determined as abnormal short texts.

13. A device for determining abnormal short text, wherein: The device comprises: A determination module is used to determine the word vector of each short text; A commonality module, used for obtaining a commonality weight of each of the short texts based on the word vectors of each of the short texts; A structure module, used for determining, for each of the short texts, a structure type contained in the short text, and converting the short text into a target short text expressed based on the structure type; The clustering module is used to cluster the target short texts and the corresponding commonality weights to obtain a plurality of short text clusters, and determine abnormal short texts based on the plurality of short text clusters.

14. The device according to claim 13, wherein: The common module is specifically used for: For each of the short texts, determining the vector distance between every two adjacent characters; The vector distance of the short text is obtained according to the vector distance between each two adjacent characters.

15. The device according to claim 14, wherein: When the commonality module obtains the vector distance of the short text according to the vector distance of each two adjacent characters, the commonality module is used to: The vector distances of each two adjacent characters are added together to obtain the vector distance of the short text.

16. The device according to any one of claims 13 to 15, wherein: The structural module is used for: For each of the short texts, determining the structure type contained in the short text; According to the included structure types, the short text is divided into a plurality of sub-texts of different structure types, and an arrangement order of each of the sub-texts is determined; According to the multiple subtexts and the arrangement order of the subtexts, the multiple subtexts are concatenated into the target short text expressed based on the structure type.

17. The device according to claim 16, wherein: The set of structure types includes at least one of the following: Chinese structure type, letter structure type, number structure type, and symbol structure type.

18. The device according to any one of claims 13 to 17, wherein: The clustering module is used to: Calculating the first short text similarity between each two target short texts respectively; Calculating a second short text similarity between each two target short texts according to the first short text similarity between each two target short texts and the corresponding commonality weight; Clustering is performed according to the second short text similarity between every two of the target short texts to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

19. The device according to claim 18, wherein: When the clustering module calculates the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts and the corresponding commonality weight, it is used to: The second short text similarity is obtained according to the sum of the first short text similarity and the corresponding commonality weight between every two target short texts.

20. The device according to any one of claims 13 to 19, wherein: The device also includes: The uncommon characters module is used to determine the uncommon characters contained in each of the short texts, and determine the uncommon characters weight of each of the short texts based on the contained uncommon characters.

21. The device according to claim 20, wherein: The clustering module is used to: Clustering is performed according to each of the target short texts, the corresponding commonality weights and the uncommon character weights to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

22. The device according to claim 21, wherein The clustering module is used to cluster the target short texts, the commonality weights and the uncommon character weights thereof to obtain a plurality of short text clusters, and determine abnormal short texts based on the plurality of short text clusters: Calculating the first short text similarity between each two target short texts respectively; Calculate the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts, the corresponding commonality weight and the corresponding uncommon character weight; Clustering is performed according to the second short text similarity between every two of the target short texts to obtain a plurality of short text clusters, and abnormal short texts are determined based on the plurality of short text clusters.

23. The device according to claim 22, wherein: The clustering module, when calculating the second short text similarity between each two target short texts according to the first short text similarity between each two target short texts, the corresponding commonality weight and the corresponding uncommon character weight, is used to: The second short text similarity is obtained according to the sum of the first short text similarity, the corresponding commonality weight and the inverse of the corresponding uncommon character weight between every two target short texts.

24. The device according to any one of claims 13 to 23, wherein: The clustering module, in the case of determining abnormal short texts based on the multiple short text clusters, is used to: Comparing the densities of the plurality of short text clusters with a preset threshold respectively; The short texts included in the short text cluster whose density is greater than the preset threshold are determined as abnormal short texts.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images