Information identification method and device, computer device, storage medium and program product

Abstract

The application provides an information identification method and device, computer equipment, a storage medium and a program product, and relates to the technical fields of artificial intelligence, big data, intelligent transportation and the like. A text feature vector of a to-be-identified text is obtained. Whether the to-be-identified text is spam is determined based on the similarity between the sample feature vectors of each sample in a sample library and the text feature vector. For each spam category, a positive context feature library corresponding to the spam category is obtained based on a positive sample set of the spam category. A negative context feature library corresponding to the spam category is obtained based on a negative sample set of the spam category. Feature words of each spam category are obtained. The positive and negative context feature libraries, topic words, category keywords or entity name words of each spam category are used to obtain the sample feature vectors of each sample in the sample library, so that the text can be identified based on multi-dimensional features, and the accuracy of identification is improved.

Description

Technical Field

[0001] This application relates to the technical fields of artificial intelligence, big data, and intelligent transportation. This application relates to an information identification method, device, computer equipment, storage medium, and program product. Background Technology

[0002] Big data is a product of the rapid development of internet technology. Text-based big data is widely present in people's daily lives, such as through text messages and emails. Inevitably, spam, such as spam advertisements and fraudulent messages, exists within this data. Therefore, the ability to identify spam is crucial in this field.

[0003] In related technologies, the information recognition process includes, for example, using rules for information recognition, employing exact matching or fuzzy matching to determine whether information in a fixed text format contains manually specified category keywords, thus classifying the information as spam. However, this information recognition relies on a fixed text format, making it prone to matching errors and resulting in low accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, computer equipment, storage medium, and program product for information identification, which can solve the problem of low accuracy in related technologies. The technical solution is as follows:

[0005] On the one hand, an information identification method is provided, the method comprising:

[0006] Obtain the text feature vector of the text to be identified;

[0007] Based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector, the spam indication information of the text to be identified is determined. The spam indication information is used to indicate whether the text to be identified belongs to spam information.

[0008] The method for obtaining the sample feature vectors of each sample in the sample set includes:

[0009] Construct a sample library, which includes positive and negative sample sets belonging to each waste category;

[0010] For each waste category, based on the positive sample set of the waste category, a positive context feature library corresponding to the waste category is obtained, and based on the negative sample set of the waste category, a negative context feature library corresponding to the waste category is obtained;

[0011] Perform waste category statistics on the positive sample set to obtain the feature words of each waste category. The feature words include at least one of the following: topic words, category keywords, or entity names of each waste category.

[0012] Based on at least one of the positive context feature library, negative context feature library, topic words, category keywords or entity name words for each waste category, obtain the sample feature vector of each sample in the sample library;

[0013] The subject terms are used to characterize the theme of all or part of the content in the sample, and the category keywords are used to distinguish the waste category to which the sample belongs.

[0014] In one possible implementation, for each waste category, obtaining the positive contextual feature library corresponding to the waste category based on the positive sample set of the waste category includes:

[0015] Each positive sample is preprocessed to obtain preprocessed data for each positive sample;

[0016] The preprocessed data of each positive sample is segmented into words to obtain each first filter word;

[0017] Based on each first filter word and a preset minimum support threshold, positive context features are mined from each positive sample to obtain a positive context feature library.

[0018] In one possible implementation, obtaining the text feature vector of the text to be identified includes:

[0019] The feature words included in the text to be identified are determined based on the feature words of each waste category;

[0020] Based on the fused feature vector of each character in the feature words included in the text to be identified, the text feature vector of the text to be identified is obtained;

[0021] The method for obtaining the fused feature vector of each character includes:

[0022] Obtain the word vector of each feature word in the text to be identified, and the character vector of each character in each feature word;

[0023] For each character of each feature word, the word vector of each feature word and the character vector of the character are fused to obtain the fused feature vector of the character.

[0024] In one possible implementation, the feature words include topic words, and the methods for obtaining the topic words include:

[0025] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0026] Based on the first word frequency of each word in each positive sample, candidate topic words for each waste category are determined from each of the word segments;

[0027] For each candidate keyword, the importance of the candidate keyword is determined based on its second word frequency in the positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category.

[0028] For each waste category, the keywords for that waste category are determined from the candidate keywords based on the importance of each candidate keyword for that waste category.

[0029] In one possible implementation, the feature words include category keywords, and the category keywords are obtained in the following ways:

[0030] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0031] For each word segment, a first ratio is determined between the number of times the word appears in the waste category and the total number of words in the waste category;

[0032] Based on the number of samples in the positive sample set, the number of samples in the positive samples of other waste categories besides the waste category mentioned above, and the first ratio, the class discrimination ability of the word segmentation for the waste category is determined.

[0033] Based on the ability of each word segment to distinguish between different types of waste, category keywords for each type of waste are determined from the word segments.

[0034] On the other hand, an information identification device is provided, the device comprising:

[0035] The first acquisition module is used to acquire the text feature vector of the text to be recognized;

[0036] The determination module is used to determine the spam indication information of the text to be identified based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector. The spam indication information is used to indicate whether the text to be identified belongs to spam information.

[0037] The device, when acquiring the sample feature vectors of each sample in the sample set, further includes:

[0038] A construction module is used to build a sample library, which includes positive and negative sample sets belonging to each waste category;

[0039] The second acquisition module is used to acquire, for each waste category, a positive context feature library corresponding to the waste category based on the positive sample set of the waste category, and a negative context feature library corresponding to the waste category based on the negative sample set of the waste category;

[0040] The third acquisition module is used to perform garbage category statistics on the positive sample set and acquire the feature words of each garbage category. The feature words include at least one of the subject words, category keywords or entity names of each garbage category.

[0041] The fourth acquisition module is used to acquire the sample feature vector of each sample in the sample library based on at least one of the positive context feature library, negative context feature library, topic words, category keywords or entity name words of each waste category;

[0042] The subject terms are used to characterize the theme of all or part of the content in the sample, and the category keywords are used to distinguish the waste category to which the sample belongs.

[0043] In one possible implementation, the second acquisition module is used for:

[0044] Each positive sample is preprocessed to obtain preprocessed data for each positive sample;

[0045] The preprocessed data of each positive sample is segmented into words to obtain each first filter word;

[0046] Based on each first filter word and a preset minimum support threshold, positive context features are mined from each positive sample to obtain a positive context feature library.

[0047] In one possible implementation, the first acquisition module is configured to:

[0048] The feature words included in the text to be identified are determined based on the feature words of each waste category;

[0049] Based on the fused feature vector of each character in the feature words included in the text to be identified, the text feature vector of the text to be identified is obtained;

[0050] Specifically, when the first acquisition module acquires the fusion feature vector of each character, it is used for:

[0051] Obtain the word vector of each feature word in the text to be identified, and the character vector of each character in each feature word;

[0052] For each character of each feature word, the word vector of each feature word and the character vector of the character are fused to obtain the fused feature vector of the character.

[0053] In one possible implementation, the feature words include topic words, and the third acquisition module is used for:

[0054] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0055] Based on the first word frequency of each word in each positive sample, candidate topic words for each waste category are determined from each of the word segments;

[0056] For each candidate keyword, the importance of the candidate keyword is determined based on its second word frequency in the positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category.

[0057] For each waste category, the keywords for that waste category are determined from the candidate keywords based on the importance of each candidate keyword for that waste category.

[0058] In one possible implementation, the feature words include category keywords, and the third acquisition module is used for:

[0059] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0060] For each word segment, a first ratio is determined between the number of times the word appears in the waste category and the total number of words in the waste category;

[0061] Based on the number of samples in the positive sample set, the number of samples in the positive samples of other garbage categories besides the garbage category mentioned above, and the first ratio, the class discrimination ability of the word segmentation for the garbage category is determined.

[0062] Based on the ability of each word segment to distinguish between different types of waste, category keywords for each type of waste are determined from the word segments.

[0063] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described information recognition method.

[0064] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described information recognition method.

[0065] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described information recognition method.

[0066] The beneficial effects of the technical solutions provided in this application are:

[0067] The information recognition method provided in this application obtains the text feature vector of the text to be recognized; and determines whether the text to be recognized belongs to spam based on the similarity between the sample feature vector of each sample in the sample library and the text feature vector. Specifically, when obtaining the sample feature vector of a sample, a sample library is constructed. For each spam category, a positive context feature library corresponding to the spam category is obtained based on the positive sample set of that spam category, and a negative context feature library corresponding to the spam category is obtained based on the negative sample set of that spam category. Furthermore, spam category statistics are performed on the positive sample set to obtain feature words for each spam category. These feature words include at least one of the following: topic words, category keywords, or entity names. By obtaining the positive and negative context features of the sample, as well as multi-dimensional features such as topic words, category keywords, or entity names, the sample feature vector of each sample in the sample library can be obtained based on the positive and negative context feature libraries, topic words, and feature words of each spam category. This allows for the recognition of the text to be recognized based on multi-dimensional features, improving the accuracy of the recognition. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0069] Figure 1 A flowchart illustrating an information identification method provided in an embodiment of this application;

[0070] Figure 2 A flowchart illustrating a method for obtaining sample feature vectors of each sample provided in an embodiment of this application;

[0071] Figure 3 A flowchart illustrating a feature fusion method provided in an embodiment of this application;

[0072] Figure 4 This is a schematic diagram of the structure of a multi-classification model provided in an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of the structure of an information identification device provided in an embodiment of this application;

[0074] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0075] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0076] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. The terms “comprising” and “including” as used in the embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, or operation, but do not exclude implementation as other features, information, data, steps, or operations supported by this art.

[0077] It is understood that in the specific implementation of this application, any data related to the object, such as garbage categories, garbage category feature words, emails, messages, and text messages used as sample data, is involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the object is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0078] Figure 1 This is a flowchart illustrating an information identification method provided in an embodiment of this application. The method can be executed by a computer device. Figure 1 As shown, the method includes the following steps.

[0079] Step 101: The computer device acquires the text feature vector of the text to be recognized.

[0080] The text to be identified can be any form of text information, including but not limited to emails, text messages, and bullet screen messages. In some possible scenarios, the information recognition method of this application can be widely applied to spam text classification, email filtering, spam SMS identification, and bullet screen spam filtering. For example, in bullet screen applications on some video playback platforms, identifying spam information such as uncivilized language in bullet screen content can improve the user's viewing experience and reduce losses caused by risk control measures. For the identification and filtering of spam SMS and spam emails, it can reduce information harassment and prevent being scammed. In addition, this application can also be applied to potential application scenarios related to spam text recognition and classification.

[0081] The text to be identified can be online text or text from an offline text library. For example, if the text to be identified is online text, the computer device can be a server. For instance, the server receives the identification request sent by the terminal and performs information identification on the text to be identified sent by the terminal. Alternatively, if the text to be identified is offline text, the computer device can be a terminal, which can be in any state, such as offline or online, and perform information identification on the locally stored text to be identified.

[0082] The text feature vector can be obtained by encoding the text to be recognized using vector encoding. For example, the embedding vector of the text to be recognized can be extracted using a trained word2vec network as the text feature vector.

[0083] It should be noted that the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, digital radio receiver, desktop computer, in-vehicle terminal (e.g., in-vehicle navigation terminal, in-vehicle computer), smart speaker, smartwatch, etc. The terminal and server can be connected directly or indirectly via wired or wireless communication, and the connection can be determined based on the actual application scenario requirements; no restrictions are imposed here.

[0084] Step 102: The computer device determines the spam indication information of the text to be identified based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector.

[0085] The spam indication information is used to indicate whether the text to be identified belongs to spam.

[0086] The sample feature vectors of each sample can be obtained by fusing multi-dimensional features such as topic word features, category keyword features, and context features of each type of waste. This application can utilize the multi-dimensional features of the samples to identify the text to be identified. In one possible implementation, the computer device determines the sample with the highest similarity to the text to be identified based on the similarity between the sample feature vector and the text feature vector, and uses the waste indication information of the determined sample as the waste indication information of the text to be identified. That is, if the sample most similar to the text to be identified is waste information, then the text to be identified is also waste information; otherwise, if the sample most similar to the text to be identified is not waste information, then the text to be identified is not waste information.

[0087] In one possible implementation, the computer device can pre-build a sample library and obtain the sample feature vector of each sample based on the sample library. Figure 2 This is a flowchart illustrating a method for obtaining sample feature vectors of various samples, provided in an embodiment of this application. The method can be executed by a computer device. Figure 2 As shown, the method includes the following steps 201-204.

[0088] Step 201: Construct a sample library, which includes positive and negative sample sets belonging to each waste category.

[0089] The computer device can acquire benchmark words, for example, benchmark words can be constructed for garbage classification scenarios; the computer device can acquire similar words of benchmark words, and use benchmark words and similar words as category keywords of garbage categories; for example, using thesaurus and word2vec word vectors, similar words similar to benchmark words are acquired using similarity calculation methods, so as to expand benchmark words into category keywords of garbage categories based on synonyms; the computer device can acquire sample sets of each garbage category based on the category keywords of each garbage category.

[0090] For example, a computer device can recall a sample library containing various waste categories using category keywords to obtain samples that match each waste category. It can then further determine whether each sample belongs to that waste category, classifying them as positive and negative samples for each category. These positive and negative samples can also be training data used to train a recognition model for information identification. This process completes the construction of waste category keywords and the acquisition of positive and negative sample libraries for each waste category, thus fulfilling the data preparation requirement.

[0091] For example, spam categories may include advertising and promotion, Category A, etc. These categories often have more obvious and frequently used category keywords compared to normal text. These category keywords can be used as the benchmark words for spam text. Taking spam text of Category A as an example, the positive sample and category keywords are shown in Table 1:

[0092] Table 1

[0093] Positive samples Category keywords A service performance A Offering massage and other A services A Health massage provides service B B

[0094] As shown in Table 1, the positive sample set includes three training positive samples: Service A performance; providing massage and other services A; and providing health massage and services B. The category keywords for "Service A performance" include A; the category keywords for "providing massage and other services A" include A; and the category keywords for "providing health massage and services B" include B.

[0095] Step 202: For each waste category, obtain the positive context feature library corresponding to the waste category based on the positive sample set of the waste category, and obtain the negative context feature library corresponding to the waste category based on the negative sample set of the waste category.

[0096] In this application, the prefixspan algorithm can be used to mine positive and negative contextual features for each sample category. It should be noted that the main process of the prefixspan algorithm is as follows: Iterate through the spam text, segmenting it into sentences. Based on the principle of forming sequence patterns from word sequences within the same sentence and the pre-defined prefixspan algorithm, it mines frequent word sequence patterns in sentences containing category keywords. For each spam category, each positive sample of that category is extracted. Based on the prefixspan algorithm, frequent word sequence patterns of various lengths for that positive sample are mined. These frequent word sequence patterns of various lengths for the positive sample constitute the positive contextual features of the category keywords. Similarly, for each spam category, negative samples of that category are extracted. Based on the prefixspan algorithm, frequent word sequence patterns of various lengths for the negative samples are mined. These frequent word sequence patterns of various lengths for the negative samples constitute the negative contextual features of the category keywords.

[0097] Based on the Prefixspan algorithm, frequent word sequence patterns of various lengths in positive or negative samples with pattern support greater than a preset minimum support threshold are mined. The pattern support β is calculated as follows: β = M / N; where β is the pattern support; M is the sum of the number of times the word segment appears in each sample; taking a positive sample as an example, if the word segment is the first filter word, then M is the number of texts in the positive sample set to which the first filter word belongs, and N is the total number of samples in the positive sample set.

[0098] Methods for obtaining the positive context feature library include:

[0099] Step 2021: Preprocess each positive sample to obtain the preprocessed data of each positive sample.

[0100] For example, preprocessing includes: regular expression filtering of punctuation marks, letters, and numbers in the training positive samples; and filtering of category keywords corresponding to the training positive samples to obtain preprocessed data, which is shown in Table 2:

[0101] Table 2

[0102] Preprocessed data Service performance Offering massage and other services Health massage packages provide services

[0103] Step 2022: Segment the preprocessed data of each positive sample to obtain each first filter word.

[0104] Optionally, the preprocessed data shown in Table 2 can be segmented into words, and each word obtained from the segmentation can be used as the first filter word. For example, each word such as service, performance, provide, massage, service, and health care can be used as the first filter word.

[0105] Step 2023: Based on each first filter word and the preset minimum support threshold, mine the positive context features in each positive sample to obtain the positive context feature library.

[0106] The computer device can perform positive context feature mining on each positive sample based on a first filter word and a minimum support threshold, obtain a positive context feature sequence, and add the positive context feature sequence to the positive context feature library of the corresponding garbage category. For each garbage category, there are corresponding positive context feature libraries and negative context feature libraries.

[0107] Optionally, for each first filter word, the number of samples in which the first filter word appears in the first positive sample text set is counted. If the pattern support corresponding to the first filter word (calculated according to Formula 1 above) is not greater than the preset minimum support threshold, then the first filter word is filtered; if the pattern support corresponding to the first filter word is greater than the preset minimum support threshold, then the first filter word is used as the second filter word.

[0108] For example, when the first threshold is 1 and the number of text samples in the first positive sample set is 3, the preset minimum support threshold is 1 / 3. Based on this, if the first filter word corresponding to the three training positive samples in Table 1 appears in at least two training positive samples, then the first filter word is used as the second filter word. Specifically, "provide" is the second filter word, appearing in two training positive samples, with a frequency of 2 in these three training positive samples. These two training positive samples are: providing massage and other services A; and providing health massage and other services B. "Massage" is the second filter word, appearing in two training positive samples, with a frequency of 2 in these three training positive samples. These two training positive samples are: providing massage and other services A; and providing health massage and other services B. "Service" is the second filter word, appearing in these three training positive samples, with a frequency of 3 in these three training positive samples. The second filter word and its frequency in these three training positive samples are shown in Table 3.

[0109] Table 3

[0110] Second filter word supply massage Serve word frequency 2 2 3

[0111] As can be seen, through the above filtering operation, we can obtain each high-frequency word in the first positive sample text set, that is, each second filter word.

[0112] Each training positive sample is subjected to word frequency filtering based on each second filter word. That is, after filtering out words other than each second filter word, the word frequency filtered samples are obtained. The word frequency filtered samples are shown in Table 4.

[0113] Table 4

[0114] Samples after word frequency filtering Serve Massage services are available Massage services

[0115] The PrefixSpan algorithm is used to mine the frequency-filtered samples to obtain frequent word sequence patterns of various lengths in each training positive sample. The second filtering word is a term prefix. First, prefixes of length 1 (i.e., term prefixes) are mined. At this time, each term prefix that meets the minimum support threshold and its corresponding adjacent suffix (i.e., the words included in the adjacent subsequent part of the prefix in the frequency-filtered samples) can be determined. For the term prefix "service", its adjacent suffixes that do not exist in the frequency-filtered samples "service", "provide massage service", and "massage provides service" are all represented by "none" in the table; for the term prefix "provide", its adjacent suffix in the frequency-filtered sample "provide massage service" is "massage service", and its adjacent suffix in the frequency-filtered sample "massage provides service" is "service"; for the term prefix "massage", its adjacent suffix in the frequency-filtered sample "provide massage service" is "service", and its adjacent suffix in the frequency-filtered sample "massage provides service" is "provide service". The mining yielded a prefix and its corresponding suffix, as shown in Table 5:

[0116] Table 5

[0117]

[0118] In the next iteration, using the same method described above for mining single-term prefixes and their corresponding suffixes, binomial prefixes and their corresponding suffixes are obtained, as shown in Table 6.

[0119] Table 6

[0120] Binomial prefix Corresponding suffix Provide services none Massage service none

[0121] Since neither the binary prefixes "provide services" nor "massage services" have adjacent suffixes, the iteration ends, yielding frequent word sequence patterns of various lengths in the training positive samples and their corresponding pattern support. The frequent word sequence patterns of various lengths in the training positive samples are positive contextual features, as shown in Table 7.

[0122] Table 7

[0123]

[0124] In this embodiment of the application, by mining the positive contextual features of category keywords, the number and types of samples required in the recognition process of the text to be recognized are expanded, thereby improving the efficiency and accuracy of text category recognition.

[0125] It should be noted that the method for obtaining the negative context feature library can similarly include the following steps (2024-2026):

[0126] Step 2024: Preprocess each negative sample to obtain the preprocessed data of each negative sample.

[0127] Step 2025: Segment the preprocessed data of each negative sample to obtain each third filtered word.

[0128] Step 2026: Based on each third filtered word and a preset minimum support threshold, mine the negative context features in each negative sample to obtain a negative context feature library.

[0129] It should be noted that the implementation manners of steps 2024 - 2026 are the same as those of steps 2021 - 2023, and will not be elaborated here one by one.

[0130] Step 203: Conduct garbage category statistics on the positive sample set to obtain the feature words of each garbage category.

[0131] The feature words include at least one of the topic words, category keywords, or entity name words of each garbage category. Among them, the topic word is used to represent the topic of all or part of the content in the sample, and the category keyword is used to distinguish the garbage category to which the sample belongs.

[0132] The topic word is used to represent the content topic in the sample. For example, a word that has the meaning of representing the central idea, main content, part of the content, part of the view, topic, etc. of the sample text.

[0133] The category keyword can be a word that has the ability to distinguish garbage categories. For example, if a word often appears in a certain garbage category and rarely appears in other garbage categories, it means that the word has the ability to distinguish the garbage category, and the word can be used as the category keyword of the garbage category.

[0134] The entity name word refers to an entity identified by a name, which is a proper noun; for example, it can be a proper noun such as a place name or a commodity name.

[0135] In a possible implementation manner, the present application can obtain the topic words of each garbage category based on the importance of word segmentation for the garbage category. Exemplarily, the feature words include topic words, and the obtaining manner of the topic words includes the following steps 2031 - 2034:

[0136] Step 2031: Segment each positive sample to obtain each word segment included in each positive sample.

[0137] When performing word segmentation, some virtual words without real meaning can be filtered out, such as "de", "le", etc.

[0138] Step 2032: Based on the first word frequency of each word segment in each positive sample, determine the candidate topic words of each garbage category from each word segment.

[0139] Word frequency refers to the number of times a word appears in a sample or sample set, and can be used to assess the degree of repetition of the word in a sample, sample set, or corpus.

[0140] The first term frequency refers to the number of times the word appears in the positive samples of each spam category. The second term frequency refers to the number of times the word appears in the resume data of all positive sample resumes across all spam categories. For example, the first term frequency can be represented as follows:

[0141] The number of times this word appears in all samples of this garbage category / the total number of all words appearing in all samples of this category.

[0142] Step 2033: For each candidate keyword, determine the importance of the candidate keyword based on its second word frequency in the positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category.

[0143] The second term frequency refers to the number of times a candidate keyword appears in all positive samples across all spam categories. Document frequency refers to the number of documents in which a particular keyword appears in a given sample set; a document can be a single sample text. Document frequency is based on the ratio of the number of positive samples containing the candidate keyword to the total number of positive samples.

[0144] Step 2034: For each waste category, determine the keywords for that waste category from the candidate keywords based on the importance of each candidate keyword for that waste category.

[0145] For each waste category, the keywords for that category can include general terms, secondary terms, or irrelevant terms. The following section describes the methods for determining the types of keywords and their importance.

[0146] The garbage text database, containing samples belonging to different garbage categories, was used to perform garbage category feature statistics and new word discovery to obtain a domain dictionary for each garbage category. Obtaining the domain dictionary mainly involves comparing the frequency differences between categories. For example, if a word has a high frequency in one garbage category but a low frequency in others, it is defined as a domain word for that category. Conversely, if a word shows little variation in frequency across different garbage categories, it is defined as an irrelevant word. Words that are neither domain words nor irrelevant words are defined as ordinary words.

[0147] For common words, their importance can be expressed as:

[0148]

[0149] Among them, W c Indicates a common word; tfc This represents the second word frequency, which is the frequency of this common word in all spam texts within the standard library (i.e., the positive sample set including each spam category). maxdf k This represents the document frequency of each common word that appears in all garbage categories in the standard library, specifically the highest document frequency. c W represents the common word c Document frequency in all garbage categories of the standard library.

[0150] For domain terms, the importance of domain terms can be expressed as:

[0151]

[0152] Among them: W f Domain term; maxPW c This represents the highest importance weight among all common words; tf f The second word frequency, which is the domain word W. f The frequency of occurrence of text in all garbage categories in the standard library; maxdf l The document frequency of each domain term appearing in all garbage category texts in the standard library, representing the highest document frequency. f For the domain term W f The document frequency of occurrence in all garbage category texts in the standard library.

[0153] For entity names, such as proper nouns, the importance can be expressed as:

[0154] W topic (w)=PW×δ prop (w)×δ noun (w);

[0155] Wherein, PW is PW c Or PW f Weighted δ for proper nouns prop (w), δ prop (w) is a positive number greater than 1. If the domain term or common term w is a proper noun, the importance weight will be multiplied by this coefficient δ. prop (w); Weighted noun part-of-speech δ noun (w), δ noun (w) is a positive number greater than 1, δ noun (w)<δ prop (w), if the part of speech of word w is a noun, the weight will be multiplied by a coefficient δ. noun (w).

[0156] For irrelevant words, the importance of irrelevant words in this application can be 0.

[0157] In one possible implementation, the present application can obtain category keywords for each category based on the discrimination ability of words in each classification category. Exemplarily, the feature word includes category keywords, and the method for obtaining the category keywords includes the following steps 2035-2038:

[0158] Step 2035: Segment each positive sample to obtain each segmented word included in each positive sample.

[0159] When segmenting words, some virtual words without real meaning can be filtered out, such as "de", "le", etc.

[0160] Step 2036: For each segmented word, determine the first ratio between the number of times the segmented word appears in the garbage category and the total number of words in the garbage category.

[0161] For example, the first ratio can also be called the category keyword frequency; the method for obtaining the category keyword frequency includes:

[0162]

[0163] Step 2037: Based on the number of samples in the positive sample set, the number of samples in which the segmented word appears in the positive samples of other garbage categories except this garbage category, and the first ratio, determine the category discrimination ability of the segmented word for this garbage category.

[0164] Step 2038: According to the category discrimination ability of each segmented word for each garbage category, determine the category keywords for each garbage category from these segmented words.

[0165] TF-IDF represents the discrimination ability of each segmented word in each garbage category. The calculation method of TF-IDF provided in the present application is as follows:

[0166]

[0167] TF-IDF = Term Frequency (TF) × Inverse Document Frequency (IDF)

[0168] The TF-IDF of a word corresponding to a garbage category is the category discrimination ability of the word for this garbage category. If a segmented word often appears in the resume of a certain garbage category and rarely appears in the resumes of other garbage categories, then it means that this word is the category discrimination keyword for this category. For each garbage category, according to the TF-IDF of each segmented word corresponding to this garbage category, in the order from largest to smallest TF-IDF, the segmented words in the top preset order of TF-IDF are used as the category keyword labels for this garbage category, that is, the category keywords for this garbage category.

[0169] Step 204: Obtain the sample feature vectors of each sample in the sample library based on at least one of the positive context feature library, negative context feature library, topic words, category keywords, or entity name words of each garbage category;

[0170] For the features obtained in the above steps, including vectorizing and encoding the positive and negative context feature libraries, feature words, etc., obtain the mixed feature vectors of each character included in each sample.

[0171] In this step, the computer device can combine the multi-dimensional features of each sample for encoding to obtain the sample feature vector. Exemplarily, for each sample, the computer device encodes each sample based on the positive context feature and its pattern support degree, negative context feature and its pattern support degree, and information such as topic words, feature words, and their importance included in each sample, and generates the sample feature vector of each sample.

[0172] In a possible example, for each sample, the method for obtaining the fusion feature vector of each character in the sample includes steps A1 - A2:

[0173] Step A1: Obtain the word vector of each feature word in the sample and the character vector of each character in each feature word;

[0174] Step A2: For each character of each feature word, fuse the word vector of the feature word and the character vector of the character to obtain the fusion feature vector of the character.

[0175] Figure 3 This is a schematic diagram of a method for fusing the fusion feature vectors of characters provided in an embodiment of the present application. As Figure 3 shown, in this example, the word is "resume". After splitting this word into characters, the two characters "jian" and "li" included in the word can be obtained. The word vector of "resume" (the word vector shown in the figure) can be obtained through the word vector model, and the character vectors of "jian" and "li" (the character vectors shown in the figure) can be obtained through the character vector model. By concatenating (the ⊕ shown in the figure) the word vector of "resume" and the character vector of "jian", the fusion feature vector of "jian" can be obtained, and by concatenating the word vector of "resume" and the character vector of "li", the fusion feature vector of "li" can be obtained. During implementation, in order to obtain the word vector aligned with the character vector, each word can be encoded repeatedly, and the number of repetitions is the number of characters constituting the word. For example, in the above example, the word vector of "resume" needs to be used twice, and is respectively fused with the character vectors of "jian" and "li".

[0176] In a possible implementation manner, for step 101, the implementation manner of obtaining the text feature vector of the text to be recognized may include the following steps 301 - 302:

[0177] Step 301: Determine the feature words included in the text to be identified based on the feature words of each type of waste.

[0178] For example, the feature words included in the text to be identified may include at least one of the following: subject words, domain words, common words, or entity name words from the feature words of each waste category.

[0179] Step 302: Based on the fused feature vector of each character in the feature words included in the text to be identified, obtain the text feature vector of the text to be identified.

[0180] For example, a computer device can extract the feature vector of the text to be recognized based on the feature words included in the text, using a neural network model. For instance, the embedding vector of the text to be recognized can be extracted using a word2vec network. When extracting features from each character in the text to be recognized, feature fusion can also be used to obtain a fused feature vector for each character. For example, the method for obtaining the fused feature vector of each character in the text to be recognized includes: obtaining the word vector of each feature word in the text to be recognized, and the character vector of each character in each feature word; for each character of each feature word, fusing the word vector of each feature word and the character vector of the character to obtain the fused feature vector of that character. This process is similar to steps A1-A2, and will not be described in detail here.

[0181] In one possible implementation, when determining the garbage indication information of the text to be identified based on the similarity between the sample feature vectors of each sample and the text feature vector, the text to be identified can be identified according to the positive sample set and negative sample set under each garbage category in the sample library. Each positive sample in the positive sample set includes at least one of at least a positive context feature, a topic word, or a feature word. Each negative sample in the negative sample set includes at least one of at least a negative context feature, a topic word, or a feature word. For example, garbage texts including at least one of positive context features, negative context features, topic words, or feature words can be matched in a pre-built corpus based on positive context features, negative context features, and topic words or feature words; a garbage classification scoring library is determined based on the matched garbage texts. The garbage classification scoring library includes a positive sample set and a negative sample set. The garbage classification scoring library is shown in Table 8.

[0182] Table 8

[0183]

[0184]

[0185] Texts with the same waste classification code are grouped together to construct a waste classification scoring database labeled with that code. For example, feature vectorization encoding is performed on the waste texts in the database. Each piece of waste text is segmented into words, yielding at least one word; word2vec is used for word vectorization to obtain the word vector corresponding to that word; and the sentence vector of the waste text is determined based on the word vectors.

[0186] For example, the computer device can calculate the text feature vector of the text to be identified and its similarity to the feature vectors of each sample in the positive and negative sample sets, respectively; the computer device uses the garbage indication information of the sample with the highest similarity as the garbage indication information of the text to be identified. Furthermore, the computer device can also use the garbage category of the sample with the highest similarity as the garbage category of the text to be identified.

[0187] In this system, the text feature vector of the text to be identified is the sentence vector of the text to be identified, and the feature vector of each sample is the sentence vector of that sample. The sentence vector of the text to be identified is input into the feature extraction network to determine the encoding features of the sentence vector of the text to be identified; based on the encoding features corresponding to the sentence vector of the text to be identified and the encoding features of multiple samples, the similarity between the encoding features of multiple samples and the encoding features of the sentence vector of the text to be identified is determined; based on the similarity between the encoding features of multiple samples and the encoding features of the sentence vector of the text to be identified, the spam category of the text to be identified is determined.

[0188] For example, the above classification process can be achieved using a multi-class classification model based on am-softmax. The feature extraction networks in this multi-class classification model include, but are not limited to, BI-LSTM, CNN, and GRU. Figure 4 As shown, the multi-class classification model based on AM-softmax includes BI-LSTM, Concat layer, FC, Attention layer and AM-Softmax.

[0189] Optionally, the sentence vector of the text to be recognized is used as the input x of the BI-LSTM included in the am-softmax-based multi-classification model, where x is the feature encoding of the sentence, including the feature encodings of words or characters in the sentence. When the text to be recognized includes n words or n characters, X0, X1, X2...Xn are the feature encodings of the n words or n characters contained in the text to be recognized, where n is a positive integer greater than or equal to 1. For example, as... Figure 4As shown, "feature embedding" represents the feature encoding of words or characters in a sentence, where n takes the value of 3, namely X0, X1, X2, and X3. X0, X1, X2, and X3 are used as inputs to the BI-LSTM. The output data of the BI-LSTM is concatenated using `concat`, and the concatenated data is output from `concat`. The concatenated data serves as the intermediate matrix parameters of the neural network. The concatenated data is then input into a fully connected layer (FC) and an Attention layer to obtain the text features y of the text to be recognized. The output y of the Attention layer is obtained according to the following formula: y represents the text features of the text to be recognized.

[0190] Formula 1: y = Bi - LSTM(x);

[0191] For example, multiple pre-defined training sentence vectors x1, x2, ... x n The input is fed into the BI-LSTM, and the Attention layer outputs y1, y2, ... y1 respectively. n Calculate y with y1, y2, ... y respectively. n The similarity between the samples is used to obtain n similarity scores. If the similarity between y and y1 is the largest, then the category of the sample corresponding to x1 is taken as the category of the text to be identified.

[0192] In one possible implementation, the training method of this multi-classification model includes the following steps 401-404:

[0193] Step 401: Determine the training sentence vectors based on the preset garbage classification scoring database;

[0194] Step 402: Input the training sentence vectors into the feature extraction network of the multi-classification model to determine the encoded features of the training sentence vectors;

[0195] Step 403: Determine the value of the loss function of the multi-classification model based on the encoding features of the training sentence vectors and the preset garbage classification scoring database;

[0196] Step 404: Based on the value of the loss function, iteratively train the multi-classification model and optimize the network parameters of the multi-classification model during the iterative training process until the termination condition is met, and then stop training to obtain the trained classification model.

[0197] like Figure 4As shown, the training sentence vector x is input into the Bi-LSTM feature extraction model, which is part of the am-softmax-based multi-classification model. The output data of the Bi-LSTM is concatenated using concat, and the concatenated data is output from concat1. The concatenated data serves as the intermediate matrix parameters of the neural network. The concatenated data is then input into a fully connected layer (FC) and an attention layer to obtain the text features y corresponding to the training sentence vector. y is then input into am-softmax, and am-softmax outputs a set W of garbage classification categories included in the garbage classification scoring database, i.e., multiple garbage classification categories c1, c2, ..., c3. n and c1, c2, ..., c n The corresponding probabilities p1, p2, ..., p n Select p1, p2, ..., p n The maximum probability p in i Corresponding category c i The input x is assigned a classification category, where i is a positive integer not greater than n. The text feature y corresponding to the training sentence vector is determined according to Formula 1, where the text feature y serves as the output of the Attention layer. Based on the text feature y corresponding to the training sentence vector x and a pre-defined garbage classification scoring database, the loss function of the multi-classification model is determined using Formulas 2 and 3. When the loss function value of the multi-classification model is not less than a pre-defined loss threshold, the multi-classification model is iteratively trained, and the network parameters are iteratively updated until the loss function value is less than the loss threshold. Formula 2 is shown below:

[0198] Formula 2: p = am - softmax(yW);

[0199] Where y is the encoded text feature of the BI-LSTM output, and W is the set of garbage classification categories included in the garbage classification scoring database, i.e., W = (c1, c2, ..., c n ), that is, p = am - softmax(<y,c1> ,<y,c2> ,…, <y,c n >).

[0200] The loss function of am-softmax is shown in Equation (6).

[0201] Formula 3:

[0202] Where, θ i Representing y and c i The angle between the two points, where s is the scaling factor (30°) and m is a positive number (0.35°). t is the target index, c1, c2, ..., c n There are multiple waste sorting categories.

[0203] The BI-LSTM and am-softmax models are trained by minimizing the loss of the am-softmax model, using a classification model as the training scheme. When applied, the features extracted by the feature extraction model (here, the BI-LSTM model) are compared and ranked to calculate the similarity between the garbage text in the garbage classification scoring database and the text to be classified. Texts with a similarity to positive samples of the garbage classification category exceeding a preset positive similarity threshold and a similarity to negative samples of the same garbage classification category below a preset negative similarity threshold are considered positive examples of that garbage classification category. Conversely, texts with a similarity to negative samples of the garbage classification category exceeding a preset negative similarity threshold and a similarity to positive samples of the same garbage classification category below a preset positive similarity threshold are considered negative examples of that garbage classification category.

[0204] For example, for the text to be classified, the model input x is [0,1,1,1,0.5,-0.33,0.79,…,0.8,0,1,0…1,0,0,…]*0.85, and the output of the trained model is W, which is the garbage classification set, i.e., W=(c1,c2,…,c n The type label is (0, 1, 2, ..., N), and y is the output of the BI-LSTM encoding model, that is, the encoded features are [0.579, 1.732, 0.9232, -0.2577...]. Whether for training or prediction, the model generates the corresponding encoded features, i.e., y.

[0205] For example, the data example illustrates the prediction: The features encoded by the BI-LSTM feature extraction model are compared and ranked. For instance, given the garbage category text A, predict the classification of the unknown label text B. Text A has features [0.579, 1.732, 0.9232, -0.2577…], and text B has features [0.472, 0.35, -0.58, 0.9224…]. Calculate the cosine distance similarity sim(A,B) between the feature vector of the known garbage category text A and the feature vector of text B. If sim(A,B) exceeds the threshold Sim… rel If the garbage classification of A is not exceeded, then the garbage classification of A will be used as the classification label of text B. rel If the text is not spam, it is considered non-spam text. Similarly, the feature vector of text B is calculated to determine candidates based on its similarity to features from other spam categories. The candidate label with the highest similarity is then selected as the label for text B to be classified. If none of the similarities exceed the threshold Sim... rel If so, then text B is non-garbage text.

[0206] In one possible implementation, the weights corresponding to the multi-dimensional features can be combined to identify the text to be identified. This process includes: the computer device determining the weights of the text to be identified based on the first weight corresponding to each positive context feature in each waste category, the second weight corresponding to each negative context feature, the third weight corresponding to each topic word, the fourth weight corresponding to each feature word, and the fifth weight corresponding to each entity word; the first weight is the pattern support corresponding to each positive context feature; the third weight is the ability of the topic word to distinguish waste categories; the fifth weight is the importance of the waste category; the second weight is the pattern support corresponding to each negative context feature; and the weights of the text to be identified are determined based on the first weight corresponding to each positive context feature, the second weight corresponding to each negative context feature, the third weight corresponding to each topic word, the fourth weight corresponding to each feature word, and the fifth weight corresponding to each entity word. For example, the weights corresponding to at least one of the positive and negative context features, feature words, topic words, or entity words that match the text to be identified can be used as the pattern support corresponding to the positive context feature words of the text to be identified.

[0207] The information recognition method provided in this application obtains the text feature vector of the text to be recognized; and determines whether the text to be recognized belongs to spam based on the similarity between the sample feature vector of each sample in the sample library and the text feature vector. Specifically, when obtaining the sample feature vector of a sample, a sample library is constructed. For each spam category, a positive context feature library corresponding to the spam category is obtained based on the positive sample set of that spam category, and a negative context feature library corresponding to the spam category is obtained based on the negative sample set of that spam category. Furthermore, spam category statistics are performed on the positive sample set to obtain feature words for each spam category. These feature words include at least one of the following: topic words, category keywords, or entity names. By obtaining the positive and negative context features of the sample, as well as multi-dimensional features such as topic words and feature words, the sample feature vector of each sample in the sample library can be obtained based on the positive and negative context feature libraries, topic words, and feature words of each spam category. This allows for the recognition of the text to be recognized based on multi-dimensional features, improving the accuracy of the recognition.

[0208] Figure 5 This is a schematic diagram of the structure of an information identification device provided in an embodiment of this application. Figure 5 As shown, the device includes:

[0209] The first acquisition module 501 is used to acquire the text feature vector of the text to be identified;

[0210] The determination module 502 is used to determine the spam indication information of the text to be identified based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector. The spam indication information is used to indicate whether the text to be identified belongs to spam information.

[0211] The device, when acquiring the sample feature vectors of each sample in the sample set, also includes:

[0212] Module 503 is used to build a sample library, which includes positive and negative sample sets belonging to each waste category;

[0213] The second acquisition module 504 is used to acquire, for each waste category, a positive context feature library corresponding to the waste category based on the positive sample set of the waste category, and a negative context feature library corresponding to the waste category based on the negative sample set of the waste category;

[0214] The third acquisition module 505 is used to perform garbage category statistics on the positive sample set and obtain the feature words of each garbage category. The feature words include at least one of the subject words, category keywords or entity name words of each garbage category.

[0215] The fourth acquisition module 506 is used to acquire the sample feature vector of each sample in the sample library based on at least one of the positive context feature library, negative context feature library, topic words, category keywords or entity name words of each waste category;

[0216] The subject terms are used to characterize the theme of all or part of the content in the sample, and the category keywords are used to distinguish the waste category to which the sample belongs.

[0217] In one possible implementation, the second acquisition module is used for:

[0218] Each positive sample is preprocessed to obtain preprocessed data for each positive sample;

[0219] The preprocessed data of each positive sample is segmented into words to obtain each first filter word;

[0220] Based on each first filter word and a preset minimum support threshold, positive context features are mined from each positive sample to obtain a positive context feature library.

[0221] In one possible implementation, the first acquisition module is used for:

[0222] The feature words included in the text to be identified are determined based on the feature words of each waste category;

[0223] Based on the fused feature vector of each character in the feature words included in the text to be identified, the text feature vector of the text to be identified is obtained;

[0224] Specifically, when acquiring the fusion feature vector of each character, the first acquisition module is used for:

[0225] Obtain the word vector of each feature word in the text to be identified, and the character vector of each character in each feature word;

[0226] For each character of each feature word, the word vector of each feature word and the character vector of the character are fused to obtain the fused feature vector of the character.

[0227] In one possible implementation, the feature words include topic words, and the third acquisition module is used for:

[0228] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0229] Based on the first word frequency of each word in each positive sample, candidate topic words for each waste category are determined from each word segment;

[0230] For each candidate keyword, the importance of the candidate keyword is determined based on its second word frequency in the positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category.

[0231] For each waste category, the keywords for that waste category are determined from the candidate keywords based on the importance of each candidate keyword for that waste category.

[0232] In one possible implementation, the feature words include category keywords, and the third acquisition module is used for:

[0233] Each positive sample is segmented into words to obtain the words contained in each positive sample;

[0234] For each word segment, determine a first ratio between the number of times the word appears in the waste category and the total number of words in the waste category;

[0235] Based on the number of samples in the positive sample set, the number of samples in the positive samples of other waste categories besides this waste category, and the first ratio, the class discrimination ability of the word for this waste category is determined.

[0236] Based on the ability of each word segment to distinguish between different types of waste, category keywords for each type of waste are determined from each word segment.

[0237] The information recognition method provided in this application obtains the text feature vector of the text to be recognized; and determines whether the text to be recognized belongs to spam based on the similarity between the sample feature vector of each sample in the sample library and the text feature vector. Specifically, when obtaining the sample feature vector of a sample, a sample library is constructed. For each spam category, a positive context feature library corresponding to the spam category is obtained based on the positive sample set of that spam category, and a negative context feature library corresponding to the spam category is obtained based on the negative sample set of that spam category. Furthermore, spam category statistics are performed on the positive sample set to obtain feature words for each spam category. These feature words include at least one of the following: topic words, category keywords, or entity names. By obtaining the positive and negative context features of the sample, as well as multi-dimensional features such as topic words and feature words, the sample feature vector of each sample in the sample library can be obtained based on the positive and negative context feature libraries, topic words, and feature words of each spam category. This allows for the recognition of the text to be recognized based on multi-dimensional features, improving the accuracy of the recognition.

[0238] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0239] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. For example... Figure 6 As shown, the computer device includes: a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the information recognition method, which, compared with related technologies, can achieve:

[0240] The information recognition method provided in this application obtains the text feature vector of the text to be recognized; and determines whether the text to be recognized belongs to spam based on the similarity between the sample feature vector of each sample in the sample library and the text feature vector. Specifically, when obtaining the sample feature vector of a sample, a sample library is constructed. For each spam category, a positive context feature library corresponding to the spam category is obtained based on the positive sample set of that spam category, and a negative context feature library corresponding to the spam category is obtained based on the negative sample set of that spam category. Furthermore, spam category statistics are performed on the positive sample set to obtain feature words for each spam category. These feature words include at least one of the following: topic words, category keywords, or entity names. By obtaining the positive and negative context features of the sample, as well as multi-dimensional features such as topic words, category keywords, or entity names, the sample feature vector of each sample in the sample library can be obtained based on the positive and negative context feature libraries, topic words, and feature words of each spam category. This allows for the recognition of the text to be recognized based on multi-dimensional features, improving the accuracy of the recognition.

[0241] In one alternative embodiment, a computer device is provided, such as Figure 6 As shown, Figure 6 The computer device 600 shown includes a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, for example, via a bus 602. Optionally, the computer device 600 may further include a transceiver 604, which can be used for data interaction between the computer device and other computer devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 604 is not limited to one type, and the structure of the computer device 600 does not constitute a limitation on the embodiments of this application.

[0242] Processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0243] Bus 602 may include a pathway for transmitting information between the aforementioned components. Bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 602 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0244] The memory 603 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0245] The memory 603 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 601. The processor 601 executes the computer programs stored in the memory 603 to implement the steps shown in the foregoing method embodiments.

[0246] Electronic devices include, but are not limited to, servers, terminals, or cloud computing center equipment.

[0247] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0248] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0249] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. The terms “comprising” and “including” as used in the embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, or operation, but do not exclude implementation as other features, information, data, steps, or operations supported by this art.

[0250] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0251] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0252] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. An information identification method, characterized in that, The method includes: Obtain the text feature vector of the text to be identified; Based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector, the spam indication information of the text to be identified is determined. The spam indication information is used to indicate whether the text to be identified belongs to spam information. The method for obtaining the sample feature vectors of each sample in the sample library includes: Construct a sample library, which includes positive and negative sample sets belonging to each waste category; For each waste category, based on the positive sample set of the waste category, a positive context feature library corresponding to the waste category is obtained, and based on the negative sample set of the waste category, a negative context feature library corresponding to the waste category is obtained; Perform waste category statistics on each positive sample set in the sample library to obtain the feature words of each waste category. The feature words include at least one of the following: topic words, category keywords, or entity name words of each waste category. For each sample in the sample library, based on the positive context feature library, negative context feature library, and feature words of the waste category to which the sample belongs, the positive context features, negative context features, and feature words included in the sample are determined; multi-dimensional feature encoding is performed on the positive context features, negative context features, and feature words included in the sample to obtain the sample feature vector; The subject terms are used to characterize the theme of all or part of the content in the sample, and the category keywords are used to distinguish the waste category to which the sample belongs.

2. The method according to claim 1, characterized in that, For each waste category, based on the positive sample set of the waste category, a positive contextual feature library corresponding to the waste category is obtained, including: Preprocess each positive sample in the positive sample set of the aforementioned waste category to obtain preprocessed data for each positive sample; The preprocessed data of each positive sample is segmented into words to obtain each first filter word; Based on each of the first filter words and the preset minimum support threshold, positive context features in each of the positive samples are mined to obtain the positive context feature library corresponding to the garbage category.

3. The method according to claim 1, characterized in that, The process of obtaining the text feature vector of the text to be identified includes: The feature words included in the text to be identified are determined based on the feature words of each waste category; Based on the fused feature vector of each character in the feature words included in the text to be identified, the text feature vector of the text to be identified is obtained; The method for obtaining the fused feature vector of each character includes: Obtain the word vector of each feature word in the text to be identified, and the character vector of each character in each feature word; For each character of each feature word, the word vector of each feature word and the character vector of the character are fused to obtain the fused feature vector of the character.

4. The method according to claim 1, characterized in that, The feature words include topic words, and the methods for obtaining the topic words include: Segment the positive samples in each set of positive samples to obtain the segmented words contained in each set of positive samples; Based on the first word frequency of each word in each of the positive samples, candidate topic words for each waste category are determined from each of the word segments; For each candidate keyword in each waste category, the importance of the candidate keyword is determined based on the second word frequency of the candidate keyword in the positive samples of that waste category and the document frequency of the candidate keyword in the positive samples of all waste categories. For each waste category, the keywords for that waste category are determined from the candidate keywords based on the importance of each candidate keyword for that waste category.

5. The method according to claim 1, characterized in that, The feature words include category keywords, and the methods for obtaining the category keywords include: Segment the positive samples in each set of positive samples to obtain the segmented words contained in each set of positive samples; For each word segment in the positive sample set of each waste category, determine a first ratio between the number of times the word appears in the waste category and the total number of words in the waste category; For each waste category, the ability of the word segment to distinguish the waste category is determined based on the number of samples in each positive sample set, the number of samples in the positive samples of other waste categories besides the waste category, and the first ratio. Based on the ability of each word segment to distinguish between different waste categories, category keywords for each waste category are determined from each word segment.

6. The method according to claim 4, characterized in that, For each candidate keyword in each garbage category, the determination of the importance of the candidate keyword based on its second term frequency in the positive samples of that garbage category and the document frequency of the candidate keyword in all positive samples of all garbage categories includes: If the vocabulary type of the candidate topic word is a common word, the relative weight of the candidate topic word is determined based on the maximum document frequency of common words in all positive samples of spam categories and the document frequency of the candidate topic word in all positive samples of spam categories. The importance of the candidate topic word is determined based on the relative weight of the candidate topic word and the second word frequency of the candidate topic word in the positive samples of the spam category. If the candidate keyword is a domain term, its relative weight is determined based on the maximum document frequency of the domain term in all positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category. Its relative importance is then determined based on its relative weight and the second word frequency in the positive samples of the spam category. Finally, its overall importance is determined based on its relative importance and the highest importance of ordinary words. If the vocabulary type of the candidate keyword is irrelevant, the importance of the candidate keyword is determined to be 0.

7. An information identification device, characterized in that, The device includes: The first acquisition module is used to acquire the text feature vector of the text to be recognized; The determination module is used to determine the spam indication information of the text to be identified based on the similarity between the sample feature vectors of each sample in the sample library and the text feature vector. The spam indication information is used to indicate whether the text to be identified belongs to spam information. The device, when acquiring the sample feature vectors of each sample in the sample library, further includes: A construction module is used to build a sample library, which includes positive and negative sample sets belonging to each waste category; The second acquisition module is used to acquire, for each waste category, a positive context feature library corresponding to the waste category based on the positive sample set of the waste category, and a negative context feature library corresponding to the waste category based on the negative sample set of the waste category; The third acquisition module is used to perform garbage category statistics on each positive sample set in the sample library and acquire the feature words of each garbage category. The feature words include at least one of the subject words, category keywords or entity name words of each garbage category. The fourth acquisition module is used to determine the positive context features, negative context features, and feature words included in each sample in the sample library, based on the positive context feature library, negative context feature library, and feature words of the garbage category to which the sample belongs; and to perform multi-dimensional feature encoding on the positive context features, negative context features, and feature words included in the sample to obtain the sample feature vector; wherein, the topic words are used to characterize the topic of all or part of the content in the sample, and the category keywords are used to distinguish the garbage category to which the sample belongs.

8. The apparatus according to claim 7, characterized in that, The second acquisition module is used for: Preprocess each positive sample in the positive sample set of the aforementioned waste category to obtain preprocessed data for each positive sample; The preprocessed data of each positive sample is segmented into words to obtain each first filter word; Based on each of the first filter words and the preset minimum support threshold, positive context features in each of the positive samples are mined to obtain the positive context feature library corresponding to the garbage category.

9. The apparatus according to claim 7, characterized in that, The first acquisition module is used for: The feature words included in the text to be identified are determined based on the feature words of each waste category; Based on the fused feature vector of each character in the feature words included in the text to be identified, the text feature vector of the text to be identified is obtained; The method for obtaining the fused feature vector of each character includes: Obtain the word vector of each feature word in the text to be identified, and the character vector of each character in each feature word; For each character of each feature word, the word vector of each feature word and the character vector of the character are fused to obtain the fused feature vector of the character.

10. The apparatus according to claim 7, characterized in that, The feature words include topic words, and the methods for obtaining the topic words include: Segment the positive samples in each set of positive samples to obtain the segmented words contained in each set of positive samples; Based on the first word frequency of each word in each of the positive samples, candidate topic words for each waste category are determined from each of the word segments; For each candidate keyword in each waste category, the importance of the candidate keyword is determined based on the second word frequency of the candidate keyword in the positive samples of that waste category and the document frequency of the candidate keyword in the positive samples of all waste categories. For each waste category, the keywords for that waste category are determined from the candidate keywords based on the importance of each candidate keyword for that waste category.

11. The apparatus according to claim 7, characterized in that, The feature words include topic words, and the methods for obtaining the topic words include: The feature words include category keywords, and the methods for obtaining the category keywords include: Segment the positive samples in each set of positive samples to obtain the segmented words contained in each set of positive samples; For each word segment in the positive sample set of each waste category, determine a first ratio between the number of times the word appears in the waste category and the total number of words in the waste category; For each waste category, the ability of the word segment to distinguish the waste category is determined based on the number of samples in each positive sample set, the number of samples in the positive samples of other waste categories besides the waste category, and the first ratio. Based on the ability of each word segment to distinguish between different waste categories, category keywords for each waste category are determined from each word segment.

12. The apparatus according to claim 10, characterized in that, For each candidate keyword, the importance of the candidate keyword is determined in the following way: If the vocabulary type of the candidate topic word is a common word, the relative weight of the candidate topic word is determined based on the maximum document frequency of common words in all positive samples of spam categories and the document frequency of the candidate topic word in all positive samples of spam categories. The importance of the candidate topic word is determined based on the relative weight of the candidate topic word and the second word frequency of the candidate topic word in the positive samples of the spam category. If the candidate keyword is a domain term, its relative weight is determined based on the maximum document frequency of the domain term in all positive samples of the spam category and the document frequency of the candidate keyword in all positive samples of the spam category. Its relative importance is then determined based on its relative weight and the second word frequency in the positive samples of the spam category. Finally, its overall importance is determined based on its relative importance and the highest importance of ordinary words. If the vocabulary type of the candidate keyword is irrelevant, the importance of the candidate keyword is determined to be 0.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images