A method and apparatus for identifying online public nuisance text

Abstract

This invention provides a method and apparatus for identifying online nuisance text, comprising: preliminarily judging the target online text by extracting URL links from the target online text; if the judgment cannot be made, calculating the invalid information score and generating a pinyin list, a keyword list, and a keyword pinyin list for the target online text; calculating the online nuisance score for each keyword by analyzing the distribution of the character elements of each keyword in the target online text and the distribution of the keyword pinyin in the pinyin list of the target online text; revising the online nuisance score based on the invalid information score, and obtaining the online nuisance text recognition result based on the revision result. This invention can accurately identify online nuisance text through fuzzy matching of online nuisance keyword characters and pinyin, and can effectively address information hiding methods such as segmentation, reverse order, vertical text arrangement, and homophones in the target text.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and specifically relates to a method and apparatus for identifying online nuisance texts. Background Technology

[0002] With the explosive growth of internet information resources, the amount of harmful online information has also increased significantly. The discovery and control of malicious websites is one of the core tasks of information content security management. Currently, the main method for discovering harmful websites is keyword matching. However, as the control of harmful online information gradually intensifies, those spreading such information actively push these websites through various channels such as SMS, Weibo, and websites, employing various methods to circumvent technical supervision within the text messages. Harmful online information is highly concealed, rendering traditional methods such as direct keyword matching less effective in identifying harmful online content. The main problems include:

[0003] 1. Senders of malicious information have taken various measures to avoid being detected, such as inserting special characters into the text, changing the word order, and replacing homophones. These circumvention methods render the direct matching method for online nuisance words ineffective.

[0004] 2. Harmful information is often spread by combining multiple information concealment methods, and even if a single keyword match is successful, it is easy to lead to misjudgment.

[0005] 3. Malicious URLs often employ various deceptive techniques, such as deliberate segmentation and the inclusion of special characters, making URL identification and extraction extremely difficult.

[0006] Therefore, the identification of online public nuisance texts requires, on the one hand, the comprehensive consideration of various fuzzy matching methods to improve the accuracy of online public nuisance information identification, rather than focusing on individual words, to avoid misjudgment; on the other hand, it requires the design of corresponding countermeasures against the common anti-monitoring methods used by information publishers, and the ability to handle various forms of online public nuisance information. Summary of the Invention

[0007] Considering the anonymity of the dissemination of harmful online information (specific texts that are illegal or violate public order and good morals), this invention provides a method and apparatus for identifying harmful online texts to improve the accuracy of such identification. By analyzing text data from internet social networking sites (such as posts, comments, and blog posts from online forums and social media platforms), the method identifies harmful online social networking site texts containing URL links and extracts these links. This identification method can accurately detect texts containing harmful online information.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for identifying online nuisance text, comprising the following steps:

[0010] 1) Extract URL links from the target online text: If no URL links exist, or if URL links exist and the text contains specific rare characters, then the result of identifying online nuisance text is obtained; if URL links exist but the text does not contain specific rare characters, then the senseless value of the target online text is calculated. degree And proceed to step 2);

[0011] 2) Generate the target network text pinyin list, keyword list, and keyword pinyin list respectively, and use each keyword kw i The distribution of character elements in the target web text and the distribution of keyword pinyin in the target web text pinyin list are analyzed, and the kw of each keyword is calculated. i network pollution sensitive_score i , where i is the keyword number;

[0012] 3) Based on the degree of invalid information, senseless degree The sensitive_score for network pollution i Revise accordingly, and based on the keyword kw i The corrected network public interest score sensitive_score′ i The results of the online public nuisance text recognition were obtained.

[0013] Furthermore, the URL links in the target web text are extracted through the following steps:

[0014] 1) Based on Unicode, mark each character in the text string as a Chinese character;

[0015] 2) Extract a set of continuous non-Chinese character substrings;

[0016] 3) Determine each non-Chinese character substring: If a substring contains the character '.' and the English characters '[A-Za-z]', then the substring is determined to be a URL link.

[0017] Furthermore, the results of identifying online nuisance texts include: non-online nuisance texts or online nuisance texts.

[0018] Furthermore, in step 1), the following strategy is used to determine whether a text is not a nuisance or is a nuisance:

[0019] 1) If no URL link exists, it is determined to be non-polluting online text;

[0020] 2) If a URL link exists and certain rare characters appear in the text, it is determined to be a piece of online nuisance text.

[0021] Furthermore, in step 3), the following strategy is used to determine whether a text is not a nuisance or is a nuisance:

[0022] 1) When each keyword kw i The corrected network public interest score sensitive_score′ i If the value is less than or equal to T, it is determined to be non-polluting online text, where T is the determination threshold.

[0023] 2) When any keyword kw i The corrected network public interest score sensitive_score′ i If the value is greater than T, it is determined to be a non-polluting online text.

[0024] Furthermore, the senseless degree of the target network text is calculated through the following steps:

[0025] 1) In the target web text with URLs removed, characters other than valid information characters are defined as invalid information characters. Valid information characters include: Chinese characters, uppercase and lowercase English letters, and numbers;

[0026] 2) Calculate the ratio of the number of invalid information characters to the total number of characters in the target network text to obtain the invalid information degree (senseless_degree).

[0027] Furthermore, the following steps are used to calculate the kw of each keyword. i network pollution sensitive_score i :

[0028] 1) For each keyword kw i Initialize keyword identifier word_exist i The pinyin identifier Py_exist i and character set identifier word_set_exist i ;

[0029] 2) If one or more keywords kw i If it appears in the target network text, then it will be based on the corresponding keyword kw i Calculate the number of times each keyword appears (kw). i Keywords: Internet pollution Change the keyword identifier word_exist i And delete the corresponding keywords kw from the target web text. i The first network text is obtained;

[0030] 3) If one or more keywords kw iIf all character elements appear sequentially in the first network text and the average distance between two adjacent character elements is less than the threshold D1, then the corresponding keyword kw is calculated based on the character element distribution. i Keywords: Internet pollution Change the keyword identifier word_exist i And delete the corresponding keywords kw from the first network text. i The character elements are used to obtain the second network text;

[0031] 4) If one or more keywords are pronounced "kp" in pinyin i If it appears in the target network text pinyin list, then it will be based on the corresponding keyword pinyin kp. i Calculate the number of times each keyword appears (kw). i The pinyin network public nuisance Change the Pinyin identifier Py_exist i And delete the corresponding keyword pinyin "kp" from the target network text pinyin list. i This yields the first list of pinyin text from the network.

[0032] 5) If one or more keywords are pronounced "kp" in pinyin i If all pinyin elements appear sequentially in the first network text pinyin list and the average distance between two adjacent pinyin elements is less than the threshold D2, then calculate the corresponding keyword kw based on the pinyin element distribution. i The pinyin network public pollution classification Change the pinyin identifier py_exist i ;

[0033] 6) For the keyword identifier word_exist i Unchanged keyword kw i The keyword kw is selected when all its elements appear in the second network text and the average distance between adjacent elements is less than the threshold D3. i Calculate the corresponding keyword kw i Character set network public pollution Change character set identifier word_set_exist i ;

[0034] 7) For the pinyin identifier word_exist i With character set identifier word_set_exist i The keyword kw remains unchanged. i Get the keyword pinyin kp where all pinyin elements appear in the first network text pinyin list and the average distance between two adjacent pinyin elements is less than the threshold D4. i Calculate the corresponding keyword kw iThe pinyin collection network public pollution classification

[0035] 8) Collective keyword kw i Keywords: Internet pollution With Pinyin network public pollution Get each keyword kw i network pollution sensitive_score i .

[0036] Furthermore, the corrected network pollution score sensitive_score′ is calculated through the following steps. i

[0037] 1) Obtain the average senseless information level of normal text. avg ;

[0038] 2) Based on average invalid information degree senseless avg Calculate the corrected network pollution score sensitive_score′ i =sensitive_score i *(1+(senseless degree -senseless avg ) / senseless avg ).

[0039] A storage medium storing a computer program, wherein the computer program is configured to execute any of the methods described above when run.

[0040] An electronic device includes a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform any of the methods described above.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] This invention can accurately identify online nuisance texts by fuzzy matching of keywords and pinyin. It can also effectively deal with information hiding methods such as segmentation, reverse order, vertical text arrangement, and homophones in the target text. The URL recognition method of this invention can efficiently extract URLs in various forms, including those with inserted spaces and the use of interference characters. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0044] To illustrate the technical solutions disclosed in this invention in detail, further explanation is provided below in conjunction with the embodiments and accompanying drawings.

[0045] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0046] (1) Extract URL links from the online text; if no URL links exist, it is determined to be non-internet pollution text; if URL links exist and specific rare characters appear in the online text, it is determined to be internet pollution text; if no judgment can be made, initialize the internet pollution score sensitive_score=0 of the text and calculate the invalid information degree senseless_degree of the text.

[0047] (2) For the keyword kw in the keyword list keywords, initialize the custom variables word_exist=0, py_exist=0 and word_set_exist=0. Word_exist indicates whether the keyword kw exists in the online text. If it exists, it is equal to 1, and if it does not exist, it is equal to 0. Py_exist indicates whether the pinyin list of the keyword kw exists in the online text. This method is used to detect homophones (there are often homophones used to replace words in online harmful texts, such as using "ma jiang" to replace "ma jiang"). Word_set_exist indicates whether all the characters that make up the keyword kw exist in the online text. This method is used to detect reversed or disordered keywords in online harmful texts, such as using "ji hu lao" to replace "laohu ji".

[0048] (3) If kw appears in the text and appears x times, then sensitive_score += x, word_exist = 1, and delete all kw in the text;

[0049] (4) If kw appears in the text in a scattered order, and the average distance between the kw characters in the text is less than the threshold D1, then sensitive_score+=1, word_exist=1, and the scattered kw characters are deleted from the text.

[0050] (5) Get the text pinyin list and the kw pinyin list. If the kw pinyin list appears continuously in the text pinyin list and appears y times, then sensitive_score+=y, py_exist=1, and delete all kw pinyin lists from the text pinyin list.

[0051] (6) If the elements of the kw pinyin list appear scattered and sequentially in the text pinyin list, and the average distance between adjacent elements of the kw pinyin list in the text pinyin list is less than D2, then sensitive_score+=0.5, py_exist=1, and delete all kw pinyin lists from the text pinyin list;

[0052] (7) Character set comparison: If word_exist = 0, and the character set consisting of kw appears in the text, and the average distance between adjacent characters in the text is less than D3, then sensitive_score + = 1, word_set_exist = 1, and the character set consisting of kw is deleted from the text.

[0053] (8) Comparison of Pinyin sets: If word_set_exist = 0 and py_exist = 0, and all elements of the kw Pinyin list appear in the text Pinyin list, and the average distance between adjacent elements in the text Pinyin list is less than D4, then sensitive_score += 0.5;

[0054] (9) Based on the invalid information degree, correct the network pollution score sensitive_score_m=sensitive_score*(1+(senseless_degree-senseless_avg) / senseless_avg). If sensitive_score_m>T, then the text is determined to be network pollution text, and True and the link are returned; otherwise, proceed to step (10).

[0055] (10) Remove kw from keywords. If keywords are not empty, return to step (2) to test the next keyword. If the keyword list is empty and sensitive_score <= T, then the text is determined to be non-network public nuisance text, and False and an empty string are returned.

[0056] Furthermore, the steps of the text link recognition method described in step (1) are as follows:

[0057] (1.1) Based on Unicode, mark each character in the text string as whether it is a Chinese character (True if yes, False if no);

[0058] (1.2) Extract a set of continuous non-Chinese character substrings;

[0059] (1.3) For each non-Chinese character substring set, if the substring contains the character '.' and also contains English characters ('[A-Za-z]'), then the substring is determined to be a URL link.

[0060] Further, the calculation method for the senseless degree in step (1) is as follows: In the text after removing URL links, characters other than valid information characters such as Chinese characters, uppercase and lowercase English letters, and numbers are defined as invalid information characters. The ratio of the number of invalid information characters to the total number of characters in the text is used as the senseless degree. The senseless_avg used in step (9) is the average senseless degree of normal text, which is obtained through the measurement of a large number of normal samples.

[0061] Furthermore, the thresholds D1, D2, D3, D4, and T mentioned in steps (4), (6), (7), (8), and (9) are reasonably set through optimization tests.

[0062] For the specific process of this invention, please refer to [link / reference]. Figure 1 As shown, it is divided into 10 parts, and the implementation of each part will be explained in detail below.

[0063] 1. For text from internet social networking sites, extract URL links from the text using URL link extraction methods; if no links are found in the text, the text is determined to be non-internet nuisance text; if URL links are found in the text and certain rare characters appear in the text, the text is determined to be internet nuisance text; if no judgment can be made, initialize the internet nuisance score of the text sensitive_score = 0, and calculate the invalid information degree of the text senseless_degree.

[0064] 2. Obtain the list of keywords related to online nuisances. For the keyword "kw" in keywords, initialize word_exist=0, py_exist=0, and word_set_exist=0.

[0065] 3. Keyword character matching: Check whether the constituent characters of kw appear completely and continuously in the text. If kw appears in the text and appears x times, then sensitive_score += x, word_exist = 1, and delete all kw in the text.

[0066] 4. Keyword character fuzzy matching: Check whether the constituent characters of kw appear scattered in the text. If kw appears in the text in a scattered order and the average distance between the constituent characters of kw in the text is less than the threshold D1, then sensitive_score+=1, word_exist=1, and the constituent characters of kw appearing in a scattered order are deleted from the text.

[0067] 5. Keyword Pinyin Matching: Check if the pinyin of "kw" appears completely and continuously in the pinyin of the text. Obtain the text pinyin list and the "kw" pinyin list. If the "kw" pinyin list appears continuously in the text pinyin list and appears y times, then sensitive_score += y, py_exist = 1, and delete all "kw" pinyin lists from the text pinyin list.

[0068] 6. Keyword pinyin fuzzy matching: Check whether the pinyin "kw" appears scattered in the text. If the elements of the pinyin "kw" list appear scattered and sequentially in the text pinyin list, and the average distance between adjacent elements of the pinyin "kw" list in the text pinyin list is less than D2, then sensitive_score += 0.5, py_exist = 1.

[0069] 7. Character set matching: If word_exist = 0, and the set of characters formed by kw appears in the text, and the average distance between adjacent characters in the text is less than D3, then sensitive_score += 1, word_set_exist = 1;

[0070] 8. Pinyin set matching: If word_set_exist = 0 and py_exist = 0, and all elements of the kw pinyin list appear in the text pinyin list, and the average distance between adjacent elements in the text pinyin list is less than D4, then sensitive_score += 0.5;

[0071] 9. Based on the invalid information degree, adjust the network pollution score sensitive_score_m = sensitive_score * (1 + (senseless_degree - senseless_avg) / senseless_avg). If sensitive_score_m > T, then the text is determined to be network pollution text from internet social networking sites, and True and the link are returned.

[0072] 10. Remove "kw" from keywords. If keywords is not empty, return part 2 to test the next keyword. If the keyword list is empty and sensitive_score <= T, then determine that the text is not from a non-polluting internet social networking site and return False and an empty string.

[0073] The link identification method in the text is as follows: First, based on Unicode, each character in the text string is marked as whether it is a Chinese character (True if yes, False if no); second, a set of continuous non-Chinese character substrings is extracted; finally, each non-Chinese character substring is judged. If the substring contains the character '.' and also contains English characters ('[A-Za-z]'), then the substring is determined to be a URL link.

[0074] The above-mentioned method for calculating the senseless degree is as follows: In the text after removing URL links, characters other than valid information characters such as Chinese characters, uppercase and lowercase English letters, and numbers are defined as invalid information characters. The ratio of the number of invalid information characters to the total number of characters in the text is used as the senseless degree. The senseless_avg used in Part 9 is the average senseless degree of normal text, obtained through measurements of a large number of normal samples. In this embodiment, senseless_avg = 0.19 is used.

[0075] Regarding the thresholds D1, D2, D3, D4, and T, this embodiment found through actual testing that setting D1=10, D2=3, D3=2, D4=2, and T=3 can achieve a better effect in recognizing online nuisance texts.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for identifying network pollution text, comprising the steps of: 1) extracting the URL link in the target network text: if there is no URL link or there is a URL link and the specific rare characters appear in the text, the network pollution text recognition result is obtained; if there is a URL link but the specific rare characters do not appear in the text, the senseless_degree of the target network text is calculated, and step 2) is entered; 2) generate target network text pinyin list, keyword list and keyword pinyin list respectively, and through each keyword kw i The distribution of the word elements in the target network text and the distribution of the keyword pinyin in the target network text pinyin list, the network pollution score sensitive_score of each keyword kw i i , wherein i is the serial number of the keyword, and the network pollution score sensitive_score of each keyword kw i i :​​ For each keyword kw i Initialize the keyword identification word exist i , the pinyin identification py exist i and the character set identification word set exist i ; If one or more keywords kw i If it appears in the target network text, then it will be based on the corresponding keyword kw i Calculate the number of times each keyword appears (kw). i Keywords: Internet pollution Change the keyword identifier word_exist i And delete the corresponding keywords kw from the target web text. i The first network text is obtained; If one or more keywords kw i all word elements of which appear sequentially in the first network text and the average distance between two adjacent word elements is less than a threshold value D1, then according to the distribution of word elements, the keyword network pollution score of the corresponding keyword kw i is calculated. The keyword identification word_exist i is changed, and the word elements of each corresponding keyword kw i in the first network text are deleted to obtain a second network text. If one or more keywords are pinyin kp i If it appears in the target network text pinyin list, then it will be based on the corresponding keyword pinyin kp. i Calculate the number of times each keyword appears (kw). i The pinyin network public nuisance Change the pinyin identifier py_exist i And delete the corresponding keyword pinyin "kp" from the target network text pinyin list. i This yields the first list of pinyin text from the network. if one or more keywords pinyin kp i all pinyin elements of which appear sequentially in the first network text pinyin list and the average distance between two adjacent pinyin elements is less than a threshold value D2, according to the distribution of pinyin elements, calculate the pinyin network pollution score of the corresponding keyword kw i kp change pinyin identifier py_exist i ; For keyword identification word_exist i Unchanged keyword kw i , obtain keywords kw whose each character element appears in the second network text and the average distance between adjacent two character elements is less than threshold D3 i , calculate the keyword network pollution score of the corresponding keyword kw i Change character set identification word_set_exist i ;​ For the pinyin identifier py exist i The character set identifier word_set_exist i The keyword kw i , the keyword pinyin kp is obtained, which is the keyword whose all pinyin elements appear in the first network text pinyin list and the average distance between adjacent two pinyin elements is less than the threshold D4 i The pinyin network pollution score of the corresponding keyword kw i is calculated collection keyword kw i keyword network pollution score pinyin network pollution score get the network pollution score sensitive_score of each keyword kw i i ;​ 3) Based on the senseless_degree of invalid information, the network pollution score sensitive_score is divided i The revised network pollution score sensitive_score' is obtained according to the modification of each keyword kw i i The network pollution text recognition result is obtained.​ 2. The method of claim 1, wherein, The URL link in the target network text is extracted by the following steps: 1) Based on the unicode judgment, each character of the text string is marked as Chinese Hanzi character or not; 2) Extract the continuous non-Hanzi character substring set; 3) Determine the non-Hanzi character substring set one by one: if the substring contains the character '.' and contains the English character '[A-Za-z]', it is determined that the substring is a URL link.

3. The method of claim 1, wherein, The network pollution text recognition result includes: non-network pollution text or network pollution text.

4. The method of claim 3, wherein, In step 1), the non-network pollution text or network pollution text is determined by the following strategy: 1) If there is no URL link, it is determined as non-network pollution text; 2) If there is a URL link and the specific rare characters appear in the text, it is determined as network pollution text.

5. The method of claim 3, wherein, In step 3), the non-network pollution text or network pollution text is determined by the following strategy: 1) when each keyword kw i sensitive_score' of the modified network pollution part i ≤ T, it is determined as a non-network pollution text, wherein T is a determination threshold; 2) When any keyword kw i of the modified network pollution sub-sensitive_score' i > T, it is determined as a non-network pollution text.

6. The method of claim 1, wherein, The senseless_degree of the target network text is calculated by the following steps: 1) In the target network text without URL link, the characters other than valid information characters are defined as invalid information characters, wherein the valid information characters include: Hanzi, English uppercase and lowercase letters and numbers; 2) Calculate the ratio of the number of invalid information characters to the total number of characters in the target network text to obtain the senseless_degree of invalid information.

7. The method of claim 1, wherein, The corrected network pollution score s sensitive_score' is calculated by the following steps i 1) obtaining the average senseless information degree senseless of normal text avg ; 2) based on the average senseless avg , the modified network pollution score sensitive_score' is calculated i = sensitive_score i *(1+(senseless degree -senseless avg ) / senseless avg ).

8. A storage medium having stored therein a computer program, wherein, The computer program is set to execute the network pollution text recognition method as claimed in any one of claims 1-7 when running.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the network pollution text recognition method as claimed in any one of claims 1-7.

Images