A big data-based information processing method and system

Abstract

This invention belongs to the field of big data information technology and discloses an information processing method and system based on big data: The method involves segmenting the sentences of the text to be detected to obtain a second set of sentences; filtering the second set of sentences using a sensitive word library to obtain a first set of candidate sensitive sentences and a third set of sentences; calculating the sentence similarity between sentences in the first set of candidate sensitive sentences and sentences in the sensitive word library, with sentences having a maximum similarity greater than or equal to a first threshold being considered sensitive sentences; recombining the morphemes of sentences in the third set of sentences; filtering the recombined sentences using a sensitive word library to obtain candidate sensitive sentences; and calculating the sentence similarity between candidate sensitive sentences and sentences in the sensitive word library, with the maximum similarity greater than or equal to a first threshold being considered sensitive sentences. 1 The statement is identified as a sensitive statement at that time; the maximum similarity is less than TH. 1 But greater than or equal to TH 2 In such cases, the information is then reviewed manually. This invention improves the detection rate and accuracy of sensitive information.

Description

A method and system for information processing based on big data Technical Field

[0001] This invention belongs to the field of big data information technology, specifically relating to an information processing method and system based on big data. Background Technology

[0002] With the development of internet technology, global data has begun to grow explosively. The rapid increase in data volume indicates that we have now entered the era of big data. With the arrival of the big data era, the monitoring and filtering of information quality has been severely tested.

[0003] In recent years, social media platforms have proliferated, allowing users to post or view various types of comments, bullet screens, articles, and other text messages anytime, anywhere. These text messages may contain sensitive content. To create a healthy online environment and ensure social stability, timely identification and processing of sensitive information to control text quality is extremely important. Existing internet monitoring typically uses a single sensitive word database to identify sensitive words, matching text information against this database to filter out text containing sensitive information. However, statements containing sensitive words are not necessarily sensitive statements, and the unfiltered text still poses a risk of sensitivity, resulting in low accuracy in sensitive information detection.

[0004] Therefore, providing a big data-based information processing method and system to improve the detection rate and accuracy of sensitive information is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an information processing method and system based on big data.

[0006] In a first aspect, the present invention provides an information processing method based on big data, the method comprising:

[0007] Step 1: Obtain the target text and preprocess it to obtain the text to be detected;

[0008] Step 2: Extract sentences from the text data to be detected to obtain a sentence set. Perform word segmentation on the sentences in the sentence set to obtain a second sentence set. Use a sensitive word library to filter the second sentence set for sensitive words to obtain the first candidate sensitive sentence set. Sensitive sentences are sentences that contain sensitive words.

[0009] Step 3: Extract the i-th candidate sensitive statement from the first candidate sensitive statement set, calculate the statement similarity between the i-th candidate sensitive statement and statements containing the same sensitive words in the sensitive statement library, and determine the i-th candidate sensitive statement as a sensitive statement in the text to be detected when the maximum similarity SIM1 is greater than or equal to the first threshold TH1.

[0010] Step 4: Filter out sensitive statements in the first candidate sensitive statement set based on the statement set to obtain the third statement set;

[0011] Step 5: Segment the sentences in the third sentence set into words to obtain the fourth sentence set. Obtain the j-th sentence in the fourth sentence set. Reorganize the words in the j-th sentence into N new sentences. Use a sensitive word library to filter the N new sentences for sensitive words to obtain the j-th candidate sensitive sentence. Calculate the sentence similarity between the j-th candidate sensitive sentence and sentences containing the same sensitive words in the sensitive sentence library. When the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th sentence is identified as a sensitive sentence in the text to be detected. When the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th sentence is submitted for manual review to determine whether the j-th sentence is a sensitive sentence.

[0012] Specifically, in step 1, the preprocessing includes: the target text contains letters, text, and emojis;

[0013] Translate English characters into first Chinese characters, expand Chinese pinyin abbreviations into second Chinese characters, and convert emoticons into emoticon text;

[0014] The first Chinese character, the second Chinese character, the text, and the emoticon text are used as the text to be detected.

[0015] Specifically, expanding the Pinyin abbreviations to a second Chinese character includes:

[0016] Step 11: Obtain the first text information containing the Chinese Pinyin abbreviation, identify the candidate Chinese characters of the Chinese Pinyin abbreviation through the online dictionary, and generate a candidate Chinese character set;

[0017] Step 12: Extract the first text sequence containing the first text information from the target text;

[0018] Step 13: Input the candidate Chinese character set and the first text sequence into the first training model for learning, and obtain the learning result, which is the second Chinese character that conforms to the semantics.

[0019] Specifically, the steps for calculating sentence similarity include:

[0020] Step 21: Obtain M sensitive statements from the sensitive statement database that have the same sensitive words as the candidate sensitive statements;

[0021] Step 22: Extract the first keyword of the candidate sensitive statement, generate the first word vector of the first keyword, and generate the sentence vector of the candidate sensitive statement based on the first word vector;

[0022] Step 23: Extract the second keyword of the k-th sensitive statement, generate the second word vector of the second keyword, and generate the sentence vector of the k-th sensitive statement based on the second word vector;

[0023] Step 24: Calculate the similarity between the candidate sensitive statement and the k-th sensitive statement based on the sentence vector of the candidate sensitive statement and the sentence vector of the k-th sensitive statement;

[0024] The formula for calculating sentence vectors is as follows: Sem is the sentence vector, p is the number of keywords, and Wem is the vector. x Let x be the word vector of the x-th keyword.

[0025] Specifically, step 5, which involves recombining the morphemes in the j-th statement into N new statements, includes the following steps:

[0026] Step 51: Extract the morphemes from the j-th sentence and determine whether the morphemes contain any other words. If they do, proceed to step 52; otherwise, proceed to step 53. Other words are words not included in the sentence grouping dictionary.

[0027] Step 52: Input the morphemes before the alternative word into the second training model, obtain A1 words in the output of the second training model whose probability of occurrence is higher than the preset value A, use A1 words as candidate words for the alternative word, replace the alternative word in the morphemes with the candidate words, and generate A1 new morphemes. The second training model outputs the next word to be used in the sentence and the probability of occurrence of the word in the sentence to be used for the input morphemes.

[0028] Step 53: Input the morphemes to be used to form sentences into the sentence formation training model to generate B1 first new sentences. Calculate the sentence likelihood value of each of the B1 first new sentences. When the sentence likelihood value is higher than the preset value B, the first new sentence is taken as a candidate new sentence. The sentence formation training model outputs sentences and the probability of occurrence of each morpheme in the sentences for the input morphemes. The sentence likelihood value is the product of the probability of occurrence of each morpheme in the sentence.

[0029] Step 54: If the candidate new sentence contains the candidate word, then replace the candidate word with another word to generate the new sentence.

[0030] Specifically, sensitive statements in the text to be processed are marked, and the sensitive statements and marks are visualized.

[0031] Secondly, the present invention also provides an information processing system based on big data, the system comprising:

[0032] The text processing module is used to acquire target text, preprocess the target text to obtain text to be detected, extract sentences from the text to be detected data to obtain a sentence set, and segment the sentences in the sentence set to obtain a second sentence set.

[0033] The first sensitive word filtering module uses a sensitive word library to filter the second set of statements to obtain a first set of candidate sensitive statements. Based on the statement set, sensitive statements in the first set of candidate sensitive statements are filtered out to obtain a third set of statements. Sensitive statements are statements that contain sensitive words.

[0034] The first sensitive statement determination module is used to extract the i-th candidate sensitive statement from the first candidate sensitive statement set, calculate the statement similarity between the i-th candidate sensitive statement and statements containing the same sensitive words in the sensitive statement library, and determine the i-th candidate sensitive statement as a sensitive statement in the text to be detected when the maximum similarity SIM1 is greater than or equal to the first threshold TH1.

[0035] The second sensitive statement determination module segments the statements in the third statement set into a fourth statement set by word morpheme. It then obtains the j-th statement from the fourth statement set, recombines the words in the j-th statement into N new statements, filters these N new statements using a sensitive word library, and obtains the j-th candidate sensitive statement. It calculates the statement similarity between the j-th candidate sensitive statement and statements in the sensitive statement library that contain the same sensitive words. If the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th statement is determined as a sensitive statement in the text to be detected; otherwise, if the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th statement is submitted for manual review to determine whether it is a sensitive statement.

[0036] Specifically, the preprocessing includes: the target text contains letters, text, and emojis;

[0037] Translate English characters into first Chinese characters, expand Chinese pinyin abbreviations into second Chinese characters, and convert emoticons into emoticon text;

[0038] The first Chinese character, the second Chinese character, the text, and the emoticon text are used as the text to be detected.

[0039] Specifically, expanding the Pinyin abbreviations to a second Chinese character includes:

[0040] Step 11: Obtain the first text information containing the Chinese Pinyin abbreviation, identify the candidate Chinese characters of the Chinese Pinyin abbreviation through the online dictionary, and generate a candidate Chinese character set;

[0041] Step 12: Extract the first text sequence containing the first text information from the target text;

[0042] Step 13: Input the candidate Chinese character set and the first text sequence into the first training model for learning, and obtain the learning result, which is the second Chinese character that conforms to the semantics.

[0043] Specifically, the steps for calculating sentence similarity include:

[0044] Step 21: Obtain M sensitive statements from the sensitive statement database that have the same sensitive words as the candidate sensitive statements;

[0045] Step 22: Extract the first keyword of the candidate sensitive statement, generate the first word vector of the first keyword, and generate the sentence vector of the candidate sensitive statement based on the first word vector;

[0046] Step 23: Extract the second keyword of the k-th sensitive statement, generate the second word vector of the second keyword, and generate the sentence vector of the k-th sensitive statement based on the second word vector;

[0047] Step 24: Calculate the similarity between the candidate sensitive statement and the k-th sensitive statement based on the sentence vector of the candidate sensitive statement and the sentence vector of the k-th sensitive statement.

[0048] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0049] 1) This invention segments sentences in the text to be detected and then filters them for sensitive words to obtain a set of candidate sensitive sentences. Then, it calculates the similarity between the sentences in the candidate sensitive sentence set and the sentences in the sensitive sentence database. Sentences with a similarity greater than a threshold are considered sensitive sentences, which improves the detection accuracy of sensitive sentences.

[0050] 2) The j-th sentence, which does not contain sensitive words, is reconstructed. Sensitive words are filtered from the reconstructed sentences to obtain candidate sensitive sentences. Then, the similarity between the sentences in the candidate sensitive sentence set and the sentences in the sensitive sentence database is calculated. If the similarity is greater than or equal to a second threshold, the j-th sentence is considered a sensitive sentence. If the similarity is less than the second threshold but greater than or equal to a third threshold, the j-th sentence is submitted for manual judgment. This method can detect sensitive sentences that circumvent sensitive words by adding symbols or using alternative words, thus improving the detection rate of sensitive sentences. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 is a flowchart of an information processing method based on big data according to the present invention;

[0053] Figure 2 is a schematic diagram of the structure of an information processing system based on big data according to the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the specific embodiments described herein are merely illustrative of the invention and represent only a portion, not all, of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0055] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0056] Figure 1 is a flowchart of an embodiment of an information processing method based on big data provided by the present invention. The flowchart specifically includes:

[0057] Step 1: Obtain the target text and preprocess it to obtain the text to be detected.

[0058] For example, the target text can be a sentence, a paragraph, or an article.

[0059] Specifically, in step 1, preprocessing includes: the target text contains letters, text, and emojis.

[0060] Translate English characters into first Chinese characters, expand Chinese pinyin abbreviations into second Chinese characters, and convert emoticons into emoticon text;

[0061] The first Chinese character, the second Chinese character, the text, and the emoticon text are used as the text to be detected.

[0062] Specifically, expanding the Pinyin abbreviations to a second Chinese character includes:

[0063] Step 11: Obtain the first text information containing the Chinese Pinyin abbreviation, identify the candidate Chinese characters of the Chinese Pinyin abbreviation through the online dictionary, and generate a candidate Chinese character set.

[0064] Step 12: Extract the first text sequence containing the first text information from the target text.

[0065] Step 13: Input the candidate Chinese character set and the first text sequence into the first training model for learning, and obtain the learning result, which is the second Chinese character that conforms to the semantics.

[0066] The text information is arranged in descending order of appearance time to form a text information sequence. The text sequence consists of two or more sequentially arranged interactive text information. For example, "I like spicy food, I want to go home ZF" (first text information). In a preferred embodiment, the text sequence can be a dialogue. Two adjacent text information in the text sequence have a certain degree of correlation, which can be a causal relationship or other relationships such as question and answer. This is not limited here.

[0067] Preferably, the first training model is a neural network model. By inputting the candidate Chinese character set and the first text sequence into the first training model, the most suitable second Chinese character for the first text information can be selected from the candidate Chinese character set. Taking the text sequence: "I like spicy food, I want to go home ZF" (first text information) as an example, the candidate Chinese character set for ZF includes: character, cook, forward. Inputting the candidate Chinese character set and the text sequence into the first training model can select "cook" as the second Chinese character.

[0068] Step 2: Extract sentences from the text data to be detected to obtain a sentence set. Perform word segmentation on the sentences in the sentence set to obtain a second sentence set. Use a sensitive word library to filter the second sentence set for sensitive words to obtain the first candidate sensitive sentence set. Sensitive sentences are sentences that contain sensitive words.

[0069] For example, sensitive words include uncivilized language, violence, etc., and can also be user-defined sensitive words.

[0070] Step 3: Extract the i-th candidate sensitive statement from the first candidate sensitive statement set, calculate the statement similarity between the i-th candidate sensitive statement and statements containing the same sensitive words in the sensitive statement library, and determine the i-th candidate sensitive statement as a sensitive statement in the text to be detected when the maximum similarity SIM1 is greater than or equal to the first threshold TH1.

[0071] Specifically, the steps for calculating sentence similarity include:

[0072] Step 21: Obtain M sensitive statements from the sensitive statement database that have the same sensitive words as the candidate sensitive statements.

[0073] Step 22: Extract the first keyword of the candidate sensitive statement, generate the first word vector of the first keyword, and generate the sentence vector of the candidate sensitive statement based on the first word vector.

[0074] Step 23: Extract the second keyword of the k-th sensitive statement, generate the second word vector of the second keyword, and generate the sentence vector of the k-th sensitive statement based on the second word vector.

[0075] Step 24: Calculate the similarity between the candidate sensitive statement and the k-th sensitive statement based on the sentence vector of the candidate sensitive statement and the sentence vector of the k-th sensitive statement.

[0076] The formula for calculating sentence vectors is as follows: Sem is the sentence vector, p is the number of keywords, and Wem is the vector. x Let x be the word vector of the x-th keyword.

[0077] Specifically, the first and second keywords contain sensitive words.

[0078] Preferably, the first keyword and the second keyword include one or more keywords.

[0079] Preferably, the similarity between two sentences is calculated using the cosine similarity between sentence vectors.

[0080] By extracting several keywords representing the semantics of sensitive sentences and calculating sentence vectors based on the word vectors of the keywords, sensitive sentences with similar semantics to candidate sensitive sentences can be obtained from the sensitive sentence database, thus improving the detection accuracy of sensitive sentences.

[0081] Step 4: Filter out sensitive statements in the first candidate sensitive statement set based on the statement set to obtain the third statement set.

[0082] Step 5: Segment the sentences in the third sentence set into words to obtain the fourth sentence set. Obtain the j-th sentence in the fourth sentence set. Reorganize the words in the j-th sentence into N new sentences. Use a sensitive word library to filter the N new sentences for sensitive words to obtain the j-th candidate sensitive sentence. Calculate the sentence similarity between the j-th candidate sensitive sentence and sentences containing the same sensitive words in the sensitive sentence library. When the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th sentence is identified as a sensitive sentence in the text to be detected. When the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th sentence is submitted for manual review to determine whether the j-th sentence is a sensitive sentence.

[0083] Among them, morphemes are the units that make up sentences; they can be a single character or a word.

[0084] Specifically, step 5, which involves recombining the morphemes in the j-th statement into N new statements, includes the following steps:

[0085] Step 51: Extract the morphemes in the j-th statement, and determine whether the morphemes contain outlier words. If they do, proceed to Step 52; if not, proceed to Step 53. An outlier word is a word not included in the sentence-forming dictionary.

[0086] Step 52: Input the morphemes before the outlier word into the second training model, and obtain A1 words whose occurrence probabilities in the words output by the second training model are higher than the preset value A. Use the A1 words as the candidate words for the outlier word, and replace the outlier word in the morphemes with the candidate words to generate A1 groups of new morphemes. Here, the second training model outputs the next sentence-forming word and the occurrence probability of the sentence-forming word for the input morphemes.

[0087] Step 53: Input the morphemes to be sentence-formed into the sentence-forming training model to generate B1 first new statements, and calculate the sentence likelihood values of the B1 first new statements respectively. When the sentence likelihood value is higher than the preset value B, take the first new statement as the candidate new statement. Here, the sentence-forming training model outputs the sentence and the occurrence probability of each morpheme in the sentence, and the sentence likelihood value is the product of the occurrence probabilities of each morpheme in the sentence.

[0088] Step 54: If the candidate new statement contains candidate words, replace the candidate words with the outlier word to generate a new statement.

[0089] Preferably, the second training model and the sentence-forming training model are recurrent neural network models.

[0090] Take the sentence containing an outlier word as an example to illustrate the technical solution of an embodiment of the present application. The sentence containing an outlier word is: I want to go to ※, where "※" is the outlier word, and the morphemes of "I want to go to ※" are I, want, go, ※. Input "I, want, go" into the second training model, and multiple words are output. Among them, the occurrence probabilities of the two words "company" and "supermarket" are higher than the preset value. Then, take "company" and "supermarket" as the candidate words for "※", and replace "※" to generate two groups of new morphemes, namely "I, want, go, company" and "I, want, go, supermarket". After generating the candidate new statements through the sentence training model, use "※" to replace "company" and "supermarket" in the candidate new statements to generate new statements.

[0091] The technical solution of one embodiment of this application is illustrated by taking a sentence without any misspellings as an example. The sentence without misspellings is: He always speaks to me with heavy words and long paragraphs. The morphemes in the sentence are extracted as follows: he, to, I, speak, always, words, heavy, words, heart, long, paragraph. The above morphemes are input into the sentence training model, and the candidate new sentences SE1 is He always speaks to me with heavy words and long paragraphs, and SE2 is He always speaks to me with a lot of nonsense. The sentence likelihood value of SE1 is L(SE1) = p(he) × p(to) × p(I) × p(speak) × p(always) × p(words) × p(heavy) × p(heart) × p(long), and the sentence likelihood value of SE2 is L(SE2) = p(he) × p(to) × p(I) × p(speak) × p(always) × p(heavy) × p(words) × p(long).

[0092] When adding symbols or using alternative words to circumvent sensitive statements containing sensitive words in a sentence, a new statement can be generated by sentence recombination. Then, sensitive words can be filtered based on the newly generated statement, and the similarity between the statement containing sensitive words and the sensitive statements in the sensitive statement database can be calculated, which can improve the detection rate of sensitive statements.

[0093] Specifically, sensitive statements in the text to be processed are marked, and the sensitive statements and marks are visualized.

[0094] Figure 2 is a schematic diagram of an embodiment of an information processing method system based on big data provided by the present invention. As shown in Figure 2, the system includes:

[0095] The text processing module is used to acquire target text, preprocess the target text to obtain text to be detected, extract sentences from the text to be detected, obtain a set of sentences, and perform word segmentation on the sentences in the set of sentences to obtain a second set of sentences.

[0096] The first sensitive word filtering module uses a sensitive word library to filter the second set of statements, obtains the first candidate set of sensitive statements, filters out the sensitive statements in the first candidate set of sensitive statements, and obtains the third set of statements, where the sensitive statements are those containing sensitive words.

[0097] The first sensitive statement determination module is used to extract the i-th candidate sensitive statement from the first candidate sensitive statement set, calculate the statement similarity between the i-th candidate sensitive statement and statements containing the same sensitive words in the sensitive statement library, and determine the i-th candidate sensitive statement as a sensitive statement in the text to be detected when the maximum similarity SIM1 is greater than or equal to the first threshold TH1.

[0098] The second sensitive statement determination module segments the statements in the third statement set into a fourth statement set by word morpheme. It then obtains the j-th statement from the fourth statement set, recombines the words in the j-th statement into N new statements, filters these N new statements using a sensitive word library, and obtains the j-th candidate sensitive statement. It calculates the statement similarity between the j-th candidate sensitive statement and statements in the sensitive statement library that contain the same sensitive words. If the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th statement is determined as a sensitive statement in the text to be detected; otherwise, if the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th statement is submitted for manual review to determine whether it is a sensitive statement.

[0099] Specifically, preprocessing includes: the target text contains letters, words, and emojis.

[0100] Translate English characters into first Chinese characters, expand Chinese pinyin abbreviations into second Chinese characters, and convert emoticons into emoticon text;

[0101] The first Chinese character, the second Chinese character, the text, and the emoticon text are used as the text to be detected.

[0102] Specifically, expanding the Pinyin abbreviations to a second Chinese character includes:

[0103] Step 11: Obtain the first text information containing the Chinese Pinyin abbreviation, identify the candidate Chinese characters of the Chinese Pinyin abbreviation through the online dictionary, and generate a candidate Chinese character set.

[0104] Step 12: Extract the first text sequence containing the first text information from the target text.

[0105] Step 13: Input the candidate Chinese character set and the first text sequence into the first training model for learning, and obtain the learning result, which is the second Chinese character that conforms to the semantics.

[0106] Specifically, the steps for calculating sentence similarity include:

[0107] Step 21: Obtain M sensitive statements from the sensitive statement database that have the same sensitive words as the candidate sensitive statements.

[0108] Step 22: Extract the first keyword of the candidate sensitive statement, generate the first word vector of the first keyword, and generate the sentence vector of the candidate sensitive statement based on the first word vector.

[0109] Step 23: Extract the second keyword of the k-th sensitive statement, generate the second word vector of the second keyword, and generate the sentence vector of the k-th sensitive statement based on the second word vector.

[0110] Step 24: Calculate the similarity between the candidate sensitive statement and the k-th sensitive statement based on the sentence vector of the candidate sensitive statement and the sentence vector of the k-th sensitive statement.

[0111] The above-described embodiments merely illustrate preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A big data-based information processing method, characterized in that, The process includes the following steps: Step 1: Obtain the target text and preprocess it to obtain the text to be detected; Step 2: Extract sentences from the text to be detected to obtain a sentence set, segment the sentences in the sentence set to obtain a second sentence set, filter the second sentence set using a sensitive word library to obtain a first candidate sensitive sentence set, where sensitive sentences are sentences containing sensitive words; Step 3: Extract the i-th candidate sensitive sentence from the first candidate sensitive sentence set, calculate the sentence similarity between the i-th candidate sensitive sentence and sentences containing the same sensitive words in the sensitive sentence library, and when the maximum similarity SIM1 is greater than or equal to the first threshold TH1, determine the i-th candidate sensitive sentence as a sensitive sentence in the text to be detected; Step 4: Filter out sentences from the first candidate sensitive sentence set based on the sentence set. Step 5: Segment the sentences in the third set of candidate sensitive sentences to obtain the fourth set of sentences. Obtain the j-th sentence in the fourth set of sentences. Reorganize the morphemes in the j-th sentence into N new sentences. Use a sensitive word library to filter the N new sentences for sensitive words to obtain the j-th candidate sensitive sentence. Calculate the sentence similarity between the j-th candidate sensitive sentence and sentences containing the same sensitive words in the sensitive sentence library. When the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th sentence is identified as a sensitive sentence in the text to be detected. When the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th sentence is handed over to a human review to determine whether the j-th sentence is a sensitive sentence. Reorganize the morphemes in the j-th statement into N new statements, including: Step 51, extract the morphemes in the j-th statement, determine whether the morphemes contain alternative words, if they do, proceed to step 52, if they do not, proceed to step 53, where alternative words are words not included in the sentence grouping dictionary; Step 52, input the morphemes before the alternative words into the second training model, obtain A1 words from the output of the second training model whose probability of occurrence is higher than the preset value A, use A1 words as candidate words for alternative words, replace the alternative words in the morphemes with the candidate words, and generate A1 new morphemes. The second training model is used to process the input... Step 53: Input the morphemes to be formed into a sentence and output the next word in the sentence and the probability of occurrence of the word in the sentence; Step 54: Input the morphemes to be formed into a sentence into the sentence training model to generate B1 first new sentences, calculate the sentence likelihood value of each of the B1 first new sentences, and when the sentence likelihood value is higher than the preset value B, the first new sentence is taken as a candidate new sentence. The sentence training model outputs a sentence and the probability of occurrence of each morpheme in the sentence for the input morphemes, and the sentence likelihood value is the product of the probability of occurrence of each morpheme in the sentence; Step 55: If the candidate new sentence contains a candidate word, replace the candidate word with another word to generate a new sentence.

2. The information processing method based on big data according to claim 1, characterized in that, In step 1, the preprocessing includes: the target text contains letters, characters, and emojis; translating English characters into first Chinese characters, expanding Chinese pinyin abbreviations into second Chinese characters, and converting emojis into emoji text; and using the first Chinese characters, second Chinese characters, characters, and emoji text as the text to be detected.

3. The information processing method based on big data according to claim 2, characterized in that, Expanding a Chinese Pinyin abbreviation into a second Chinese character includes: Step 11, obtaining first text information containing the Chinese Pinyin abbreviation, identifying candidate Chinese characters for the Chinese Pinyin abbreviation through a network thesaurus, and generating a candidate Chinese character set; Step 12, extracting a first text sequence containing the first text information from the target text; Step 13, inputting the candidate Chinese character set and the first text sequence into a first training model for learning, obtaining the learning result, which is a second Chinese character that conforms to semantics.

4. The information processing method based on big data according to claim 1, characterized in that, The steps for calculating sentence similarity include: Step 21, obtaining M sensitive sentences with the same sensitive words as the candidate sensitive sentences from the sensitive sentence database; Step 22, extracting the first keyword of the candidate sensitive sentences, generating the first word vector of the first keyword, and generating the sentence vector of the candidate sensitive sentences based on the first word vector; Step 23, extracting the second keyword of the k-th sensitive sentence, generating the second word vector of the second keyword, and generating the sentence vector of the k-th sensitive sentence based on the second word vector; Step 24, calculating the similarity between the candidate sensitive sentences and the k-th sensitive sentences based on the sentence vectors of the candidate sensitive sentences and the sentence vector of the k-th sensitive sentences; wherein, the formula for calculating the sentence vector is: ， Let p be the sentence vector, and p be the number of keywords. Let x be the word vector of the x-th keyword.

5. The information processing method based on big data according to claim 1, characterized in that, The system marks sensitive statements in the text to be processed and visualizes the sensitive statements and the marks.

6. An information processing system based on big data, used to implement the method as described in any one of claims 1-5, characterized in that, include: The text processing module is used to acquire target text, preprocess the target text to obtain text to be detected, extract sentences from the text to be detected data to obtain a sentence set, and segment the sentences in the sentence set to obtain a second sentence set. The first sensitive word filtering module uses a sensitive word library to filter the second set of statements to obtain a first set of candidate sensitive statements. Based on the statement set, sensitive statements in the first set of candidate sensitive statements are filtered out to obtain a third set of statements. Sensitive statements are statements that contain sensitive words. The first sensitive statement determination module is used to extract the i-th candidate sensitive statement from the first candidate sensitive statement set, calculate the statement similarity between the i-th candidate sensitive statement and statements containing the same sensitive words in the sensitive statement library, and determine the i-th candidate sensitive statement as a sensitive statement in the text to be detected when the maximum similarity SIM1 is greater than or equal to the first threshold TH1. The second sensitive statement determination module segments the statements in the third statement set into a fourth statement set by word morpheme. It then obtains the j-th statement from the fourth statement set, recombines the words in the j-th statement into N new statements, filters these N new statements using a sensitive word library, and obtains the j-th candidate sensitive statement. It calculates the statement similarity between the j-th candidate sensitive statement and statements in the sensitive statement library that contain the same sensitive words. If the maximum similarity SIM2 satisfies SIM2≥TH1, the j-th statement is determined as a sensitive statement in the text to be detected; otherwise, if the maximum similarity SIM2 satisfies TH2≤SIM2<TH1, the j-th statement is submitted for manual review to determine whether it is a sensitive statement.

Images