Text auditing method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a text auditing method and device, electronic equipment, a storage medium and a product, and relates to the technical field of text auditing. The text auditing method comprises the steps that sensitive words in a to-be-audited text are acquired, and position information and suspicious types of the sensitive words are determined; generating statement embedding of the to-be-audited text and sensitive word embedding of sensitive words according to the position information, and determining suspicious type embedding of suspicious types; determining the content security probability of the to-be-audited text according to statement embedding, sensitive word embedding and suspicious type embedding based on a preset sensitive extraction model; and determining the security condition of the to-be-audited text according to the content security probability. According to the embodiment of the invention, the safety condition of the to-be-audited text is determined by combining the sentence pattern characteristics, the position where the sensitive word is embedded and the sensitive word category, misjudgment of the safety condition of the to-be-audited text caused by hitting the preset sensitive word bank is avoided, and the accuracy of text safety judgment is improved.

Description

Technical Field

[0001] The present invention relates to the field of text review technology, and in particular to a text review method, device, electronic equipment, storage medium and product. Background Art

[0002] Text content review is a process that uses artificial intelligence technology to automatically identify, classify and filter text content on the Internet. It aims to ensure that text content meets specific specifications and standards, eliminate illegal, inappropriate or harmful information, and maintain the healthy development of the network environment. Text content review has a wide range of application scenarios, including social media, online platforms, etc. The current text content review mainly ensures that the uploaded content complies with regulations through a variety of means such as formulating clear community guidelines, establishing an automatic review mechanism, encouraging user reporting and feedback, and conducting manual review.

[0003] Since the workload of text content review is very large, online systems widely use automatic content review mechanisms based on machine learning to help complete the initial screening of large amounts of text content, such as establishing a keyword filtering system and using sentiment analysis technology. Automatic text review technology mainly includes two types of methods. The first is a rule-based review strategy, which is to discover and filter unsafe text content by setting a series of rules and key / sensitive word libraries. The second is content detection and recognition based on natural language models, which mainly identifies unsafe content through syntactic, lexical, and sentiment analysis, as well as understanding of the context, combined with text detection or classification algorithms.

[0004] The rule-based audit strategy can flexibly add / change the vocabulary and rules according to external changes, and quickly adapt to the changing language environment, but the rules have a weak understanding of the language connotation. Strict filtering rules will mistakenly kill a large number of normal text contents, and loose filtering rules can be easily bypassed by attackers. The audit strategy based on the natural language model has a stronger ability to understand language than the rule strategy, but there are still problems such as missing the detection of ambiguous semantics and insufficient adaptability to new vocabulary. How to improve the accuracy of text auditing has become a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a text review method, device, electronic device, storage medium and product to solve the problem of low accuracy of text review in the prior art.

[0006] According to one aspect of the present invention, a text review method is provided, wherein the method comprises:

[0007] Obtain sensitive words in the text to be reviewed, and determine the location information and suspicious type of the sensitive words;

[0008] Generating sentence embedding of the text to be reviewed and sensitive word embedding of the sensitive word according to the position information, and determining the suspicious type embedding of the suspicious type;

[0009] Determine the content safety probability of the to-be-audited text according to the sentence embedding, the sensitive word embedding, and the suspicious type embedding based on a preset sensitive extraction model;

[0010] The safety status of the text to be reviewed is determined according to the content safety probability.

[0011] According to another aspect of the present invention, a text review device is provided, wherein the device comprises:

[0012] A sensitive word determination module is used to obtain sensitive words in the text to be reviewed and determine the location information and suspicious type of the sensitive words;

[0013] An information embedding determination module, configured to generate sentence embeddings of the text to be reviewed and sensitive word embeddings of the sensitive words according to the position information, and determine suspicious type embeddings of the suspicious type;

[0014] A security probability determination module, configured to determine the content security probability of the to-be-audited text according to the sentence embedding, the sensitive word embedding, and the suspicious type embedding based on a preset sensitive extraction model;

[0015] A security situation determination module is used to determine the security situation of the text to be reviewed based on the content security probability.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text review method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text review method described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the text review method of any embodiment of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention obtains sensitive words in the text to be reviewed, determines the position information of the sensitive words and the suspicious type, generates sentence embeddings of the text to be reviewed and sensitive word embeddings of the sensitive words according to the position information, and determines the suspicious type embeddings of the suspicious type, determines the content safety probability of the text to be reviewed according to the sentence embeddings, sensitive word embeddings and suspicious type embeddings based on a preset sensitive extraction model, determines the security situation of the text to be reviewed according to the content safety probability, and realizes the determination of the security situation of the text to be reviewed in combination with the sentence characteristics, the position of the sensitive word embedding and the sensitive word category, avoids the wrong judgment of the security situation of the text to be reviewed due to hitting the preset sensitive word library, and improves the accuracy of text security judgment.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flowchart of a text review method provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a text review method provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a flowchart of a text review method provided according to Embodiment 3 of the present invention;

[0028] Figure 4 is a flowchart of another text review method provided according to Embodiment 4 of the present invention;

[0029] Figure 5 is a structural diagram of a text review device provided according to Embodiment 3 of the present invention;

[0030] Figure 6 It is a structural schematic diagram of an electronic device for implementing a text review method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 1 is a flowchart of a text audit method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of determining the security of a text to be audited. The method can be executed by a text audit device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110: Acquire sensitive words in the text to be reviewed, and determine the location information and suspicious type of the sensitive words.

[0036] Among them, the text to be reviewed can be understood as the text that needs to be reviewed for safety. Generally speaking, the text to be reviewed can be the text content of social media or online platforms. Sensitive words may exist in the text to be reviewed, that is, there may be unsafe content in the text to be reviewed. Sensitive words refer to words whose content, meaning or associations may cause controversy, discomfort, violate laws and regulations or threaten safety in a specific environment and context. Location information refers to the location of sensitive words in the text to be reviewed. Suspicious types can be understood as categories of sensitive words. For example, suspicious types can include suspicious types of sensitive words. Generally speaking, suspicious types can include but are not limited to illegal crimes, hatred / harassment / insults / immorality, physical / mental harm, fraud or deception, pornographic content, sensitive topics, privacy / copyright violations, special references, etc.

[0037] In an embodiment, the text to be reviewed can be segmented, and each phrase can be matched with pre-stored sensitive words to identify sensitive words in the text to be reviewed; or, the pre-stored sensitive words can be directly matched with the text to be reviewed to determine the sensitive words in the text to be reviewed.

[0038] The position of the sensitive word in the text to be reviewed is used as the position information, and the suspicious type of the sensitive word is determined as the suspicious type. In the actual operation process, the suspicious type of each sensitive word can be preset, and the corresponding suspicious type can be determined according to the sensitive word.

[0039] S120: Generate sentence embeddings of the text to be reviewed and sensitive word embeddings of sensitive words according to the position information, and determine suspicious type embeddings of suspicious types.

[0040] Among them, sentence embedding refers to converting sentences into numerical vector representations in a low-dimensional continuous vector space; sensitive word embedding refers to converting sensitive words into numerical vector representations in a low-dimensional continuous vector space; suspicious type embedding refers to converting suspicious types into numerical vector representations in a low-dimensional continuous vector space.

[0041] In an embodiment, sensitive words in the text to be reviewed can be extracted according to the position information, a vector representation of the text to be reviewed after the sensitive words are extracted can be determined to obtain sentence embedding, and then the sensitive words are spliced ​​according to the position information, and the vector representation of the spliced ​​sensitive words is determined as the sensitive word embedding. The suspicious types are spliced ​​according to the position information corresponding to the sensitive words, and the vector representation of the spliced ​​characteristic information is determined as the sensitive word embedding. Alternatively, the vector representation of the text to be reviewed can be determined first, and the sensitive words in the vector representation of the text to be reviewed can be determined according to the position information, and the sensitive words can be removed as sentence embedding; the vector representations corresponding to the sensitive words are spliced ​​to obtain sensitive word embedding, the suspicious types are encoded, and spliced ​​according to the position information to obtain the suspicious type embedding.

[0042] In one embodiment, the text to be reviewed can be first converted into a vector representation to obtain a word vector for each character. The vector corresponding to the sensitive word is identified by embedding a mask, and the vector corresponding to the sensitive word is hidden to obtain a sentence embedding. The sensitive words are then spliced ​​and converted into vectors according to the position information to obtain a sensitive word embedding. The preset codes of each suspicious type can be determined, and the preset codes of the suspicious types corresponding to the sensitive words of the suspicious type can be spliced ​​according to the position information to obtain the suspicious type embedding. Exemplarily, the preset codes can be predetermined by one-hot coding. In the actual operation process, before splicing sensitive words or suspicious types, the sensitive words and suspicious types can be separated by [sep]. In the actual operation process, the preset codes of the suspicious types may be insufficient in length, and the insufficient length can be filled by [PAD] to facilitate the division of each sensitive word or suspicious class.

[0043] S130. Determine the content safety probability of the text to be reviewed according to sentence embedding, sensitive word embedding, and suspicious type embedding based on a preset sensitivity extraction model.

[0044] Among them, the preset sensitive extraction model can be a model that is pre-created and trained to determine the content safety probability of the text to be reviewed. Among them, the preset sensitive extraction model consists of a Bert layer, two encoding layers, a self-attention layer, and a fully connected layer. The two encoding layers are connected, the Bert layer and the two encoding layers are respectively connected to the self-attention layer, and the self-attention layer is connected to the fully connected layer. The content safety probability can be understood as the probability that the text content in the text to be reviewed is safe, and the content safety probability can be between 0 and 1.

[0045] In an embodiment, sentence embedding, sensitive word embedding and suspicious type embedding may be input into a preset sensitive extraction model, which performs feature fusion on sentence embedding, sensitive word embedding and suspicious type embedding to determine the content safety probability of the text to be reviewed.

[0046] S140. Determine the security status of the text to be reviewed based on the content security probability.

[0047] In an embodiment, the security situation of the text to be reviewed can be determined according to the content security probability. Generally speaking, a content security probability threshold value can be preset, such as 0.5. When the content security probability is greater than or equal to the preset content security probability threshold value, the security situation of the text to be reviewed can be determined to be safe; when the content security probability is less than the preset content security probability threshold value, the security situation of the text to be reviewed is determined to be unsafe.

[0048] The embodiments of the present invention obtain sensitive words in the text to be reviewed, determine the position information of the sensitive words and the suspicious type, generate sentence embeddings of the text to be reviewed and sensitive word embeddings of the sensitive words according to the position information, and determine the suspicious type embeddings of the suspicious type, determine the content safety probability of the text to be reviewed according to the sentence embeddings, sensitive word embeddings and suspicious type embeddings based on a preset sensitive extraction model, determine the safety situation of the text to be reviewed according to the content safety probability, and realize the determination of the safety situation of the text to be reviewed in combination with the sentence characteristics, the position of the sensitive word embeddings and the sensitive word categories, avoid the situation where the safety situation of the text to be reviewed is judged incorrectly by only judging the safety situation of the text to be reviewed by the sensitive word library, and improve the accuracy of text safety judgment.

[0049] Embodiment 2

[0050] Figure 2 1 is a flowchart of a text review method provided according to Embodiment 2 of the present invention. This embodiment is further optimized and expanded based on the above embodiment and can be combined with various optional technical solutions in the above embodiment. Figure 2 As shown, the method includes:

[0051] S201: Match sensitive words in the text to be reviewed according to a preset sensitive word database to determine the suspicious type of the sensitive words.

[0052] The preset sensitive word word library can be understood as a pre-set word library for storing sensitive word words, which is a word library composed of words that may actually contain unsafe content. The preset sensitive word word library may include nouns and gerunds. For each sensitive word in the preset sensitive word word library, a corresponding suspicious type can be pre-set.

[0053] In an embodiment, the text to be reviewed may be matched with a preset sensitive word library to determine the sensitive words contained in the text to be reviewed, and to identify the suspicious types of the sensitive words.

[0054] S202: Identify the position of the sensitive word in the text to be reviewed, and use the position as the position information of the sensitive word.

[0055] In an embodiment, the position where the sensitive word is located in the text to be reviewed may be determined, and the position may be used as the position information of the sensitive word.

[0056] S203: Convert each character in the text to be reviewed into a word vector.

[0057] In an embodiment, the word vector corresponding to each character in the text to be reviewed can be determined. In the actual operation process, the word vector of each character can be determined by multiple methods such as One-hot encoding, neural network model (such as Word2Vec), etc.

[0058] S204: determine the text type of the word vector in sequence according to the position information, determine the embedding mask of the word vector based on the text type, and determine the sentence embedding of the text to be reviewed according to the embedding mask.

[0059] The text types include sensitive words and non-sensitive words. Embedding masking can be understood as setting a special value (such as 0 or other specific mark) for the word vector at a specific position to hide or ignore the corresponding word vector.

[0060] In an embodiment, the text type of the word vector can be determined based on the sequential position information, that is, for the characters corresponding to the position information of the sensitive word, the text type can be a sensitive word. The embedding mask of each word vector can be determined according to the text type, and the sentence embedding of the text to be reviewed can be determined according to the embedding mask.

[0061] In one embodiment, determining the text type of the word vector in sequence according to the position information, determining the embedding mask of the word vector based on the text type, and determining the sentence embedding of the text to be reviewed according to the embedding mask includes:

[0062] When it is determined that the word vector belongs to a sensitive word, the word vector is marked with the first embedding mask;

[0063] When it is determined that the word vector belongs to a non-sensitive word, the word vector is marked with a second embedding mask, and the first embedding mask and the second embedding mask are used as embedding masks;

[0064] The word vectors corresponding to the second embedding mask are concatenated to obtain the sentence embedding of the text to be reviewed.

[0065] Among them, the first embedding mask refers to the embedding mask of the characters corresponding to the sensitive words, and the first embedding mask is 0; the second embedding mask refers to the embedding mask of the characters corresponding to the non-sensitive words, and the second embedding mask is 1.

[0066] In an embodiment, when it is determined that a word vector belongs to a sensitive word, the word vector may be marked with a first embedding mask; when it is determined that a word vector belongs to a non-sensitive word, the word vector may be marked with a second embedding mask. The first embedding mask and the second embedding mask may be used as embedding masks, and the word vectors corresponding to the second embedding mask may be concatenated to obtain the sentence embedding of the text to be reviewed. That is, the sensitive word information is hidden.

[0067] S205: Concatenate the sensitive words according to the position information to obtain the target sensitive words, and convert the target sensitive words into vector form to obtain sensitive word embedding.

[0068] In an embodiment, sensitive words can be extracted according to their position information, and the sensitive words can be spliced ​​according to the order of appearance in the text to be reviewed corresponding to the position information to obtain target sensitive words, and the target sensitive words can be converted into vector form to obtain sensitive word embedding. In the actual operation process, before splicing sensitive words, the sensitive words can be separated by [sep] to facilitate the division of each sensitive word.

[0069] S206: Determine the preset codes of each suspicious type, and concatenate the preset codes of the suspicious types corresponding to the sensitive words according to the position information to obtain the suspicious type embedding.

[0070] In an embodiment, the preset code of each suspicious type can be determined, the corresponding preset code can be extracted according to the position information of the sensitive word, and the preset codes can be spliced ​​to obtain the suspicious type embedding. In the actual operation process, before splicing the preset codes, the preset codes of the suspicious types can be separated by [sep] to facilitate the division of each suspicious type. When the preset code length corresponding to the type is insufficient, it can be filled by [PAD]. In one embodiment, the preset code can be pre-set based on One-hot coding.

[0071] S207, input the sentence embedding, sensitive word embedding and suspicious type embedding into a preset sensitive extraction model.

[0072] Among them, the preset sensitive extraction model consists of a Bert layer, two encoding layers, a self-attention layer and a fully connected layer.

[0073] S208. Extract sentence features of sentence embedding based on the Bert layer, and extract sensitive word features of sensitive word embedding based on the two encoding layers.

[0074] In an embodiment, the sentence embedding may be input into the Bert layer, and the sentence features of the sentence embedding may be determined through the Bert layer. The sensitive word embedding may be input into two encoding layers in sequence, and the sensitive word features of the sensitive word embedding may be determined through the two encoding layers.

[0075] S209. Determine the attention weights of sentence features, sensitive word features and suspicious types respectively according to the sentence features, sensitive word features and suspicious type embedding through the self-attention layer, and determine the fusion feature information according to the attention weights and the sentence features, sensitive word features and suspicious types.

[0076] In an embodiment, sentence features, sensitive word features and suspicious type embeddings can be simultaneously input into the self-attention layer, and the attention weights corresponding to the sentence features, sensitive word features and suspicious types are determined through the self-attention layer; the product of the sentence features and the attention weights corresponding to the sentence features, the product of the sensitive word features and the attention weights corresponding to the sensitive word features, and the product of the suspicious type embedding and the attention weights corresponding to the suspicious type embedding are determined respectively, and the sum of the products is used as the fused feature information.

[0077] S210. Determine the content safety probability of the text to be reviewed based on the fused feature information based on the fully connected layer.

[0078] In an embodiment, the fused feature information may be input into a fully connected layer, and the fully connected layer may determine the content safety probability of the to-be-audited text according to the fused feature information.

[0079] S211. When the content safety probability is greater than or equal to a preset content safety probability threshold, determine that the security status of the text to be reviewed is safe.

[0080] The preset content safety probability threshold may be understood as a preset threshold for determining whether the security situation of the text to be reviewed is safe. Exemplarily, the preset content safety probability threshold may be 0.5.

[0081] In an embodiment, if the content safety probability is greater than or equal to a preset content safety probability threshold, it can be determined that the security status of the text to be reviewed is safe.

[0082] S212: When the content safety probability is less than a preset content safety probability threshold, determine that the safety status of the text to be reviewed is unsafe.

[0083] In an embodiment, if the content safety probability is less than a preset content safety probability threshold, it can be determined that the security status of the text to be reviewed is unsafe.

[0084] In an embodiment of the present invention, by matching sensitive words in a text to be reviewed according to a preset sensitive word library, determining suspicious types of sensitive words, identifying positions of sensitive words in the text to be reviewed, using the positions as position information of sensitive words, converting each character in the text to be reviewed into a word vector, determining the text type of the word vector in turn according to the position information, determining an embedding mask of the word vector based on the text type, and determining sentence embedding of the text to be reviewed according to the embedding mask, splicing sensitive words according to the position information to obtain target sensitive words, converting the target sensitive words into a vector form to obtain sensitive word embedding, determining preset codes of each suspicious type, splicing preset codes of suspicious types corresponding to sensitive words according to the position information to obtain suspicious type embedding; and embedding sentence, sensitive word and suspicious type embedding. A preset sensitive extraction model is input, sentence features of sentence embedding are extracted based on the Bert layer, and sensitive word features of sensitive word embedding are extracted based on the two encoding layers, so as to realize the extraction of sentence features and sensitive word features; the attention weights of sentence features, sensitive word features and suspicious types are respectively determined through the self-attention layer according to the sentence features, sensitive word features and suspicious types, fused feature information is determined according to the attention weights and sentence features, sensitive word features and suspicious types, the content safety probability of the text to be reviewed is determined according to the fused feature information based on the fully connected layer, the security situation of the text to be reviewed is determined to be safe according to the content safety probability, the security situation of the text to be reviewed is automatically determined, and the efficiency and accuracy of the security situation judgment of the text to be reviewed is improved.

[0085] Embodiment 3

[0086] Figure 3 is a flowchart of a text audit method provided according to the third embodiment of the present invention. This embodiment is based on the above embodiment, takes the sensitive word library as the preset sensitive word library, and takes the sensitive word category embedding as the suspicious type embedding as an example, and specifically describes a text audit method. Figure 3 As shown, the method includes:

[0087] Step 1: Enter the text to be reviewed.

[0088] Step 2: Obtain a pre-built sensitive word library, and match sensitive words and sensitive types of sensitive words through the sensitive word library.

[0089] Among them, the sensitive word library refers to the word library composed of words that may involve unsafe content. The sensitive word library in this method only contains nouns or gerund structures, and does not contain verbs alone. According to the suspicious types of sensitive words, they are divided into 8 categories: illegal crimes, hatred / harassment / insults / immorality, physical / mental harm, fraud or deception, pornographic content, sensitive topics, privacy / copyright violations, and special references.

[0090] When the content is matched, the text to be reviewed and the words in the sensitive word library can be matched one by one. If they match, the position of the sensitive word in the text to be reviewed is marked. When the type is matched, the suspicious type to which the matched sensitive word belongs can be marked. In one embodiment, the suspicious type can be marked when the sensitive word library is constructed.

[0091] Step 3: Determine sentence embedding, sensitive word embedding, and sensitive word category embedding.

[0092] In actual operation, the text to be reviewed can be converted into a vector representation to obtain a word vector T(1, n, H), where n is the number of characters and H is the hidden dimension. Based on the content matching result in step 2, a sentence embedding mask S(1, n, H) is generated, where if the i-th character does not belong to a sensitive word, S(1, i, H) = 1, otherwise, S(1, i, H) = 0. Sentence embedding SentE = T*S.

[0093] The process of generating sensitive word embedding is to concatenate the matched sensitive words in the order of appearance in the text to be reviewed, with [sep] as the interval, and then convert them into vector representation to obtain sensitive word embedding.

[0094] The process of determining the sensitive word category embedding is to concatenate the sensitive words in the order of appearance in the text to be reviewed, with [sep] as the interval, and use one-hot encoding from 1 to 8 for the categories, and use [PAD] to fill in the insufficient length.

[0095] Step 4: Based on the preset sensitive extraction model, perform fusion feature extraction according to sentence embedding, sensitive word embedding and sensitive word category embedding.

[0096] In one embodiment, Figure 4 is a flowchart of another text review method provided according to the fourth embodiment of the present invention. Figure 4 As shown in the figure, the preset sensitive extraction model consists of a Bert layer, two encoding layers, a self-attention layer, and a fully connected layer (FC), where the fully connected layer can use the Sigmoid function as an activation function. The sentence embedding, sensitive word embedding, and sensitive word category embedding can be input into the preset sensitive extraction model respectively. After the sentence embedding is input, the sentence features can be extracted through a feature extraction network such as Bert. The sensitive word embedding is passed through two encoder layers for feature extraction, and the sensitive word category embedding is directly combined with the above two types of features through a self-attention layer to obtain a fusion feature.

[0097] Step 5: Determine whether the text to be reviewed is safe based on the fusion features.

[0098] In an embodiment, the fused features can be classified after passing through the fully connected layer to determine whether the text to be reviewed is safe. For example, the text content can be 0 for safety and 1 for unsafe.

[0099] Since language is constantly evolving and new words are constantly being generated, model-based automatic content review often finds it difficult to keep up with the development of language in a timely manner. The sensitive word library can flexibly and quickly follow the changes in development, but the rule-based method does not accurately grasp the semantics. In actual language expression, some unsafe content mainly comes from highly sensitive words, and some unsafe content is mainly due to the way the sentence is expressed. The original method is difficult to distinguish between these two situations, resulting in judgment failure. This method uses the design of sentence embedding to allow the network to exclude the interference of sensitive words and focus on learning the sentence characteristics of unsafe texts; through the design of sensitive word embedding, the network can focus on the insecurity of the word itself. Finally, combined with the embedding information of the sensitive word type, this method helps this method to quickly adapt to the expression of new words.

[0100] Embodiment 4

[0101] Figure 5 Schematic diagram of the structure of a text review device provided according to the third embodiment of the present invention. Figure 5 As shown, the device includes: a sensitive word determination module 51, an information embedding determination module 52, a security probability determination module 53 and a security situation determination module 54.

[0102] The sensitive word determination module 51 is used to obtain sensitive words in the text to be reviewed, and determine the location information and suspicious type of the sensitive words.

[0103] The information embedding determination module 52 is used to generate sentence embeddings of the text to be reviewed and sensitive word embeddings of sensitive words according to the position information, and determine suspicious type embeddings of suspicious types.

[0104] The safety probability determination module 53 is used to determine the content safety probability of the text to be reviewed according to sentence embedding, sensitive word embedding and suspicious type embedding based on a preset sensitivity extraction model.

[0105] The safety status determination module 54 is used to determine the safety status of the text to be reviewed according to the content safety probability.

[0106] In an embodiment of the present invention, a sensitive word determination module is used to obtain sensitive words in a text to be reviewed, and position information and suspicious types of sensitive words are determined. An information embedding determination module generates sentence embeddings of the text to be reviewed and sensitive word embeddings of sensitive words according to the position information, and determines suspicious type embeddings of suspicious types. A security probability determination module determines the content security probability of the text to be reviewed according to sentence embeddings, sensitive word embeddings and suspicious type embeddings based on a preset sensitive extraction model. A security situation determination module determines the security situation of the text to be reviewed according to the content security probability, and realizes the determination of the security situation of the text to be reviewed by combining sentence characteristics, the position of sensitive word embeddings and the category of sensitive words, thereby avoiding the problem of erroneous judgment caused by judging the security situation of the text to be reviewed only by judging the security situation of the text to be reviewed through a sensitive word library, and improving the accuracy of text security judgment.

[0107] In one embodiment, the sensitive word determination module 51 includes:

[0108] A type determination unit, used to match sensitive words in the text to be reviewed according to a preset sensitive word library, and determine the suspicious type of the sensitive words;

[0109] The position determination unit is used to identify the position of the sensitive word in the text to be reviewed and use the position as the position information of the sensitive word.

[0110] In one embodiment, the information embedding determination module 52 includes:

[0111] A word vector determination unit, used to convert each character in the text to be reviewed into a word vector;

[0112] A sentence embedding determination unit, used to determine the text type of the word vector in sequence according to the position information, determine the embedding mask of the word vector based on the text type, and determine the sentence embedding of the text to be reviewed according to the embedding mask; wherein the text type includes sensitive words and non-sensitive words;

[0113] A sensitive word embedding determination unit is used to concatenate sensitive words according to position information to obtain target sensitive words, and convert the target sensitive words into vector form to obtain sensitive word embedding;

[0114] The type embedding determination unit is used to determine the preset codes of each suspicious type, and to obtain the suspicious type embedding by concatenating the preset codes of the suspicious types corresponding to the sensitive words according to the position information.

[0115] In one embodiment, the sentence embedding determination unit is specifically used to:

[0116] When it is determined that the word vector belongs to a sensitive word, the word vector is marked with the first embedding mask;

[0117] When it is determined that the word vector belongs to a non-sensitive word, the word vector is marked with a second embedding mask, and the first embedding mask and the second embedding mask are used as embedding masks;

[0118] The word vectors corresponding to the second embedding mask are concatenated to obtain the sentence embedding of the text to be reviewed.

[0119] In one embodiment, the safety probability determination module 53 includes:

[0120] A model input unit, used to input sentence embedding, sensitive word embedding, and suspicious type embedding into a preset sensitive extraction model; wherein the preset sensitive extraction model is composed of a Bert layer, two encoding layers, a self-attention layer, and a fully connected layer;

[0121] A feature extraction unit, used to extract sentence features of sentence embedding based on the Bert layer, and to extract sensitive word features of sensitive word embedding based on the two encoding layers;

[0122] A fusion information determination unit, used to respectively determine the attention weights of sentence features, sensitive word features and suspicious types according to the sentence features, sensitive word features and suspicious types through the self-attention layer, and determine fusion feature information according to the attention weights and the sentence features, sensitive word features and suspicious types;

[0123] The probability determination unit is used to determine the content safety probability of the text to be reviewed based on the fully connected layer according to the fused feature information.

[0124] In one embodiment, the safety situation determination module 54 includes:

[0125] A first situation determination unit, configured to determine that the security situation of the to-be-audited text is safe when the content safety probability is greater than or equal to a preset content safety probability threshold;

[0126] The second situation determination unit is used to determine that the security situation of the text to be reviewed is unsafe when the content safety probability is less than a preset content safety probability threshold.

[0127] The text review device provided in the embodiment of the present invention can execute the text review method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Embodiment 5

[0129] Figure 6It is a structural schematic diagram of an electronic device for implementing a text audit method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the text review method.

[0133] In some embodiments, the text review method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the text review method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the text review method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0140] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and the computer program implements the text review method of any embodiment of the present invention when executed by a processor.

[0141] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text review method, characterized in that: include: Obtain sensitive words in the text to be reviewed, and determine the location information and suspicious type of the sensitive words; Generating sentence embedding of the text to be reviewed and sensitive word embedding of the sensitive word according to the position information, and determining the suspicious type embedding of the suspicious type; Determine the content safety probability of the to-be-audited text according to the sentence embedding, the sensitive word embedding, and the suspicious type embedding based on a preset sensitive extraction model; The safety status of the text to be reviewed is determined according to the content safety probability.

2. The method according to claim 1, characterized in that The step of obtaining sensitive words in the text to be reviewed and determining the location information and suspicious type of the sensitive words includes: Matching sensitive words in the text to be reviewed according to a preset sensitive word library to determine the suspicious type of the sensitive words; The position of the sensitive word in the text to be reviewed is identified, and the position is used as the position information of the sensitive word.

3. The method according to claim 1, characterized in that The step of generating sentence embeddings of the text to be reviewed and sensitive word embeddings of the sensitive words according to the position information, and determining suspicious type embeddings of the suspicious type includes: Convert each character in the text to be reviewed into a word vector; Determine the text type of the word vector in sequence according to the position information, determine the embedding mask of the word vector based on the text type, and determine the sentence embedding of the text to be reviewed according to the embedding mask; wherein the text type includes sensitive words and non-sensitive words; splicing the sensitive words according to the position information to obtain a target sensitive word, and converting the target sensitive word into a vector form to obtain a sensitive word embedding; Determine the preset codes of each of the suspicious types, and concatenate the preset codes of the suspicious types corresponding to the sensitive words according to the position information to obtain the suspicious type embedding.

4. The method according to claim 3, characterized in that The step of determining the text type of the word vector in sequence according to the position information, determining the embedding mask of the word vector based on the text type, and determining the sentence embedding of the text to be reviewed according to the embedding mask includes: When it is determined that the word vector belongs to a sensitive word, marking the word vector with a first embedding mask; When it is determined that the word vector belongs to a non-sensitive word, marking the word vector with a second embedding mask, and using the first embedding mask and the second embedding mask as embedding masks; The word vectors corresponding to the second embedding mask are concatenated to obtain the sentence embedding of the text to be reviewed.

5. The method according to claim 1, characterized in that The determining the content safety probability of the to-be-audited text according to the sentence embedding, the sensitive word embedding, and the suspicious type embedding based on the preset sensitive extraction model includes: Inputting the sentence embedding, the sensitive word embedding and the suspicious type embedding into a preset sensitive extraction model; wherein the preset sensitive extraction model is composed of a Bert layer, two encoding layers, a self-attention layer and a fully connected layer; Extracting sentence features of the sentence embedding based on the Bert layer, and extracting sensitive word features of the sensitive word embedding based on the two encoding layers; Determine the attention weights of the sentence features, the sensitive word features, and the suspicious type through the self-attention layer according to the sentence features, the sensitive word features, and the suspicious type embedding, and determine the fusion feature information according to the attention weights and the sentence features, the sensitive word features, and the suspicious type; Based on the fully connected layer, the content safety probability of the text to be reviewed is determined according to the fused feature information.

6. The method according to claim 1, characterized in that The determining the security status of the to-be-audited text according to the content security probability includes: When the content safety probability is greater than or equal to a preset content safety probability threshold, determining that the security status of the to-be-audited text is safe; When the content safety probability is less than the preset content safety probability threshold, it is determined that the safety status of the text to be reviewed is unsafe.

7. A text review device, characterized in that: include: A sensitive word determination module is used to obtain sensitive words in the text to be reviewed and determine the location information and suspicious type of the sensitive words; An information embedding determination module, configured to generate sentence embeddings of the text to be reviewed and sensitive word embeddings of the sensitive words according to the position information, and determine suspicious type embeddings of the suspicious type; A security probability determination module, configured to determine the content security probability of the to-be-audited text according to the sentence embedding, the sensitive word embedding, and the suspicious type embedding based on a preset sensitive extraction model; A security situation determination module is used to determine the security situation of the text to be reviewed based on the content security probability.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the text review method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text review method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the text review method according to any one of claims 1 to 6.

Images