A content review method, system, device and medium

Abstract

The present application discloses a content review method, system, device and medium, the method comprising receiving a detected document input by a user, and performing multi-level sensitive word detection on the detected document; if the multi-level sensitive word detection of the detected document passes, inputting the detected document into a trained security intent classification model, and outputting the intent category of the detected document through the security intent classification model; when the intent category is a non-sensitive intent, obtaining the business data corresponding to the detected document, and performing multi-level sensitive word detection on the business data; if the multi-level sensitive word detection of the business data passes, it is determined that the content review has passed. The present application first performs sensitive word detection on the detected document through multi-level sensitive word detection, and then performs intent detection on the detected document through the security intent classification model, so that the accuracy of content review can be improved through incremental detection, thereby reducing the problems of misjudgment and missed judgment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a content review method, system, device and medium. Background Art

[0002] With the development of the mobile Internet industry, the content resources of the entire network have exploded, including information flow content, short videos, pictures and other content. Content security review is a key link to ensure the health and legal dissemination of information. At present, it is generally conducted through manual review or systematic machine review. However, during the content review process, the existing intelligent text automatic review system may mistakenly judge compliant content as illegal, or fail to detect the real illegal content in time, which may not only restrict the freedom of speech of normal users, but also allow harmful content to evade supervision.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by this application is to provide a content review method, system, device and medium in view of the deficiencies in the existing technology.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a content review method, wherein the content review method specifically includes:

[0006] Receiving a document to be detected input by a user, and performing multi-level sensitive word detection on the document to be detected, wherein the multi-level sensitive word detection is used to perform multiple sensitivity level detection on the document to be detected;

[0007] If the multi-level sensitive word detection of the detected document passes, the detected document is input into a trained security intent classification model, and the intent category of the detected document is output through the security intent classification model;

[0008] When the intent category is a non-sensitive intent, obtaining business data corresponding to the detected document, and performing multi-level sensitive word detection on the business data;

[0009] If the multi-level sensitive word detection of the business data passes, it is determined that the content review has passed.

[0010] The content audit method, wherein after receiving the document to be checked input by the user, the method further comprises:

[0011] Establishing an interactive session for the detected document;

[0012] The content audit query of the detected document is executed through the interactive session, and the query status of the content audit query is maintained, wherein the query status includes sensitive word result labels, priority labels and / or the number of times a new detected document is generated.

[0013] The content audit method, wherein the multi-level sensitive word detection on the detected document specifically includes:

[0014] Detect whether there is a waiting document in the preset waiting queue;

[0015] When there is no waiting document, performing multi-level sensitive word detection on the detected document;

[0016] When there is a waiting document, obtaining the priority of the detected document, and comparing the priority of the detected document with the priority of each waiting document;

[0017] If the priority of the detected document is greater than the priority of each waiting document, multi-level sensitive word detection is performed on the detected document. If the priority of at least one waiting document is higher than the priority of the detected document, the detected document is sent to the preset waiting queue.

[0018] The content audit method, wherein the detection process of the multi-level sensitive word detection specifically includes:

[0019] Segmenting a target document to obtain a word set corresponding to the target document, wherein the target document is a document for multi-level sensitive word detection, which is a detected document or business data;

[0020] Matching the word set with a preset multi-level sensitive word library, wherein the preset multi-level sensitive word library includes a strong sensitive word library and a weak sensitive word library;

[0021] If the word set matches a strongly sensitive word in the strongly sensitive word library, or matches at least a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has failed, and further determined that the content review of the detected document has failed;

[0022] If the word set does not match a strongly sensitive word in the strongly sensitive word library, and matches less than a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection passes.

[0023] The content review method, wherein the matching of the word set with a preset multi-level sensitive word library specifically includes:

[0024] Matching the word set with a highly sensitive word library in a preset multi-level sensitive word library;

[0025] When the word set does not match a strongly sensitive word in the strongly sensitive word library, the word set is matched with a weakly sensitive word library in the preset multi-level sensitive word library.

[0026] The content review method, wherein the training data set of the security intention classification model includes training sample data in multiple languages, so that the security intention classification model is applicable to the detected documents in multiple languages.

[0027] The content audit method, wherein when the intent category is a non-sensitive intent, obtaining the business data corresponding to the detected document specifically includes:

[0028] When the intent category is a non-sensitive intent, the detected document is input into a business module, and business data is output through the business module.

[0029] The content audit method, wherein the method further comprises:

[0030] When the intent category is a sensitive intent, it is determined that the content review fails.

[0031] The content review method, wherein if the multi-level sensitive word detection of the business data passes, determining that the content review passes specifically includes:

[0032] If the multi-level sensitive word detection of the business data passes, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data;

[0033] When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the detected document is spliced ​​with the business data to generate a new detected document, the new detected document is used as the detected document, and the step of performing multi-level sensitive word detection on the detected document is performed;

[0034] When no sensitive words of other sensitivity levels except the highest sensitivity level are detected, the content review is determined to have passed.

[0035] The content review method, wherein if the multi-level sensitive word detection of the business data passes, determining that the content review passes specifically includes:

[0036] If the multi-level sensitive word detection of the business data passes, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data;

[0037] When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the detected document is spliced ​​with the business data to generate a new detected document, the new detected document is used as the detected document, and the step of inputting the detected document into a trained security intent classification model is performed;

[0038] When no sensitive words of other sensitivity levels except the highest sensitivity level are detected, the content review is determined to have passed.

[0039] The content audit method, wherein, after splicing the detected document with the business data to generate a new detected document, the method further comprises:

[0040] The number of times a new document to be detected is generated;

[0041] If the number of times of generation is less than or equal to the preset number threshold, executing the step of taking the new detected document as the detected document;

[0042] If the generation times are greater than the preset times threshold, it is determined that the content review fails.

[0043] The content review method, wherein the content review method further comprises:

[0044] When the detected document is determined to have failed content review, a warning message is generated for the detected document, and the detected document is transmitted to the review background for review analysis to mine sensitive words in the detected document that are not included in the preset multi-level sensitive word library, and the mined sensitive words are fed back to the preset multi-level sensitive word library.

[0045] The second aspect of the present application provides a content review system, wherein the content review system specifically includes:

[0046] A first multi-level sensitive word detection module, used to receive a document to be detected input by a user, and perform multi-level sensitive word detection on the document to be detected;

[0047] An intention review module, used for inputting the detected document into a trained security intention classification model when the detected document passes the multi-level sensitive word detection, and outputting the intention category of the detected document through the security intention classification model;

[0048] a second multi-level sensitive word detection module, configured to, when the intent category is a non-sensitive intent, obtain business data corresponding to the detected document and perform multi-level sensitive word detection on the business data;

[0049] The determination module is used to determine that the content review has passed when the multi-level sensitive word detection of the business data passes.

[0050] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the content review methods described above.

[0051] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0052] The memory stores a computer-readable program executable by the processor;

[0053] When the processor executes the computer-readable program, the steps in any of the content review methods described above are implemented.

[0054] Beneficial effects:

[0055] 1. This application first reviews the document under inspection through multi-level sensitive word detection, and then uses a pre-trained security intent classification model to conduct content review on the document under inspection after the multi-level sensitive word detection is passed. This incremental detection can improve the accuracy of content review, thereby reducing the problems of misjudgment and missed judgment.

[0056] 2. This application adopts a security intent classification model trained based on training sample data in multiple languages, so that the security intent classification model can support documents being tested in multiple languages, thereby accurately identifying sensitive words and expressions in various languages, thereby improving the adaptability of the intent review method.

[0057] 3. For the detected documents that fail the review, this application will send the newly appearing sensitive words in the detected documents to the preset multi-level sensitive word library to dynamically update the preset multi-level sensitive word library, so that these newly appearing sensitive words can be identified and processed in a timely and effective manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A flowchart of a content review method provided in an embodiment of the present application.

[0060] Figure 2 A flowchart of an implementation method of the content review method provided in an embodiment of the present application.

[0061] Figure 3A flowchart of another implementation method of the content review method provided in an embodiment of the present application.

[0062] Figure 4 This is a functional block diagram of the content review device provided in an embodiment of the present application.

[0063] Figure 5 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application provide a content review method, system, device and medium. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0067] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0068] The content review method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms; the software can realize the application of the content review method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.

[0070] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0071] This embodiment provides a content review method, such as Figure 1 As shown, the method includes:

[0072] S10: receiving a document to be detected input by a user, and performing multi-level sensitive word detection on the document to be detected.

[0073] Specifically, the document content of the detected document is the content to be reviewed, and the language of the detected document is a language recognizable by the security intent classification model. For example, when the languages ​​recognizable by the security intent classification model include Chinese, English, and French, the language of the detected document can be Chinese. In addition, this embodiment does not limit the formation method of the detected document. For example, the detected document can be a text document, a voice document, a picture document, etc.

[0074] Further, after receiving the detected document input by the user, a session mechanism can be established for the detected document, and the session mechanism indicates that the user is interacting with the large model. That is, after receiving the detected document input by the user, the method further includes: establishing an interactive session for the detected document; executing a content audit query of the detected document through the interactive session, and maintaining the query state of the content audit query. Among them, the interactive session is used to indicate that the user is interacting with the large model, and the content audit query is executed in the form of an interactive session to realize the content review of the detected document. The content audit query is an interactive instruction executed by the interactive session, and the interactive instruction is used to interact with the large model to perform content audit on the detected document through the large model. The query state may include a sensitive word result label, and the sensitive word label reflects the detection result of the multi-level sensitive word detection of the detected document, wherein the sensitive word label may include the detection result in the multi-level sensitive word detection, for example, the sensitive word label may include a sensitive word memory label and the sensitivity of the sensitive word, the sensitive word memory label is used to indicate the hit sensitive word, and the sensitivity of the sensitive word is used to reflect the sensitivity level of the sensitive word.

[0075] It should be noted that, in actual applications, the query status may also include other status information, for example, it may include the priority label of the detected document, the generation count label of the new detected document, etc. The priority label is used to reflect the degree to which the detected document is given priority for detection, and the generation count label is used to indicate the number of times the content is reviewed for the new detected document generated by concatenating the detected document with its corresponding business data. It is used as an end condition for the content review to avoid an infinite loop in the detection process of the detected document, or a slow content review response due to too many execution times.

[0076] In one implementation, the performing multi-level sensitive word detection on the detected document specifically includes:

[0077] Detect whether there is a waiting document in the preset waiting queue;

[0078] When there is no waiting document, performing multi-level sensitive word detection on the detected document;

[0079] When there is a waiting document, obtaining the priority of the detected document, and comparing the priority of the detected document with the priority of each waiting document;

[0080] If the priority of the detected document is greater than the priority of each waiting document, multi-level sensitive word detection is performed on the detected document. If the priority of at least one waiting document is higher than the priority of the detected document, the detected document is sent to the preset waiting queue.

[0081] Specifically, the preset waiting queue is used to store the detected documents in the waiting state, that is, the detected documents stored in the preset waiting queue are documents waiting for content review, which are recorded as waiting documents here. To this end, after receiving the detected document, it will first check whether there is a waiting document in the preset waiting queue. If there is no waiting document, the detected document can be directly subjected to multi-level sensitive word detection. If there is a waiting document, it can be determined whether to perform multi-level sensitive word detection on the detected document or send the detected document to the preset waiting queue according to the priority of the detected document and the priority of the waiting document.

[0082] Further, the priority of the detected document can be determined according to the document category of the user grade and / or the detected document. For example, the priority corresponding to the user grade can be used as the priority of the detected document with a high priority in the priority corresponding to the document category of the detected document, or the user grade and the document category can be configured with a weight coefficient, and then both priorities are weighted according to the weight coefficient of the user grade and the weight coefficient of the document category, to obtain the priority of the detected document. Of course, in actual applications, after obtaining the priority of the detected document, the priority can also be displayed to the user, and the user's modification operation to this priority is received, based on this modification operation, the priority of the detected document is modified, and the modified priority is used as the priority of the detected document etc.

[0083] Furthermore, multi-level sensitive word detection can be performed through a large model, or a preset multi-level sensitive word library can be used as a "dictionary", and then the sensitive words in the detected document can be detected by querying the dictionary. In addition, in an embodiment of the present application, in addition to performing multi-level sensitive word detection on the detected document, multi-level sensitive word detection will also be performed on the business data output by the business module based on the detected document, and the multi-level sensitive word detection process of the two is the same. For this reason, the target document is used as the document to be subjected to multi-level sensitive detection, which can be the detected document and the business data output by the business module based on the detected document.

[0084] Exemplarily, the detection process of the multi-level sensitive word detection specifically includes:

[0085] Segmenting the target document to obtain a word set corresponding to the target document;

[0086] Matching the word set with a preset multi-level sensitive word library, wherein the preset multi-level sensitive word library includes a strong sensitive word library and a weak sensitive word library;

[0087] If the word set matches a strongly sensitive word in the strongly sensitive word library, or matches at least a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has failed, and further determined that the content review of the detected document has failed;

[0088] If the word set does not match a strongly sensitive word in the strongly sensitive word library, and matches less than a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection passes.

[0089] Specifically, word segmentation of a target document refers to dividing the target document into a number of words through a word segmentation tool (such as Transformer, etc.), and the target document is composed of a number of words. The target document may be a text document or a voice document, and when the target document is a voice document, the voice document may be converted into a text document first, and then the target document may be word segmented to obtain a word set corresponding to the target document.

[0090] The preset multi-level sensitive word library may be a sensitive word library formed through multiple filtering rules, data cleaning and manual screening, that is, the preset multi-level sensitive word library includes multiple sensitive words obtained by pre-screening. In addition, the preset multi-level sensitive word library may include sensitive word libraries of multiple sensitivity levels, each sensitive word library of each sensitivity level includes multiple sensitive words, and the sensitivity level of each sensitive word library of each sensitivity level is the same.

[0091] In a specific implementation, the preset multi-level sensitive word library includes a strong sensitive word library and a weak sensitive word library, the sensitivity level of the strong sensitive word library is the first sensitive level, and the sensitivity level of the weak sensitive word library is the second sensitive level, and the first sensitive level is higher than the second sensitive level. For example, the strong sensitive word library includes words involving multiple sensitive dimensions, such as environmental dimensions and ecological dimensions, and the weak sensitive word library includes dimensions involving sensitive words in the strong sensitive word library, but is not directly used as a sensitive word in the strong sensitive word library, or ambiguous words (for example, yellow, etc.).

[0092] In addition, when matching the word set with a preset multi-level sensitive word library, each word in the word set can be directly matched with a strong sensitive word library and a weak sensitive word library respectively to obtain the detection result of the multi-level sensitive word detection; or the word set can be matched with a strong sensitive word library and a weak sensitive word library in order of sensitivity level from high to low.

[0093] Exemplarily, matching the word set with a preset multi-level sensitive word library specifically includes:

[0094] Matching the word set with a highly sensitive word library in a preset multi-level sensitive word library;

[0095] When the word set does not match a strongly sensitive word in the strongly sensitive word library, the word set is matched with a weakly sensitive word library in the preset multi-level sensitive word library.

[0096] Specifically, matching the word set with the strong sensitive word library in the preset multi-level sensitive word library refers to matching each word in the word set with the preset multi-level sensitive word library, that is, searching for each word in the word set in the preset multi-level sensitive word library, and if any word in the word set is found in the preset multi-level sensitive word library, it is determined that the word set matches the strong sensitive word in the strong sensitive word library, and conversely, if any word in the word set is not found in the preset multi-level sensitive word library, it is determined that the word set does not match the strong sensitive word in the strong sensitive word library.

[0097] Further, when the word set matches a strong sensitive word in the strong sensitive word library, it is directly determined that the multi-level sensitive word detection has not passed, and then the content review of the document being detected has not passed. When the word set does not match a strong sensitive word in the strong sensitive word library, it means that the strong sensitive word detection of the word set has passed, and then the word set will be subjected to weakly sensitive word detection, that is, the word set will be matched with a weakly sensitive word library in a preset multi-level sensitive word library. When the word set matches at least a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has not passed, and then the content review of the document being detected has not passed. On the contrary, when the word set matches less than a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has passed.

[0098] It should be noted that when the preset multi-level sensitive word library includes sensitive word libraries of more than two sensitivity levels, it can be matched with the sensitive word libraries of each sensitivity level in order from high to low sensitivity level until the multi-level sensitive word detection is different, or, it is matched to the sensitive word library of the lowest sensitivity level. In addition, starting from the third sensitivity level, the detection can be carried out according to the detection process of the second sensitivity level, or starting from the second sensitivity level, a preset number is configured for each sensitivity level, and the preset number configured for each sensitivity level will increase as the sensitivity level decreases, etc.

[0099] S20: If the multi-level sensitive word detection of the detected document passes, the detected document is input into a trained security intent classification model, and the intent category of the detected document is output through the security intent classification model.

[0100] Specifically, the security intent classification model is a large model with small parameters that has been trained, such as the qwen7b base model, etc. The security intent classification model is used to classify the intent of the detected document to obtain the intent category of the detected document. The intent category may include sensitive intent and non-sensitive intent. Sensitive intent indicates that the detected document is sensitive content, and non-sensitive intent indicates that a business response can be made to the detected document to obtain the business data of the detected document.

[0101] The security intent classification model can be trained using a preset training data set, wherein the training data set includes a number of training data and word categories corresponding to the training data, the word categories include sensitive word categories, meaningless word categories, and business word categories, the sensitive word categories belong to sensitive intent, and the meaningless word categories and business word categories belong to non-sensitive intent. In addition, the training data set can include training sample data in multiple languages, so that the security intent classification model is applicable to documents to be detected in multiple languages.

[0102] For example, the training data corresponding to the sensitive word category may include sensitive words in multiple sensitive dimensions such as environmental problem dimension and legal problem dimension. For example, the environmental problem dimension may include sensitive words in dimensions such as air pollution dimension and ecological damage dimension; the legal problem dimension may include sensitive words in dimensions such as intellectual property dimension. The category of words without specific meaning may include words in dimensions such as "meaningless sentences", "unclear or unknown", and "others". The category of business words may include scenario words in business scenarios. For example, in the scenario of paper reading, the category of business words may include "information extraction-experimental part", "paper summary", "formula analysis", "in-depth discussion", "research methods", etc.

[0103] S30. When the intent category is a non-sensitive intent, obtain business data corresponding to the detected document, and perform multi-level sensitive word detection on the business data.

[0104] Specifically, the business data is obtained by performing business responses to the detected document. For example, in the paper reading module, the detected document requires query statements and paper content, and the business data is the text answer corresponding to the query statement output by the business module when the user interacts with the business module based on the detected document. In addition, the detection process of performing multi-level sensitive word detection on the business data is the same as the detection process of performing multi-level sensitive word detection on the detected document, and will not be repeated here.

[0105] Furthermore, the business data may be generated by the business module according to the user's interaction based on the detected document. To this end, when the intent category is a non-sensitive intent, obtaining the business data corresponding to the detected document specifically includes:

[0106] When the intent category is a non-sensitive intent, the detected document is input into a business module, and business data is output through the business module.

[0107] Specifically, when the intent category is non-sensitive intent, it means that a business response can be made to the detected document, so that the detected document can be input into the business module, and then the business module responds to the user's interaction based on the detected document and outputs business data. When outputting business data through the business module, the detected document can be directly used as the input content of the business module, or the interactive content between the detected document and the user based on the detected document can be used as the input content of the business module.

[0108] Exemplarily, when the user does not interact with the big model for the detected document, the detected document is directly output to the business module, and when the user interacts with the big model for the detected document, the detected document and the interactive content are input to the business module together. For example, when the detected document is the content of a paper, the user interacts with the big model based on the content of the paper, and the interactive content is asking questions about the content of the paper, then the input content input to the business module includes the content of the paper and the question content asking questions about the content of the paper.

[0109] In addition, when the intent category is sensitive intent, the detected document is directly determined to have failed content review, and no business response is made to the detected document, that is, the detected document is not input into the business module. The business module can be a business module for responding to the detected document, or a large model for responding to the detected document.

[0110] It should be noted that in actual applications, other methods can also be used to process detected documents with sensitive intent categories. For example, the detected document can be marked as an unsafe document and then fed back to the user, and based on the user feedback, it can be determined whether to make a business response to the detected document.

[0111] S40: If the multi-level sensitive word detection of the business data passes, it is determined that the content review has passed.

[0112] Specifically, if the multi-level sensitive word detection of business data is passed, it means that both the document being detected and the business data of the document being detected meet the multi-level sensitive detection requirements, so that it can be determined that the content review has passed. In addition, the business data can be presented on the display interface through the business module to feedback to the user, or the business data can be fed back to the user end, etc.

[0113] Furthermore, when the multi-level sensitive word detection of business data passes, sensitive words of other sensitivity levels except the highest sensitivity level may be detected, or sensitive words of other sensitivity levels except the highest sensitivity level may not be detected. When sensitive words of other sensitivity levels except the highest sensitivity level are not detected, it means that the number of sensitive words of other sensitivity levels included in the detected document and business data meets the passing requirements of the multi-level sensitive word detection. When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the number of sensitive words of other sensitivity levels included in the detected document and business data may not meet the passing requirements of the multi-level sensitive word detection. For this reason, when sensitive words of other sensitivity levels are detected, it is necessary to perform multi-level sensitive word detection on the new detected document formed by the business data and the detected document to further improve the accuracy of content review.

[0114] For example, Figure 2 As shown, if the multi-level sensitive word detection of the business data passes, it is determined that the content review has passed, which specifically includes:

[0115] If the multi-level sensitive word detection of the business data passes, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data;

[0116] When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the detected document is spliced ​​with the business data to generate a new detected document, the new detected document is used as the detected document, and the step of performing multi-level sensitive word detection on the detected document is performed;

[0117] When no sensitive words of other sensitivity levels except the highest sensitivity level are detected, the content review is determined to have passed.

[0118] Specifically, the new detected document can be obtained by splicing the detected document with the business data, that is, the new detected document includes the detected document and the business data. In this way, by performing multi-level sensitive word detection on the new detected document, it can be determined whether the document composed of the detected document and the business data meets the multi-level sensitive word detection.

[0119] In addition, in actual applications, after the multi-level sensitive word detection of business data passes, it means that both the detected document and the business data meet the multi-level sensitive word detection requirements, and the sensitive category of the detected document is non-sensitive intent. At this time, multi-level sensitive word detection may not be performed on the new document formed by the detected document and the business data, but only the sensitive category of the new document may be detected. To this end, in one implementation, Figure 3As shown, if the multi-level sensitive word detection of the business data is passed, it is determined that the content review is passed. Specifically, it includes: if the multi-level sensitive word detection of the business data is passed, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data; when sensitive words of other sensitivity levels except the highest sensitivity level are detected, splicing the detected document with the business data to generate a new detected document, using the new detected document as the detected document, and executing the step of inputting the detected document into a trained security intent classification model; when sensitive words of other sensitivity levels except the highest sensitivity level are not detected, it is determined that the content review is passed.

[0120] Furthermore, in order to avoid the problem of too long content review time or an infinite loop, before executing the step of using the new detected document as the detected document, it is possible to first check whether the number of times of generation reaches a preset number threshold, and then execute the step of using the new detected document as the detected document. Based on this, before executing the step of using the new detected document as the detected document, the method further includes:

[0121] The number of times a new document to be detected is generated;

[0122] If the number of times of generation is less than or equal to the preset number threshold, executing the step of taking the new detected document as the detected document;

[0123] If the generation times are greater than the preset times threshold, it is determined that the content review fails.

[0124] Of course, it should be noted that when performing multi-level sensitive word detection on business data, the multi-level sensitive word detection of the business data may also fail, and if it fails, it can be determined that the content review of the detected document has failed.

[0125] Furthermore, when the content of the detected document fails to pass the review, the detected document will be intercepted, and a warning message can also be generated for the detected document to inform the user that the detected document has failed the review. To this end, the content review method also includes:

[0126] When the detected document is determined to have failed content review, a warning message is generated for the detected document, and the detected document is transmitted to the review background for review analysis to mine sensitive words in the detected document that are not included in the preset multi-level sensitive word library, and the mined sensitive words are fed back to the preset multi-level sensitive word library.

[0127] Specifically, the failure of the content review of the detected document may be due to the fact that the detected document fails the multi-level sensitive word detection, or because the intent category is a sensitive intent, or because the business data fails the multi-level sensitive word detection, or because the number of generations reaches a preset threshold.

[0128] The warning information is used to inform the user that the document under review has not passed the review, and the warning information can be fed back to the user. At the same time, the document under review can also be transmitted to the review background, which will conduct a review and analysis on the document under review to mine the sensitive words in the document under review, and feed back the sensitive words that are not included in the preset multi-level sensitive word library to the preset multi-level sensitive word library to dynamically update the preset multi-level sensitive word library, so that the preset multi-level sensitive word library can timely and effectively identify and process new sensitive words.

[0129] In summary, this embodiment provides a content review method, which includes receiving a detected document input by a user, and performing multi-level sensitive word detection on the detected document; if the multi-level sensitive word detection of the detected document passes, inputting the detected document into a trained security intent classification model, and outputting the intent category of the detected document through the security intent classification model; when the intent category is a non-sensitive intent, obtaining the business data corresponding to the detected document, and performing multi-level sensitive word detection on the business data; if the multi-level sensitive word detection of the business data passes, it is determined that the content review has passed. The present application first performs sensitive word detection on the detected document through multi-level sensitive word detection, and then performs intent detection on the detected document through the security intent classification model, so that the accuracy of content review can be improved through incremental detection, thereby reducing the problems of misjudgment and missed judgment.

[0130] Based on the above content review method, this embodiment provides a content review system, such as Figure 4 As shown, the content review system specifically includes:

[0131] The first multi-level sensitive word detection module 100 is used to receive a document to be detected input by a user, and perform multi-level sensitive word detection on the document to be detected, wherein the multi-level sensitive word detection is used to perform multiple sensitivity level detection on the document to be detected;

[0132] The intention review module 200 is used to input the detected document into a trained security intention classification model when the detected document passes the multi-level sensitive word detection, and output the intention category of the detected document through the security intention classification model;

[0133] A second multi-level sensitive word detection module 300, for obtaining business data corresponding to the detected document and performing multi-level sensitive word detection on the business data when the intent category is a non-sensitive intent;

[0134] The determination module 400 is used to determine that the content review has passed when the multi-level sensitive word detection of the business data passes.

[0135] In one implementation, the content review system further includes a multi-level sensitive word library, which is used to store sensitive words of several sensitivity levels.

[0136] Based on the above content review method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the content review method described in the above embodiment.

[0137] Based on the above content review method, the present application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communications interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communications interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0138] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0139] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0140] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0141] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A content review method, characterized in that: The content review method specifically includes: Receiving a document to be detected input by a user, and performing multi-level sensitive word detection on the document to be detected, wherein the multi-level sensitive word detection is used to perform multiple sensitivity level detection on the document to be detected; If the multi-level sensitive word detection of the detected document passes, the detected document is input into a trained security intent classification model, and the intent category of the detected document is output through the security intent classification model; When the intent category is a non-sensitive intent, obtaining business data corresponding to the detected document, and performing multi-level sensitive word detection on the business data; If the business data passes the multi-level sensitive word detection, it is determined that the content review has passed; The detection process of the multi-level sensitive word detection specifically includes: Segmenting a target document to obtain a word set corresponding to the target document, wherein the target document is a document for multi-level sensitive word detection, which is a detected document or business data; Matching the word set with a preset multi-level sensitive word library, wherein the preset multi-level sensitive word library includes a strong sensitive word library and a weak sensitive word library; If the word set matches a strongly sensitive word in the strongly sensitive word library, or matches at least a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has failed, and further determined that the content review of the detected document has failed; If the word set does not match a strongly sensitive word in the strongly sensitive word library, and matches less than a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection passes.

2. The content review method according to claim 1, characterized in that: After receiving the detected document input by the user, the method further includes: Establishing an interactive session for the detected document; The content audit query of the detected document is executed through the interactive session, and the query status of the content audit query is maintained, wherein the query status includes sensitive word result labels, priority labels and / or the number of times a new detected document is generated.

3. The content review method according to claim 1 or 2, characterized in that: The performing multi-level sensitive word detection on the detected document specifically includes: Detect whether there is a waiting document in the preset waiting queue; When there is no waiting document, performing multi-level sensitive word detection on the detected document; When there is a waiting document, obtaining the priority of the detected document, and comparing the priority of the detected document with the priority of each waiting document; If the priority of the detected document is greater than the priority of each waiting document, multi-level sensitive word detection is performed on the detected document. If the priority of at least one waiting document is higher than the priority of the detected document, the detected document is sent to the preset waiting queue.

4. The content review method according to claim 1, characterized in that: The matching of the word set with the preset multi-level sensitive word library specifically includes: Matching the word set with a highly sensitive word library in a preset multi-level sensitive word library; When the word set does not match a strongly sensitive word in the strongly sensitive word library, the word set is matched with a weakly sensitive word library in the preset multi-level sensitive word library.

5. The content review method according to claim 1, characterized in that: The training data set of the security intention classification model includes training sample data in multiple languages, so that the security intention classification model is applicable to detected documents in multiple languages.

6. The content review method according to claim 1, characterized in that: When the intent category is a non-sensitive intent, obtaining the business data corresponding to the detected document specifically includes: When the intent category is a non-sensitive intent, the detected document is input into a business module, and business data is output through the business module.

7. The content review method according to claim 1 or 6, characterized in that: The method further comprises: When the intent category is a sensitive intent, it is determined that the content review fails.

8. The content review method according to claim 1, characterized in that: If the multi-level sensitive word detection of the business data passes, determining that the content review passes specifically includes: If the multi-level sensitive word detection of the business data passes, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data; When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the detected document is spliced ​​with the business data to generate a new detected document, the new detected document is used as the detected document, and the step of performing multi-level sensitive word detection on the detected document is performed; When no sensitive words of other sensitivity levels except the highest sensitivity level are detected, the content review is determined to have passed.

9. The content review method according to claim 1, characterized in that: If the multi-level sensitive word detection of the business data passes, determining that the content review passes specifically includes: If the multi-level sensitive word detection of the business data passes, detecting whether sensitive words of other sensitivity levels except the highest sensitivity level are detected in the multi-level sensitive word detection of the business data; When sensitive words of other sensitivity levels except the highest sensitivity level are detected, the detected document is spliced ​​with the business data to generate a new detected document, the new detected document is used as the detected document, and the step of inputting the detected document into a trained security intent classification model is performed; When no sensitive words of other sensitivity levels except the highest sensitivity level are detected, the content review is determined to have passed.

10. The content review method according to claim 8 or 9, characterized in that: After the detected document is spliced ​​with the business data to generate a new detected document, the method further includes: The number of times a new document to be detected is generated; If the number of times of generation is less than or equal to the preset number threshold, executing the step of taking the new detected document as the detected document; If the generation times are greater than the preset times threshold, it is determined that the content review fails.

11. The content review method according to claim 1, characterized in that: The content review method further includes: When the detected document is determined to have failed content review, a warning message is generated for the detected document, and the detected document is transmitted to the review background for review analysis to mine sensitive words in the detected document that are not included in the preset multi-level sensitive word library, and the mined sensitive words are fed back to the preset multi-level sensitive word library.

12. A content review system, characterized in that: The content review system specifically includes: A first multi-level sensitive word detection module, used to receive a document to be detected input by a user, and perform multi-level sensitive word detection on the document to be detected; An intention review module, used for inputting the detected document into a trained security intention classification model when the detected document passes the multi-level sensitive word detection, and outputting the intention category of the detected document through the security intention classification model; a second multi-level sensitive word detection module, configured to, when the intent category is a non-sensitive intent, obtain business data corresponding to the detected document and perform multi-level sensitive word detection on the business data; A determination module, configured to determine that the content review has passed when the multi-level sensitive word detection of the business data has passed; The detection process of the multi-level sensitive word detection specifically includes: Segmenting a target document to obtain a word set corresponding to the target document, wherein the target document is a document for multi-level sensitive word detection, which is a detected document or business data; Matching the word set with a preset multi-level sensitive word library, wherein the preset multi-level sensitive word library includes a strong sensitive word library and a weak sensitive word library; If the word set matches a strongly sensitive word in the strongly sensitive word library, or matches at least a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection has failed, and further determined that the content review of the detected document has failed; If the word set does not match a strongly sensitive word in the strongly sensitive word library, and matches less than a preset number of weakly sensitive words in the weakly sensitive word library, it is determined that the multi-level sensitive word detection passes.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the content review method as described in any one of claims 1-11.

14. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the content review method as described in any one of claims 1-11.

Images