Harmful text content detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a harmful text content detection method and device, electronic equipment and a storage medium, relates to the technical field of natural languages, and mainly aims to solve the problem of poor text detection accuracy caused by missing detection and false detection of harmful text content. The method comprises the steps of performing sensitive word matching on a to-be-detected text statement based on a first data set, wherein the first data set comprises a plurality of sensitive words; performing first detection on the text statement matched with the sensitive word based on a semantic detection model to obtain a first detection result, the semantic detection model being obtained by performing model training on a lightweight natural language processing model based on a second data set, statement samples in the second data set are statements including sensitive words and having harmless meanings; and performing second detection on the harmful statements in the first detection result based on a large language model to obtain a second detection result, and generating a harmful text content detection result based on verification information of the second detection result.

Description

Technical Field

[0001] This disclosure relates to the field of natural language technology, and in particular to a method and apparatus, electronic device and storage medium for detecting harmful text content. Background Technology

[0002] With the rapid development of natural language processing technology, the focus of text detection has shifted to the detection of harmful text content. Harmful text content detection typically includes methods for identifying sensitive words and methods for identifying semantically sensitive information.

[0003] Currently, commonly used sensitive word detection methods primarily rely on comparing and matching text content against a sensitive word lexicon. If a matching word is found, the content is deemed sensitive. However, if the text contains sensitive words along with negative or suggestive meanings, false positives can easily occur. For example, sentences like "We oppose violence" or "Effectively prevent online fraud" may contain sensitive words such as "violence" or "fraud," but the overall meaning of the sentence does not contain harmful content, significantly reducing the accuracy and effectiveness of harmful text content detection. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for detecting harmful text content. Its main purpose is to address the problem of poor text detection accuracy caused by missed or false detections of harmful text content. According to a first aspect of this disclosure, a method for detecting harmful text content is provided, comprising: Sensitive word matching is performed on the text statement to be detected based on the first dataset, which contains multiple sensitive words; The first detection is performed on the text sentences that match the sensitive words based on the semantic detection model to obtain the first detection result. The semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences that include sensitive words and have harmless meaning. The harmful statements in the first detection result are subjected to a second detection based on the large language model, and a harmful text content detection result is generated based on the verification information of the second detection result.

[0005] Furthermore, before obtaining the first detection result by performing a first detection on the text statements that match the sensitive words based on the semantic detection model, the method further includes: Load the second dataset, which contains updated sample statements tagged with sensitive word categories; A lightweight natural language processing model deployed locally is retrieved, and the model is trained based on the second dataset to obtain the semantic detection model after training is completed.

[0006] Furthermore, after performing a second detection on the harmful statements in the first detection result based on a large language model to obtain the second detection result, the method further includes: Output the second detection result; The system receives first verification information from the second detection result, the first verification information including the verification result of statements in the second detection result that contain sensitive words and have harmful semantics; If the first verification information includes the first updated sensitive word, then the first dataset is updated based on the first updated sensitive word; If the first verification information includes a first update statement that contains sensitive words and has harmless semantics, then the second dataset is updated based on the first update statement.

[0007] Furthermore, the method also includes: Output the first detection result containing harmless statements; The system receives second verification information from the first detection result, the second verification information including the result of verification of sensitive words; If the second verification information includes the second updated sensitive word, then the first dataset is updated based on the second updated sensitive word.

[0008] Furthermore, before obtaining the second detection result by performing a second detection on harmful statements in the first detection result based on a large language model, the method further includes: If the large language model is deployed in the cloud, a third dataset is loaded, which includes de-identified keywords for different business scenarios; The harmful statements are anonymized based on the third dataset, and then the anonymized harmful statements are input into the large language model.

[0009] Furthermore, before performing sensitive word matching on the text statement to be detected based on the first dataset, the method further includes: In response to text detection commands, extract the text content; The text content is segmented into sentences to obtain the text sentences to be detected; The method further includes: The updated first dataset is loaded, which includes sensitive words of different categories, to enable the first detection of the text statements with category identifiers.

[0010] According to a second aspect of this disclosure, a device for detecting harmful text content is provided, comprising: The matching module is used to perform sensitive word matching on the text statement to be detected based on the first dataset, which contains multiple sensitive words; The first detection module is used to perform a first detection on the text sentences that match the sensitive words based on the semantic detection model, and obtain a first detection result. The semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences that include sensitive words and have harmless meanings. The second detection module is used to perform a second detection on the harmful statements in the first detection result based on a large language model, obtain a second detection result, and generate a harmful text content detection result based on the verification information of the second detection result.

[0011] Furthermore, the device also includes: The loading module is used to load the second dataset, which contains updated sentence samples tagged with sensitive word categories; The training module is used to retrieve a lightweight natural language processing model deployed locally and train the model based on the second dataset to obtain the semantic detection model after training.

[0012] Furthermore, the device also includes: The output module is used to output the second detection result; The receiving module is configured to receive first verification information of the second detection result, wherein the first verification information includes the result of verifying the statements in the second detection result that contain sensitive words and have harmful semantics; The first update module is used to update the first dataset based on the first update sensitive word if the first verification information includes the first update sensitive word; The second update module is used to update the second dataset based on the first update statement if the first verification information includes a sensitive word and a first update statement with harmless semantics.

[0013] Furthermore, The output module is also used to output the first detection result containing harmless statements; The receiving module is further configured to receive second verification information of the first detection result, wherein the second verification information includes the result of verifying sensitive words; The first update module is further configured to update the first dataset based on the second update sensitive word if the second verification information includes the second update sensitive word.

[0014] Furthermore, The loading module is further configured to load a third dataset if the large language model is deployed in the cloud, the third dataset including desensitized keywords for different business scenarios; desensitize the harmful statements based on the third dataset, and input the desensitized harmful statements into the large language model.

[0015] Furthermore, the device also includes: an extraction module and a segmentation module. The extraction module is used to extract text content in response to a text detection command; The segmentation module is used to segment the text content into sentences to obtain the text sentences to be detected; The loading module is used to load the updated first dataset, which includes sensitive words of different categories, so as to perform a first detection on the text statements with category identifiers.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for detecting harmful text content described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method for detecting harmful text content described in the first aspect above.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method for detecting harmful text content as described in the first aspect above.

[0019] The method, apparatus, electronic device, and storage medium for detecting harmful text content disclosed herein perform sensitive word matching on a first dataset containing multiple sensitive words; perform a first detection on the text sentences that match the sensitive words based on a semantic detection model to obtain a first detection result, wherein the semantic detection model is obtained by training a lightweight natural language processing model on a second dataset, wherein the sentence samples in the second dataset are sentences that include sensitive words but have harmless meanings; perform a second detection on the harmful sentences in the first detection result based on a large language model to obtain a second detection result, and generate a harmful text content detection result based on the verification information of the second detection result. This achieves the purpose of using a combination of a lightweight model and a large language model for harmful text content detection. By filtering out part of the corpus in advance through the lightweight model and using high-risk sentences as the detection objects of the large language model, the waste of data resources and the occurrence of missed detections and false detections are greatly reduced, thereby improving the accuracy of harmful text content detection.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A schematic flowchart illustrating a method for detecting harmful text content provided in this embodiment of the present disclosure; Figure 2 A schematic flowchart illustrating another method for detecting harmful text content provided in this embodiment of the present disclosure; Figure 3 A schematic flowchart illustrating yet another method for detecting harmful text content provided in this disclosure embodiment; Figure 4 This is a schematic diagram of the overall process for detecting harmful text content provided in an embodiment of this disclosure; Figure 5 A schematic diagram of the structure of a device for detecting harmful text content provided in an embodiment of this disclosure; Figure 6 A schematic block diagram of an example electronic device 500 provided for embodiments of this disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] The following description, with reference to the accompanying drawings, outlines a method, apparatus, electronic device, and storage medium for detecting harmful text content according to embodiments of this disclosure.

[0024] Figure 1 This is a schematic flowchart illustrating a method for detecting harmful text content provided in an embodiment of this disclosure. Figure 1 As shown, the method includes the following steps: Step 101: Perform sensitive word matching on the text statement to be detected based on the first dataset.

[0025] In this embodiment, the text to be detected is a statement obtained by splitting the text. The text can be automatically input or manually imported. The applicable business scenarios for the text include, but are not limited to, table text or image text, and can also be question-and-answer text, knowledge text, etc. This embodiment does not make specific limitations. The current execution entity, as the execution end for detecting harmful text content, can be a terminal device or a server device, and performs text statement splitting after obtaining the text to be detected. At the same time, before performing sensitive word matching, the current execution end loads a first dataset. At this time, the first dataset contains multiple sensitive words. The sensitive words can be pre-configured for different business detection needs. For example, words based on fraud, violence, etc. can be used as the first dataset. They can also be constructed based on the variants of sensitive words, such as homonyms, abbreviations, and homophones of words in the fraud, violence, etc., so as to match them one by one with the words in the text statement.

[0026] It should be noted that, in this embodiment of the disclosure, when performing sensitive word matching based on the first dataset, it can be matched one by one with the sensitive words in the first database using a fuzzy matching method. Specifically, the core of fuzzy matching is to identify the similarity between data through a specific algorithm or function, so as to determine whether the text statement contains sensitive words based on the similarity. The similarity threshold for fuzzy matching can be configured according to the detection requirements, that is, the matching threshold of fuzzy matching can be flexibly adjusted and customized according to the characteristics and requirements of different application scenarios. This embodiment of the disclosure does not make specific limitations.

[0027] For example, if the text to be detected is "Xiaoming beat me up", and the first dataset contains "beat up" as a sensitive word related to violence, then fuzzy matching can be used to identify the presence of sensitive words in the text.

[0028] Step 102: Perform a first detection on the text statements that match the sensitive words based on the semantic detection model to obtain the first detection result.

[0029] In this embodiment of the disclosure, when the current execution terminal determines that a text statement contains sensitive words, it uses a semantic detection model to detect the text statement containing sensitive words. At this time, the semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences containing sensitive words but with harmless meanings, in order to detect whether the text statement contains sensitive but harmless content. The semantic detection model is a natural language processing module deployed locally on the current execution terminal, including but not limited to models such as Sentence-BERT and paraphrase-multilingual-MiniLM-L12-v2, for semantic analysis.

[0030] It should be noted that the natural language model used in this embodiment is a lightweight model with low deployment cost and fast inference speed. It can be supported by both CPU and small GPU and has high accuracy. Therefore, when performing the first detection in step 102, it is more cost-effective than the large language model used in the second detection and can support localized deployment.

[0031] In addition, the second dataset is pre-constructed based on a set of sentences containing sensitive words with harmless meanings. In this case, the word classification in the second dataset is the same as the word classification of sensitive words in the first dataset, and the sentences contain the same sensitive words as in the first dataset, but the semantic meaning is harmless. For example, sentences such as "We oppose violence" and "Effectively prevent online fraud" contain the sensitive words "violence" and "fraud", and the semantic meaning is harmless.

[0032] Step 103: Perform a second detection on the harmful statements in the first detection result based on the large language model to obtain a second detection result, and generate a harmful text content detection result based on the verification information of the second detection result.

[0033] In the embodiments of the present disclosure, after the current execution end performs the first detection, the first detection result may include the detection results of sensitive and harmful statements, as well as the detection results of sensitive and harmless statements. In order to further determine whether the harmfulness of the statements is accurate, the large language model is used to perform secondary detection on the harmful statements in the first detection result to obtain a more accurate secondary detection result. At this time, the second detection result may include statements with sensitive and harmless content, as well as statements with sensitive and harmful content. In order to avoid misjudgment caused by artificial intelligence detection, the second detection result is manually reviewed, that is, the harmfulness of the statements in the second detection result is verified to generate the final harmful text content detection result based on the verification information of the second detection result. Among them, the large language model (Large Language Model, LLM) may be an artificial intelligence system based on deep learning. By training and learning the grammar, semantics, and patterns of language on a large amount of text data, it can understand and generate natural language text, including but not limited to a certain Qianwen, a certain Seek, etc. It can be deployed locally on the current execution end or deployed in the cloud server. The embodiments of the present disclosure do not make specific limitations.

[0034] It should be noted that for the verification of the second detection result, it can be sent to the manual review end for manual review to obtain the verification result of the second detection result in the verification information (such as the harmful statement is judged correctly), that is, the reviewer uploads the verification information obtained from the review to the current execution end to generate the harmful text content detection result. For example, for the second detection result that "The little sister opposes beating others" is a sensitive and harmless statement, after manual review, the current execution end receives the verification information of the manual review that the statement is indeed sensitive and harmless (when conducting the review, the verification information can be generated in the form of labels, such as correct labels and wrong labels, etc.). Therefore, the generated harmful text content detection result is that "The little sister opposes beating others" is a sensitive and harmless statement. Among them, after obtaining the second detection result, the current execution end displays the second detection result to the reviewer in an output manner for manual review. After the reviewer completes the review, the current execution end receives the verification information obtained by uploading or inputting as the manual review result to generate the final harmful text content detection result.

[0035] Furthermore, when calling a large language model for semantic detection, the second dataset can be used as a constraint condition for detecting harmful statements in the first detection result, and both can serve as input to the large language model for detection. In this case, a prompt can be generated based on a prompt word template to trigger the large language model's detection step. The prompt word template can include five parts: "role," "instruction," "input," "constraint," and "output." Based on the classification of the text statements to be detected, statements from the second dataset are used as the constraint part of the prompt word (examples of harmless statements), generating a prompt suitable for triggering the large language model to analyze semantics. For example, if "role" is "text analyst," "instruction" is "please analyze," "input" is the harmful statements in the first detection result, "constraint" is the second dataset, and "output" is whether harmful content is contained, the resulting prompt would be: "Text analyst, please analyze. The second dataset serves as a constraint condition. Output whether the harmful statements in the first detection result contain harmful content." This allows the large language model to output the second detection result based on this prompt.

[0036] In another embodiment of this disclosure, such as Figure 2 As shown, in step 102, before performing a first detection on the text statements that match the sensitive words based on the semantic detection model and obtaining the first detection result, the method further includes: 201. Load the second dataset; 202. Retrieve the lightweight natural language processing model deployed locally, and train the lightweight natural language processing model based on the second dataset to obtain the semantic detection model after model training is completed.

[0037] To analyze sensitive but harmless statements and identify those containing sensitive words and harmful content, the current execution server pre-loads a second dataset. This second dataset contains updated statement samples categorized with sensitive words, allowing for the pre-training of a lightweight natural language processing (NLP) model based on this dataset. The lightweight NLP model is deployed locally and includes, but is not limited to, models such as Sentence-BERT and paraphrase-multilingual-MiniLM-L12-v2, for lightweight language processing purposes.

[0038] It should be noted that, in this embodiment of the disclosure, when using the second dataset as training samples for the sentence parsing model, sentences with the same classification labels as those in the first dataset can be selected for classification training. For example, for sentences classified as "violent," sample sentences in the second dataset containing the sensitive word "violent" classification label but with harmless semantic content can be retrieved for training. Furthermore, during model training, similarity metrics such as cosine similarity can be used as semantic detection thresholds. By setting a threshold similarity threshold, the model's learning accuracy on samples can be constrained. The threshold similarity threshold can be flexibly adjusted and customized according to different application scenarios and needs; this embodiment of the disclosure does not impose specific limitations.

[0039] In another embodiment of this disclosure, such as Figure 3 As shown, in step 103, after performing a second detection on the harmful statements in the first detection result based on the large language model and obtaining the second detection result, the method further includes: 301. Output the second detection result; 302. Receive the first verification information of the second detection result; 303a. If the first verification information includes the first updated sensitive word, then the first dataset is updated based on the first updated sensitive word; Step 303b, which runs parallel to step 303a, involves updating the second dataset based on a first update statement if the first verification information includes sensitive words and has harmless semantics.

[0040] To improve the accuracy of malicious statement parsing using a large language model by combining manual review with other methods, after obtaining the second detection result based on the large language model, the detection result containing harmful statements is output and sent to the manual review end. At this point, reviewers can directly review the results on the current execution end, or the detection results can be sent to their own terminals for review. After the review, the current execution end receives the first verification information obtained from reviewing the second detection result. This first verification information includes the results of manual review of statements containing sensitive words and harmful semantics in the second detection result. For example, tags can be used to indicate whether the review passed or failed, including manual review results for statements containing sensitive words or sensitive words with harmless semantics. This updates both the first and second datasets. This embodiment of the disclosure does not impose specific limitations.

[0041] It should be noted that during the manual review process, since the second detection result obtained from the manual review can include statements that are sensitive but have harmless semantics as well as statements that are sensitive but have harmful semantics, the reviewers can also review and verify sensitive words and add sensitive words that fail the review to the verification information as the result of the manual review. That is, the words in the statement are re-labeled and categorized to update the sensitive words.

[0042] In a specific implementation scenario, if the first verification information includes the first update sensitive word, the first dataset is updated based on the first update sensitive word. For example, if the first verification information is: "I oppose violence, but support corporal punishment is a harmless statement, which has not passed manual review and has marked "corporal punishment" as an update sensitive word", after the current execution terminal receives the above verification information, it determines that the first update sensitive word is "corporal punishment", and then updates "corporal punishment" to the first dataset to improve the accuracy of the first detection.

[0043] In a specific implementation scenario, if the first verification information includes a sensitive word and a first update statement with harmless semantics, then the second dataset is updated based on the first update statement. For example, if the first verification information is "The first verification information is: 'I oppose violence' is a harmless statement and has passed manual review," after receiving the above verification information, the current execution end updates the second dataset based on the first update statement so that the lightweight natural language processing model can be retrained based on the updated second dataset, thereby improving the recognition accuracy of the lightweight model.

[0044] In another embodiment of this disclosure, such as Figure 4 In the complete flowchart of the drawing analysis shown, the steps of this embodiment of the disclosure further include: Output the first detection result containing harmless statements; Receive the second verification information of the first detection result; If the second verification information includes the second updated sensitive word, then the first dataset is updated based on the second updated sensitive word.

[0045] To improve the accuracy of harmful statement parsing using the lightweight model by combining manual review with other methods, after the first detection based on the lightweight model, the first detection results obtained from the lightweight natural language model include not only statements containing sensitive words and harmful statements, but also statements containing sensitive words but harmless statements. For these harmless statements, this embodiment employs manual review for further judgment and verification. Therefore, during the manual review process, the current execution terminal outputs the first detection results to the reviewers, i.e., outputting the first detection results containing harmless statements to the manual review terminal for review and verification. After review, the reviewers obtain second verification information and upload or input it to the current execution terminal, i.e., the current execution terminal receives this second verification information. At this point, an extraction method can also be used to review a large number of harmless statements; this embodiment does not impose specific limitations on this method.

[0046] It should be noted that after the reviewer reviews the first detection result, the system receives the second verification information obtained from the reviewer's review of the first detection result. Furthermore, to improve the accuracy of sensitive word detection, the reviewer can also update the sensitive words in harmless statements. Therefore, the second verification information can include the results of sensitive word verification. In a specific embodiment, if the second verification information includes a second updated sensitive word, it indicates that the reviewer updated the sensitive words manually. The first dataset is then updated based on the second updated sensitive word to enable re-matching of sensitive words based on the updated first dataset, thereby improving the accuracy of sensitive word identification.

[0047] In another embodiment of this disclosure, before step 103, which involves performing a second detection on harmful statements in the first detection result based on a large language model to obtain the second detection result, the method further includes: If the large language model is deployed in the cloud, then load the third dataset; The harmful statements are anonymized based on the third dataset, and then the anonymized harmful statements are input into the large language model.

[0048] To achieve the goal of combining a lightweight natural language model with a large language model and improve the accuracy of detecting harmful content, the large language model can be deployed either on the current execution platform or in the cloud. In a specific implementation scenario, if the large language model is deployed in the cloud, a third dataset is loaded. This third dataset includes de-identified keywords for different business scenarios. The de-identified keywords represent privacy-sensitive words that need to be de-identified, including but not limited to names, place names, and contact information. Harmful statements are then de-identified based on the third dataset and input into the large language model.

[0049] It should be noted that when desensitizing harmful statements based on a third dataset, fuzzy matching can be used to match each word in the harmful statement with the desensitized keywords in the third dataset. If the harmful statement contains desensitized keywords in the third dataset, the corresponding desensitized keywords can be replaced with default word placeholders. For example, the matched desensitized keywords "Xiaoming", "Beijing", and "1234****" can be replaced with placeholders such as "[name]", "[city]", and "[contact information]". This disclosure does not impose specific limitations.

[0050] In another embodiment of this disclosure, before step 101, which involves performing sensitive word matching on the text statement to be detected based on the first dataset, the method further includes: In response to text detection commands, extract the text content; The text content is segmented into sentences to obtain the text sentences to be detected; To achieve automatic detection of text statements and improve the accuracy of harmful text content detection, the current execution terminal retrieves text content in response to a text detection command before performing sensitive word matching. The text content includes multiple text statements, each of which may include multiple words. Since the text can be in image or document form, it can be extracted using image recognition or natural language processing (NLP) methods. Furthermore, because the text content contains multiple statements, it can be segmented to obtain the statements to be detected one by one. Segmentation can be based on punctuation marks or pre-trained NLP algorithms; this disclosure does not impose specific limitations.

[0051] Correspondingly, such as Figure 4 In the flowchart shown, to make the sensitive words in the first dataset more accurate, thereby improving the accuracy of harmfulness identification, the steps of this embodiment of the disclosure further include: The first dataset, now updated, has been loaded.

[0052] The first dataset contains sensitive words of different categories. When performing fuzzy matching, different categories can be labeled to obtain different category identifiers. Then, natural language processing models are pre-trained for sentence samples containing sensitive words of different categories to obtain semantic detection models of multiple categories. This allows for the retrieval of semantic detection models corresponding to the category identifiers for the first detection of text sentences with category identifiers, making the detection of harmful content sentences more accurate.

[0053] It should be noted that the manual review in this embodiment can be triggered at any step, and the first dataset and the second dataset can be updated according to the review results to meet the flexible update requirements of the first dataset and the second dataset, thereby improving the accuracy of identifying harmful text content.

[0054] The method for detecting harmful text content disclosed herein involves: matching sensitive words in a first dataset containing multiple sensitive words; performing a first detection on the text sentences that match the sensitive words using a semantic detection model to obtain a first detection result; the semantic detection model being trained on a lightweight natural language processing model based on a second dataset containing sentences that include sensitive words but have harmless meanings; and performing a second detection on the harmful sentences in the first detection result using a large language model to obtain a second detection result. Based on the verification information of the second detection result, a harmful text content detection result is generated. This method achieves the goal of detecting harmful text content by combining a lightweight model with a large language model. By filtering out some data in advance using the lightweight model and using high-risk sentences as the detection targets of the large language model, the waste of data resources and the occurrence of missed and false detections are greatly reduced, thereby improving the accuracy of harmful text content detection.

[0055] Figure 5 This is a schematic diagram of the structure of a device for detecting harmful text content provided in an embodiment of this disclosure, as shown below. Figure 5 As shown, it includes: Matching module 41 is used to perform sensitive word matching on the text statement to be detected based on the first dataset, which contains multiple sensitive words; The first detection module 42 is used to perform a first detection on the text sentences that match the sensitive words based on the semantic detection model, and obtain a first detection result. The semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences that include sensitive words and have harmless meanings. The second detection module 43 is used to perform a second detection on the harmful statements in the first detection result based on a large language model, obtain a second detection result, and generate a harmful text content detection result based on the verification information of the second detection result.

[0056] Furthermore, the device also includes: The loading module is used to load the second dataset, which contains updated sentence samples tagged with sensitive word categories; The training module is used to retrieve a lightweight natural language processing model deployed locally and train the model based on the second dataset to obtain the semantic detection model after training.

[0057] Furthermore, the device also includes: The output module is used to output the second detection result; The receiving module is configured to receive first verification information of the second detection result, wherein the first verification information includes the result of verifying the statements in the second detection result that contain sensitive words and have harmful semantics; The first update module is used to update the first dataset based on the first update sensitive word if the first verification information includes the first update sensitive word; The second update module is used to update the second dataset based on the first update statement if the first verification information includes a sensitive word and a first update statement with harmless semantics.

[0058] Furthermore, The output module is also used to output the first detection result containing harmless statements; The receiving module is further configured to receive second verification information of the first detection result, wherein the second verification information includes the result of verifying sensitive words; The first update module is further configured to update the first dataset based on the second update sensitive word if the second verification information includes the second update sensitive word.

[0059] Furthermore, The loading module is further configured to load a third dataset if the large language model is deployed in the cloud, the third dataset including desensitized keywords for different business scenarios; desensitize the harmful statements based on the third dataset, and input the desensitized harmful statements into the large language model.

[0060] Furthermore, the device also includes: an extraction module and a segmentation module. The extraction module is used to extract text content in response to a text detection command; The segmentation module is used to segment the text content into sentences to obtain the text sentences to be detected; The loading module is used to load the updated first dataset, which includes sensitive words of different categories, so as to perform a first detection on the text statements with category identifiers.

[0061] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0062] This disclosure provides another harmful text content detection device, which performs sensitive word matching on the text sentences to be detected based on a first dataset containing multiple sensitive words; performs a first detection on the text sentences that match the sensitive words based on a semantic detection model to obtain a first detection result, wherein the semantic detection model is obtained by training a lightweight natural language processing model on a second dataset, wherein the sentence samples in the second dataset are sentences that include sensitive words and have harmless meanings; performs a second detection on the harmful sentences in the first detection result based on a large language model to obtain a second detection result, and generates a harmful text content detection result based on the verification information of the second detection result. This achieves the purpose of using a combination of a lightweight model and a large language model for harmful text content detection. By filtering out part of the data in advance through the lightweight model, and using high-risk sentences as the detection objects of the large language model, the waste of data resources and the situation of missed detections and false detections are greatly reduced, thereby improving the accuracy of harmful text content detection.

[0063] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0064] Figure 6 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0065] like Figure 6As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 502 or a computer program loaded from storage unit 508 into RAM (Random Access Memory) 503. The RAM 503 can also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0066] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0067] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as methods for detecting harmful text content. For example, in some embodiments, methods for detecting harmful text content may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the aforementioned method for detecting harmful text content by any other suitable means (e.g., by means of firmware).

[0068] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0069] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0070] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0071] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0072] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0073] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0074] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0075] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0076] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting harmful text content, characterized in that, include: Sensitive word matching is performed on the text statement to be detected based on the first dataset, which contains multiple sensitive words; The first detection is performed on the text sentences that match the sensitive words based on the semantic detection model to obtain the first detection result. The semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences that include sensitive words and have harmless meaning. The harmful statements in the first detection result are subjected to a second detection based on the large language model, and a harmful text content detection result is generated based on the verification information of the second detection result.

2. The method according to claim 1, characterized in that, Before obtaining the first detection result by performing a first detection on the text statement that matches the sensitive word based on the semantic detection model, the method further includes: Load the second dataset, which contains updated sample statements tagged with sensitive word categories; A lightweight natural language processing model deployed locally is retrieved, and the model is trained based on the second dataset to obtain the semantic detection model after training is completed.

3. The method according to claim 1, characterized in that, After performing a second detection on harmful statements in the first detection result based on a large language model to obtain the second detection result, the method further includes: Output the second detection result; The system receives first verification information from the second detection result, the first verification information including the verification result of statements in the second detection result that contain sensitive words and have harmful semantics; If the first verification information includes the first updated sensitive word, then the first dataset is updated based on the first updated sensitive word; If the first verification information includes a first update statement that contains sensitive words and has harmless semantics, then the second dataset is updated based on the first update statement.

4. The method according to claim 1, characterized in that, The method further includes: Output the first detection result containing harmless statements; The system receives second verification information from the first detection result, the second verification information including the result of verification of sensitive words; If the second verification information includes the second updated sensitive word, then the first dataset is updated based on the second updated sensitive word.

5. The method according to claim 1, characterized in that, Before performing a second detection on harmful statements in the first detection result based on a large language model to obtain the second detection result, the method further includes: If the large language model is deployed in the cloud, a third dataset is loaded, which includes de-identified keywords for different business scenarios; The harmful statements are anonymized based on the third dataset, and then the anonymized harmful statements are input into the large language model.

6. The method according to claim 1, characterized in that, Before performing sensitive word matching on the text statement to be detected based on the first dataset, the method further includes: In response to text detection commands, extract the text content; The text content is segmented into sentences to obtain the text sentences to be detected; The method further includes: The updated first dataset is loaded, which includes sensitive words of different categories, to enable the first detection of the text statements with category identifiers.

7. A device for detecting harmful text content, characterized in that, include: The matching module is used to perform sensitive word matching on the text statement to be detected based on the first dataset, which contains multiple sensitive words; The first detection module is used to perform a first detection on the text sentences that match the sensitive words based on the semantic detection model, and obtain a first detection result. The semantic detection model is obtained by training a lightweight natural language processing model based on a second dataset. The sentence samples in the second dataset are sentences that include sensitive words and have harmless meanings. The second detection module is used to perform a second detection on the harmful statements in the first detection result based on a large language model, obtain a second detection result, and generate a harmful text content detection result based on the verification information of the second detection result.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images