Method for detecting security of AIGC text and related equipment

Abstract

The application discloses a method for detecting the safety of AIGC text and related equipment, which is used to improve the accuracy of safety detection. The method comprises the following steps: determining whether a safety detection score can be obtained according to a first score obtained by detecting the AIGC text to be detected by using a preset keyword data set and a second score obtained by performing text segment matching on the AIGC text to be detected by using a preset text segment data set; if not, determining the safety detection score based on the first score and the second score; if yes, determining the safety detection score of the AIGC text to be detected according to a third score obtained by inputting the AIGC text to be detected into a pre-trained abnormal text detection model, a sentiment category and a fourth score obtained by inputting the AIGC text to be detected into a pre-trained sentiment classification detection model; and comparing the safety detection score with a preset threshold to obtain a safety detection result.

Description

Technical Field

[0001] This application relates to the field of network information security, specifically to a security detection method and related equipment for generative artificial intelligence (AIGC) text. Background Technology

[0002] As AI (Artificial Intelligence) big data technology empowers various industries, it is profoundly changing them. However, with the continuous enhancement of AI big data capabilities, ensuring that it truly creates value for human society rather than posing a threat has become a crucial issue in current AI development. As an important dimension of AI for good and sustainable development, the safe, trustworthy, and responsible development of content generated by industry big data models increasingly relies on effective governance and security measures.

[0003] Large language model training data largely comes from publicly available online data, making it difficult to construct the model's values. This can lead to potentially problematic values ​​in the output content. Furthermore, the presence of direct and indirect user prompts can result in unpredictable and inappropriate content. Therefore, content security review of AIGC (Artificial Intelligence Generated Content) text has always been a key focus for online platforms and regulatory agencies. The aim is to filter out illegal, harmful, or inappropriate content to reduce the negative impact of its dissemination.

[0004] Currently, content moderation of AIGC texts mainly relies on sensitive word databases for matching and identification. However, the effectiveness of this method depends heavily on the richness of the database and is prone to false positives due to semantic ambiguity, resulting in low accuracy in AIGC text detection. Summary of the Invention

[0005] This application provides a method and related equipment for security detection of generative artificial intelligence (AIGC) text, which can improve the accuracy of security detection of AIGC text.

[0006] Firstly, this application provides a method for security detection of generative artificial intelligence (AIGC) text, the method comprising:

[0007] The AIGC text to be detected is matched with keywords using a pre-set keyword dataset to obtain a first score; and the AIGC text to be detected is matched with text fragments using a pre-set text fragment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text fragment dataset includes text fragments with negative and / or neutral information.

[0008] Based on the first score and the second score, determine whether a security detection score for the AIGC text to be detected can be obtained;

[0009] If not, then based on the first score and the second score, determine the security detection score of the AIGC text to be detected;

[0010] If so, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC to be detected; based on the third score, the sentiment category of the AIGC to be detected, and the fourth score, a security detection score for the AIGC text to be detected is obtained; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category;

[0011] The security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0012] In this embodiment, a first score is obtained by keyword matching of the AIGC text to be detected; a second score is obtained by text segment matching of the AIGC text to be detected using a pre-set text segment dataset. Then, based on the first and second scores, it is determined whether a security detection score for the AIGC text to be detected can be obtained. If not, a security detection score is determined based on the first and second scores; if yes, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score for the AIGC text to be detected. Based on the third score, the sentiment category, and the fourth score, a security detection score for the AIGC text to be detected is obtained. Finally, the security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected. Therefore, this embodiment improves the accuracy of the security detection result by determining the security detection result of the AIGC text to be detected through multi-dimensional detection.

[0013] In one possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity to text fragments in the text fragment dataset that contain negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that contain neutral information.

[0014] The step of determining whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score includes:

[0015] If the first score is a preset value, and the first value in the second score is the preset value, then it is determined that a security detection score for the AIGC text to be detected cannot be obtained; or,

[0016] If there is a score among the first values ​​of the first score and the second score that is not the preset value, then it is determined that the security detection score of the AIGC text to be detected is obtained.

[0017] In one embodiment, the second score includes a first value and a second value; the first value represents the similarity to text fragments containing negative information in the text fragment dataset, and the second value represents the similarity to text fragments containing neutral information in the text fragment dataset.

[0018] The process of determining the security detection score of the AIGC text to be detected based on the first score and the second score includes:

[0019] If the first score is greater than the first value, then the first score is determined as the security detection score; or,

[0020] If the first score is not greater than the first value, then the first value is determined as the security detection score.

[0021] In one embodiment, the emotion categories include positive emotion categories, neutral emotion categories, and negative emotion categories;

[0022] The process of obtaining a security detection score for the AIGC text based on the third score, the sentiment category of the AIGC to be detected, and the fourth score includes:

[0023] If the sentiment category of the AIGC to be detected is the negative sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the third score and the fourth score;

[0024] If the sentiment category of the AIGC to be detected is the positive sentiment category, then the preset value is determined as the security detection score of the AIGC text to be detected;

[0025] If the sentiment category of the AIGC to be detected is the neutral sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score.

[0026] In one embodiment, the second score includes a first value and a second value, wherein the first value is used to represent the similarity to text fragments in the text fragment dataset that include negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that include neutral information;

[0027] The process of obtaining the security detection score of the AIGC text to be detected based on the second score, the third score, and the fourth score includes:

[0028] Using the first weights of the second value, the third score, and the fourth score, a weighted sum is performed on the second value, the third score, and the fourth score to obtain the security detection score of the AIGC text to be detected.

[0029] The process of obtaining the security detection score of the AIGC text to be detected based on the third score and the fourth score includes:

[0030] Using the second weights of the third and fourth scores respectively, the third and fourth scores are weighted and summed to obtain the security detection score of the AIGC text to be detected.

[0031] In one embodiment, the pre-trained abnormal text detection model is obtained in the following manner:

[0032] Each keyword in the keyword dataset is input into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is text including the keyword; and,

[0033] Each keyword in the keyword dataset is used to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus that includes at least one of the keywords;

[0034] The first target text corpus and the second target text corpus are identified as abnormal text corpus, and the third target text corpus is identified as normal text corpus; wherein, the third target text corpus is the other text corpus besides the second target text corpus among the preset text corpus;

[0035] Training samples are obtained based on the abnormal text corpus and the normal text corpus;

[0036] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0037] Secondly, this application provides a security detection device for generative artificial intelligence (AIGC) text, the device comprising:

[0038] The matching module is used to perform keyword matching on the AIGC text to be detected using a pre-set keyword dataset to obtain a first score; and to perform text segment matching on the AIGC text to be detected using a pre-set text segment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text segment dataset includes text segments with negative and / or neutral information.

[0039] The judgment module is used to determine whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score.

[0040] The first security detection score determination module is used to determine the security detection score of the AIGC text to be detected based on the first score and the second score if no.

[0041] The second security detection score determination module is used to, if so, input the AIGC text to be detected into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and input the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC to be detected; and obtain the security detection score of the AIGC text to be detected based on the third score, the sentiment category of the AIGC to be detected, and the fourth score; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category;

[0042] The security detection result determination module is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0043] In one possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity to text fragments in the text fragment dataset that contain negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that contain neutral information.

[0044] The judgment module is specifically used for:

[0045] If the first score is a preset value, and the first value in the second score is the preset value, then it is determined that a security detection score for the AIGC text to be detected cannot be obtained; or,

[0046] If there is a score among the first values ​​of the first score and the second score that is not the preset value, then it is determined that the security detection score of the AIGC text to be detected is obtained.

[0047] In one possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity to text fragments in the text fragment dataset that contain negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that contain neutral information.

[0048] The first security detection score determination module is specifically used for:

[0049] If the first score is greater than the first value, then the first score is determined as the security detection score; or,

[0050] If the first score is not greater than the first value, then the first value is determined as the security detection score.

[0051] In one possible embodiment, the emotion categories include positive emotion categories, neutral emotion categories, and negative emotion categories;

[0052] The second security detection score determination module is specifically used for:

[0053] If the sentiment category of the AIGC to be detected is the negative sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the third score and the fourth score;

[0054] If the sentiment category of the AIGC to be detected is the positive sentiment category, then the preset value is determined as the security detection score of the AIGC text to be detected;

[0055] If the sentiment category of the AIGC to be detected is the neutral sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score.

[0056] In one possible embodiment, the second score includes a first value and a second value, wherein the first value is used to represent the similarity to text fragments in the text fragment dataset that include negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that include neutral information.

[0057] The second security detection score determination module performs the step of obtaining the security detection score of the AIGC text to be detected based on the second score, the third score, and the fourth score, specifically for:

[0058] Using the first weights of the second value, the third score, and the fourth score, a weighted sum is performed on the second value, the third score, and the fourth score to obtain the security detection score of the AIGC text to be detected.

[0059] The second security detection score determination module performs the step of obtaining the security detection score of the AIGC text to be detected based on the third score and the fourth score, specifically for:

[0060] Using the second weights of the third and fourth scores respectively, the third and fourth scores are weighted and summed to obtain the security detection score of the AIGC text to be detected.

[0061] In one possible embodiment, the device further includes:

[0062] The model training module is used to obtain the pre-trained abnormal text detection model in the following ways:

[0063] Each keyword in the keyword dataset is input into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is text including the keyword; and,

[0064] Each keyword in the keyword dataset is used to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus that includes at least one of the keywords;

[0065] The first target text corpus and the second target text corpus are identified as abnormal text corpus, and the third target text corpus is identified as normal text corpus; wherein, the third target text corpus is the other text corpus besides the second target text corpus among the preset text corpus;

[0066] Training samples are obtained based on the abnormal text corpus and the normal text corpus;

[0067] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0068] Thirdly, this application provides an electronic device, comprising:

[0069] Memory, used to store program instructions;

[0070] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0071] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0072] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0073] Figure 1 A scenario diagram provided for an embodiment of this application;

[0074] Figure 2 One of the flowcharts for a security detection method for generative artificial intelligence AIGC text provided in this application embodiment;

[0075] Figure 3 This is a schematic diagram illustrating the process of training an abnormal text detection model according to an embodiment of this application;

[0076] Figure 4 A second schematic flowchart illustrating the security detection method for generative artificial intelligence AIGC text provided in this application embodiment;

[0077] Figure 5 A schematic diagram of a system for security detection of generative artificial intelligence AIGC text provided in this application embodiment;

[0078] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0080] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0081] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0082] Before introducing the security detection method for generative artificial intelligence AIGC text provided in the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0083] In existing technologies, content moderation of AIGC texts mainly relies on sensitive word databases for matching and identification. However, the effectiveness of this method depends heavily on the richness of the database and is prone to false positives due to semantic ambiguity, resulting in low accuracy in AIGC text detection.

[0084] Therefore, this application provides a method for security detection of generative artificial intelligence (AIGC) text. First, a first score is obtained by keyword matching of the AIGC text to be detected. Then, a second score is obtained by text segment matching using a pre-set text segment dataset. Based on the first and second scores, it is determined whether a security detection score for the AIGC text can be obtained. If not, a security detection score is determined based on the first and second scores. If yes, the AIGC text is input into a pre-trained abnormal text detection model for abnormal text detection, obtaining a third score. The AIGC text is then input into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score. Based on the third score, the sentiment category, and the fourth score, a security detection score for the AIGC text is obtained. Finally, the security detection score of the AIGC text is compared with a preset threshold to obtain the security detection result. Thus, this application embodiment uses multi-dimensional detection to determine the security detection result of the AIGC text, improving the accuracy of the security detection result. The solution of this disclosure will now be described in detail with reference to the accompanying drawings.

[0085] like Figure 1 The diagram shows an application scenario of a generative artificial intelligence (AIGC) text security detection method, which includes a server 110 and a terminal device 120.

[0086] In one possible application scenario, server 110 performs keyword matching on the AIGC text to be detected using a pre-set keyword dataset to obtain a first score; and performs text segment matching on the AIGC text to be detected using a pre-set text segment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text segment dataset includes text segments with negative and / or neutral information; then server 110 determines whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score; if not, server 110 determines a security detection score for the AIGC text to be detected based on the first score and the second score; if yes, server 110 detects the AIGC text to be detected... The AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection, and a third score is obtained. The AIGC text to be detected is then input into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score. Based on the third score, the sentiment category, and the fourth score, a security detection score is obtained for the AIGC text to be detected. The third score represents the confidence level that the AIGC text to be detected is abnormal text, and the fourth score represents the confidence level that the AIGC text to be detected belongs to the specified sentiment category. Finally, the server 110 compares the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected. The security detection result of the AIGC text to be detected is then sent to the terminal device 120 for display.

[0087] in, Figure 1 The server 110 and the terminal device 120 can exchange information through a communication network. The communication network can be either wireless or wired.

[0088] For example, server 110 can access the network via cellular mobile communication technology and communicate with terminal device 120, wherein the cellular mobile communication technology includes, for example, 5th Generation Mobile Networks (5G) technology.

[0089] Optionally, server 110 can access the network and communicate with terminal device 120 via short-range wireless communication, wherein the short-range wireless communication method includes, for example, Wireless Fidelity (Wi-Fi) technology.

[0090] Furthermore, the server 110 in this embodiment can be a physical server or a cloud server. The description in this application details only one server 110 and one slave terminal device 120; however, those skilled in the art should understand that the illustrated server 110 and slave terminal device 120 are intended to illustrate the operation of the server 110 and slave terminal device 120 involved in the technical solution of this application, and do not imply any limitation on the number, type, or location of the server 110 and slave terminal device 120. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0091] It should be noted that the security detection method for generative AIGC text proposed in this application is not only applicable to... Figure 1 The application scenarios shown also apply to any security detection device with generative artificial intelligence (AIGC) text.

[0092] The following describes an exemplary embodiment of the security detection method for generative artificial intelligence AIGC text, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way.

[0093] like Figure 2 The diagram shown illustrates the flowchart of a generative artificial intelligence (AIGC) text security detection method, which includes the following steps:

[0094] Step 201: Perform keyword matching on the AIGC text to be detected using a pre-set keyword dataset to obtain a first score; and perform text segment matching on the AIGC text to be detected using a pre-set text segment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text segment dataset includes text segments with negative and / or neutral information.

[0095] The keywords in the keyword dataset of this application embodiment can be collected from the Internet and news corpora. Furthermore, each keyword in this application embodiment has a tag, which indicates whether the keyword contains negative or neutral information.

[0096] Furthermore, in this embodiment, keyword matching of the AIGC text to be detected can be performed using the AC automaton matching algorithm or a traversal method. Since the AC automaton matching algorithm or the traversal method are existing technologies and not innovative aspects of this invention, they will not be elaborated upon here.

[0097] In one embodiment, the first score is obtained in the following manner:

[0098] The AIGC text to be detected is matched using a pre-set dataset of text fragments. If a keyword labeled as negative information is detected in the AIGC text to be detected, the first score is determined as a first specified score. If a keyword labeled as neutral information is detected in the AIGC text to be detected, the first score is determined as a second specified score. If no keyword is detected in the AIGC text to be detected, the first score is determined as a third specified score. The second specified score is greater than the first specified score and less than the first specified score.

[0099] It should be noted that: in this embodiment of the application, the first specified score is 1, the second specified score is 0.5, and the third specified score is 0. However, this embodiment of the application does not limit the first specified score, the second specified score, and the third specified score. The specific values ​​of the first specified score, the second specified score, and the third specified score in this embodiment of the application can be set according to the specific actual situation.

[0100] In this embodiment of the application, the method of using a pre-set text fragment dataset to perform text fragment matching on the AIGC text to be detected to obtain a second score can be specifically implemented as follows:

[0101] Each text segment in the pre-set text segment dataset is vectorized to obtain a first vector corresponding to each text segment; the AIGC text to be detected is vectorized to obtain a second vector; for any text segment, based on the first vector of the any text segment and the second vector of the AIGC text to be detected, the cosine value between the first vector and the second vector is obtained, and the cosine value is determined as the similarity between the first vector and the second vector, and the similarity with the largest value among all similarities is determined as the second score.

[0102] It should be noted that the methods for vectorizing each text segment and the AIGC text to be detected in this embodiment can be: TF-IDF (Term Frequency-Inverse Document Frequency) algorithm, Word2Vec (word to vector) algorithm, and BOW (Bag of Words) algorithm, etc. The specific method can be set according to the actual situation, and this embodiment does not limit it.

[0103] Step 202: Based on the first score and the second score, determine whether a security detection score for the AIGC text to be detected can be obtained. If not, proceed to step 203; if yes, proceed to step 204.

[0104] The second score in this embodiment includes a first value and a second value; the first value is used to represent the similarity with text fragments containing negative information in the text fragment dataset, and the second value is used to represent the similarity with text fragments containing neutral information in the text fragment dataset.

[0105] In one embodiment, step 202 can be specifically implemented as follows: if the first score is a preset value and the first value in the second score is the preset value, then it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or if there is a score in the first score and the first value in the second score that is not the preset value, then it is determined that the security detection score of the AIGC text to be detected can be obtained.

[0106] It should be noted that the preset value in this application embodiment is 0, but this does not limit the preset value in this application embodiment. The preset value in this application embodiment can be set according to the specific actual situation.

[0107] Step 203: Based on the first score and the second score, determine the security detection score of the AIGC text to be detected;

[0108] In one embodiment, step 203 can be specifically implemented as follows: if the first score is greater than the first value, then the first score is determined as the security detection score; or, if the first score is not greater than the first value, then the first value is determined as the security detection score. The security detection score of the AIGC text to be detected can be obtained through formula (1):

[0109] S = max(s1, s2) 21 )……(1);

[0110] Where S is the security detection score of the AIGC text to be detected, s1 is the first score, and s 21 It is the first value in the second fraction.

[0111] Step 204: Input the AIGC text to be detected into a pre-trained abnormal text detection model to perform abnormal text detection and obtain a third score; and input the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC text to be detected; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category;

[0112] Below, we will first introduce the training method of the abnormal text detection model in the embodiments of this application, such as... Figure 3 The diagram shown illustrates the process of training an abnormal text detection model, which may include the following steps:

[0113] Step 301: Input each keyword in the keyword dataset into the pre-trained text generation model to obtain the first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is the text including the keyword;

[0114] In this embodiment, the tags corresponding to the first target text corpus are the same as the tags of the keywords corresponding to the first target text corpus.

[0115] For example, if the keyword is "safety", then the corresponding first target text corpus would be: xxxxxxxxxsafexxxxxxxxxxxxxxx.

[0116] Step 302: Use each keyword in the keyword dataset to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus that includes at least one of the keywords;

[0117] In this embodiment, the labels of the second target text corpus are the same as the labels of the keywords included in the second target text corpus.

[0118] In this embodiment, the preset text corpora are text corpora obtained by randomly inputting words into the text generation model. The detection method in this embodiment can be a traversal method, that is, searching for the existence of each keyword in each text corpus, and determining the text corpus containing at least one keyword as the second text corpus.

[0119] Step 303: Determine the first target text corpus and the second target text corpus as abnormal text corpus, and determine the third target text corpus as normal text corpus; wherein, the third target text corpus is the other text corpus among the preset text corpus excluding the second target text corpus;

[0120] Step 304: Obtain training samples based on the abnormal text corpus and the normal text corpus;

[0121] In this embodiment of the application, the scores of each text corpus in the abnormal text corpus and the normal text corpus are manually labeled to obtain training samples. That is, the training samples include each text corpus and the score corresponding to each text corpus.

[0122] Step 305: Train the abnormal text detection model using the training samples to obtain the pre-trained abnormal text detection model.

[0123] This application does not limit the training method, and the training method can be set according to the specific actual situation. In addition, the abnormal text detection model in this application embodiment can also determine the specific location of abnormal text fragments in the AIGC text to be detected. Therefore, in this application embodiment, after obtaining the pre-trained abnormal text detection model, the first target text corpus and the second target text corpus can be input into the pre-trained abnormal text detection model to obtain each abnormal text fragment; and the labels of the keywords contained in the abnormal text fragments are determined as the labels of the abnormal text fragments, and each abnormal text fragment is determined as the pre-set text fragment dataset mentioned above.

[0124] The embodiments in this application are for illustrative purposes only and do not limit the method for determining the text fragment dataset.

[0125] The emotion categories in this application embodiment include positive emotion categories, neutral emotion categories, and negative emotion categories. The positive emotion category represents emotions with positive value; the neutral emotion category represents emotions that do not reflect values; and the negative emotion category represents emotions with negative value.

[0126] The training method for the sentiment classification detection model is described below: Training samples are obtained, including text corpora and the sentiment categories corresponding to each text corpus; the sentiment classification detection model is trained using the training samples to obtain the pre-trained sentiment classification detection model.

[0127] The text corpora in this application embodiment are the first target text corpus mentioned above and the preset text corpora.

[0128] It should be noted that the training method of the emotion classification and detection model is not limited in this embodiment of the application, and can be set according to the actual situation.

[0129] Step 205: Obtain the security detection score of the AIGC text to be detected based on the third score, the sentiment category of the AIGC to be detected, and the fourth score;

[0130] In one embodiment, step 205 can be implemented in the following three ways:

[0131] Method 1: If the sentiment category of the AIGC to be detected is the negative sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the third score and the fourth score.

[0132] In one embodiment, Method 1 can be specifically implemented as follows: using the second weights of the third score and the fourth score respectively, a weighted sum is performed on the third score and the fourth score to obtain the security detection score of the AIGC text to be detected. The security detection score of the AIGC text to be detected can be obtained through formula (2):

[0133] S=A×s3+B×s4……(2);

[0134] Where S is the security detection score of the AIGC text to be detected, s3 is the third score, s4 is the fourth score, A is the second weight corresponding to the third score, and B is the weight corresponding to the fourth score.

[0135] It should be noted that: A is 0.5 and B is 0.5 in the embodiments of this application, but the embodiments of this application do not limit the specific values ​​of A and B. The specific values ​​can be set according to the actual situation.

[0136] Method 2: If the sentiment category of the AIGC to be detected is the positive sentiment category, then the preset value is determined as the security detection score of the AIGC text to be detected.

[0137] The preset value in this application embodiment is 0, but the specific value of the preset value in this application embodiment is not limited. The preset value in this application embodiment can be set according to the specific actual situation.

[0138] Method 3: If the sentiment category of the AIGC to be detected is the neutral sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score.

[0139] In one embodiment, method three can be specifically implemented as follows: using the first weights of the second value, the third score, and the fourth score, a weighted sum is performed on the second value, the third score, and the fourth score to obtain the security detection score of the AIGC text to be detected. The security detection score of the AIGC text to be detected can be obtained through formula (3):

[0140] S = M × s 22 +N×s3+P×s4……(3);

[0141] Where S is the security detection score of the AIGC text to be detected, s 22 s3 is the second value, s4 is the third score, s4 is the fourth score, M is the first weight corresponding to the second value, N is the first weight corresponding to the third score, and P is the first weight corresponding to the fourth score.

[0142] It should be noted that in this embodiment, M is 0.2, N is 0.4, and P is 0.4. However, the specific values ​​of each first weight are not limited. In this embodiment, the values ​​of each first weight can be set according to the specific actual situation.

[0143] Step 206: Compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0144] In one embodiment, step 206 can be specifically implemented as follows: if the security detection score of the AIGC text to be detected is greater than a preset threshold, then the security detection result of the AIGC text to be detected is determined to be unsafe; if the security detection score of the AIGC text to be detected is not greater than the preset threshold, then the security detection content of the AIGC text to be detected is determined to be safe.

[0145] To further understand the security detection method for generative AIGC text in this application, such as Figure 4 The diagram shown illustrates the flowchart of a generative artificial intelligence (AIGC) text security detection method, which includes the following steps:

[0146] Step 401: Use a pre-set keyword dataset to perform keyword matching on the AIGC text to be detected, and obtain the first score;

[0147] The keyword dataset includes words containing negative and / or neutral information.

[0148] Step 402: Use a pre-set text fragment dataset to perform text fragment matching on the AIGC text to be detected, and obtain a second score;

[0149] The text fragment dataset includes text fragments containing negative and / or neutral information.

[0150] It should be noted that the execution order of steps 401 and 402 is not limited in this embodiment of the application. Step 401 can be executed first, followed by step 402; step 402 can be executed first, followed by step 401; or steps 401 and 402 can be executed simultaneously.

[0151] Step 403: Based on the first score and the second score, determine whether a security detection score for the AIGC text to be detected can be obtained. If not, proceed to step 404; if yes, proceed to step 405.

[0152] Step 404: Based on the first score and the second score, determine the security detection score of the AIGC text to be detected;

[0153] Step 405: Input the AIGC text to be detected into a pre-trained abnormal text detection model to perform abnormal text detection and obtain a third score;

[0154] Step 406: Input the AIGC text to be detected into the pre-trained sentiment classification detection model to obtain the sentiment category and fourth score of the AIGC to be detected;

[0155] It should be noted that the execution order of steps 405 and 406 in this embodiment is not limited here. Step 405 can be executed first, followed by step 406; or step 406 can be executed first, followed by step 405; or steps 405 and 406 can be executed simultaneously.

[0156] Step 407: If the sentiment category of the AIGC to be detected is the negative sentiment category, then the third score and the fourth score are weighted and summed using the second weights of the third score and the fourth score respectively to obtain the security detection score of the AIGC text to be detected;

[0157] Step 408: If the sentiment category of the AIGC to be detected is the positive sentiment category, then the preset value is determined as the security detection score of the AIGC text to be detected;

[0158] Step 409: If the sentiment category of the AIGC to be detected is the neutral sentiment category, then the second value, the third score and the fourth score are weighted and summed using the first weights of the second value, the third score and the fourth score to obtain the security detection score of the AIGC text to be detected.

[0159] Based on the same inventive concept, this application provides a security detection device for generative artificial intelligence AIGC text. The effect of this security detection device for generative artificial intelligence AIGC text is similar to that of the aforementioned method, and will not be described in detail here.

[0160] Figure 5 This is a schematic diagram of the structure of a security detection device for generative artificial intelligence AIGC text according to an embodiment of the present disclosure.

[0161] like Figure 5 As shown, the security detection device 500 for generative artificial intelligence AIGC text disclosed herein may include a matching module 510, a judgment module 520, a first security detection score determination module 530, a second security detection score determination module 540, and a security detection result determination module 550.

[0162] The matching module 510 is used to perform keyword matching on the AIGC text to be detected using a pre-set keyword dataset to obtain a first score; and to perform text segment matching on the AIGC text to be detected using a pre-set text segment dataset to obtain a second score, wherein the keyword dataset includes words with negative information and / or neutral information, and the text segment dataset includes text segments with negative information and / or neutral information.

[0163] The judgment module 520 is used to determine whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score.

[0164] The first security detection score determination module 530 is used to determine the security detection score of the AIGC text to be detected based on the first score and the second score if no.

[0165] The second security detection score determination module 540 is used to, if so, input the AIGC text to be detected into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and input the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC to be detected; and obtain the security detection score of the AIGC text to be detected based on the third score, the sentiment category of the AIGC to be detected, and the fourth score; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category;

[0166] The security detection result determination module 550 is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0167] In one possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity to text fragments in the text fragment dataset that contain negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that contain neutral information.

[0168] The judgment module 520 is specifically used for:

[0169] If the first score is a preset value, and the first value in the second score is the preset value, then it is determined that a security detection score for the AIGC text to be detected cannot be obtained; or,

[0170] If there is a score among the first values ​​of the first score and the second score that is not the preset value, then it is determined that the security detection score of the AIGC text to be detected is obtained.

[0171] In one possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity to text fragments in the text fragment dataset that contain negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that contain neutral information.

[0172] The first security detection score determination module 530 is specifically used for:

[0173] If the first score is greater than the first value, then the first score is determined as the security detection score; or,

[0174] If the first score is not greater than the first value, then the first value is determined as the security detection score.

[0175] In one possible embodiment, the emotion categories include positive emotion categories, neutral emotion categories, and negative emotion categories;

[0176] The second security detection score determination module 540 is specifically used for:

[0177] If the sentiment category of the AIGC to be detected is the negative sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the third score and the fourth score;

[0178] If the sentiment category of the AIGC to be detected is the positive sentiment category, then the preset value is determined as the security detection score of the AIGC text to be detected;

[0179] If the sentiment category of the AIGC to be detected is the neutral sentiment category, then the security detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score.

[0180] In one possible embodiment, the second score includes a first value and a second value, wherein the first value is used to represent the similarity to text fragments in the text fragment dataset that include negative information, and the second value is used to represent the similarity to text fragments in the text fragment dataset that include neutral information.

[0181] The second security detection score determination module 540 performs the step of obtaining the security detection score of the AIGC text to be detected based on the second score, the third score, and the fourth score, specifically for:

[0182] Using the first weights of the second value, the third score, and the fourth score, a weighted sum is performed on the second value, the third score, and the fourth score to obtain the security detection score of the AIGC text to be detected.

[0183] The second security detection score determination module 540 performs the step of obtaining the security detection score of the AIGC text to be detected based on the third score and the fourth score, specifically for:

[0184] Using the second weights of the third and fourth scores respectively, the third and fourth scores are weighted and summed to obtain the security detection score of the AIGC text to be detected.

[0185] In one possible embodiment, the device further includes:

[0186] Model training module 560 is used to obtain the pre-trained abnormal text detection model in the following manner:

[0187] Each keyword in the keyword dataset is input into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is text including the keyword; and,

[0188] Each keyword in the keyword dataset is used to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus that includes at least one of the keywords;

[0189] The first target text corpus and the second target text corpus are identified as abnormal text corpus, and the third target text corpus is identified as normal text corpus; wherein, the third target text corpus is the other text corpus besides the second target text corpus among the preset text corpus;

[0190] Training samples are obtained based on the abnormal text corpus and the normal text corpus;

[0191] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0192] After introducing a security detection method and apparatus for generative artificial intelligence AIGC text according to an exemplary embodiment of the present invention, the following describes an electronic device according to another exemplary embodiment of the present invention.

[0193] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0194] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the security detection method for generative artificial intelligence AIGC text according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 2 Steps 201-206 are shown in the diagram.

[0195] The following reference Figure 6 To describe an electronic device 600 according to this embodiment of the present invention. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0196] like Figure 6 As shown, the components of the electronic device 600 may include, but are not limited to: at least one processor 601, at least one computer storage medium 602, and a bus 603 connecting different system components (including the computer storage medium 602 and the processor 601).

[0197] Bus 603 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0198] Computer storage medium 602 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 621 and / or cache storage medium 622, and may further include read-only computer storage medium (ROM) 623.

[0199] The computer storage medium 602 may also include a program / utility 625 having a set (at least one) of program modules 624, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0200] Electronic device 600 can also communicate with one or more external devices 604 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other AR devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 605. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 606. As shown, network adapter 606 communicates with other modules used in electronic device 600 via bus 603. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0201] In some possible implementations, various aspects of the generative AIGC text security detection method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the generative AIGC text security detection method according to various exemplary embodiments of the present invention described above.

[0202] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for security detection of generative artificial intelligence (AIGC) text, characterized in that, The method includes: The AIGC text to be detected is matched with keywords using a pre-set keyword dataset to obtain a first score; and the AIGC text to be detected is matched with text fragments using a pre-set text fragment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text fragment dataset includes text fragments with negative and / or neutral information. Based on the first score and the second score, determine whether a security detection score for the AIGC text to be detected can be obtained; If so, then based on the first score and the second score, determine the security detection score of the AIGC text to be detected; If not, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC to be detected; if the sentiment category of the AIGC to be detected is negative, a safe detection score of the AIGC text to be detected is obtained based on the third score and the fourth score; if the sentiment category of the AIGC to be detected is positive, a preset value is determined as the safe detection score of the AIGC text to be detected; if the sentiment category of the AIGC to be detected is neutral, a safe detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category; The security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected.

2. The method according to claim 1, characterized in that, The second score includes a first value and a second value; the first value is used to represent the similarity with text fragments containing negative information in the text fragment dataset, and the second value is used to represent the similarity with text fragments containing neutral information in the text fragment dataset; The step of determining whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score includes: If the first score is a preset value, and the first value in the second score is the preset value, then it is determined that a security detection score for the AIGC text to be detected cannot be obtained; or, If there is a score among the first values ​​of the first score and the second score that is not the preset value, then it is determined that the security detection score of the AIGC text to be detected can be obtained.

3. The method according to claim 1, characterized in that, The second score includes a first value and a second value; the first value is used to represent the similarity with text fragments containing negative information in the text fragment dataset, and the second value is used to represent the similarity with text fragments containing neutral information in the text fragment dataset; The process of determining the security detection score of the AIGC text to be detected based on the first score and the second score includes: If the first score is greater than the first value, then the first score is determined as the security detection score; or, If the first score is not greater than the first value, then the first value is determined as the security detection score.

4. The method according to claim 1, characterized in that, The second score includes a first value and a second value, wherein the first value is used to represent the similarity with text fragments in the text fragment dataset that include negative information, and the second value is used to represent the similarity with text fragments in the text fragment dataset that include neutral information; The process of obtaining the security detection score of the AIGC text to be detected based on the second score, the third score, and the fourth score includes: Using the first weights of the second value, the third score, and the fourth score, a weighted sum is performed on the second value, the third score, and the fourth score to obtain the security detection score of the AIGC text to be detected. The process of obtaining the security detection score of the AIGC text to be detected based on the third score and the fourth score includes: Using the second weights of the third and fourth scores respectively, the third and fourth scores are weighted and summed to obtain the security detection score of the AIGC text to be detected.

5. The method according to claim 1, characterized in that, The pre-trained abnormal text detection model is obtained through the following method: Each keyword in the keyword dataset is input into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is text including the keyword; and, Each keyword in the keyword dataset is used to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus that includes at least one of the keywords; The first target text corpus and the second target text corpus are identified as abnormal text corpus, and the third target text corpus is identified as normal text corpus; wherein, the third target text corpus is the other text corpus besides the second target text corpus among the preset text corpus; Training samples are obtained based on the abnormal text corpus and the normal text corpus; The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

6. A security detection device for generative artificial intelligence (AIGC) text, characterized in that, The device includes: The matching module is used to perform keyword matching on the AIGC text to be detected using a pre-set keyword dataset to obtain a first score; and to perform text segment matching on the AIGC text to be detected using a pre-set text segment dataset to obtain a second score, wherein the keyword dataset includes words with negative and / or neutral information, and the text segment dataset includes text segments with negative and / or neutral information. The judgment module is used to determine whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score. The first security detection score determination module is used to determine the security detection score of the AIGC text to be detected based on the first score and the second score if the condition is met. The second security detection score determination module is used to, if not, input the AIGC text to be detected into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and input the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category and a fourth score of the AIGC text to be detected; if the sentiment category of the AIGC text to be detected is negative, then the security detection score of the AIGC text to be detected is obtained based on the third score and the fourth score; if the sentiment category of the AIGC text to be detected is positive, then a preset value is determined as the security detection score of the AIGC text to be detected; if the sentiment category of the AIGC text to be detected is neutral, then the security detection score of the AIGC text to be detected is obtained based on the second score, the third score, and the fourth score; wherein, the third score is used to represent the confidence level that the AIGC text to be detected is abnormal text, and the fourth score is used to represent the confidence level that the AIGC text to be detected belongs to the sentiment category; The security detection result determination module is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

7. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method as described in any one of claims 1-5.

Images