Security detection method for generative artificial intelligence (AIGC) text and related equipment

Abstract

The invention discloses a safety detection method for a generative artificial intelligence (AIGC) text and related equipment, which are used for improving the accuracy of safety detection. Comprising the following steps: determining whether a security detection score can be obtained or not according to a first score obtained by detecting a to-be-detected AIGC text by utilizing a preset keyword data set and a second score obtained by performing text fragment matching on the to-be-detected AIGC text by utilizing a preset text fragment data set; if not, determining a security detection score based on the first score and the second score; if yes, determining a security detection score of the to-be-detected AIGC text according to a third score obtained by inputting the to-be-detected AIGC text into a pre-trained abnormal text detection model and an emotion category and a fourth score obtained by inputting the to-be-detected AIGC text into a pre-trained emotion classification detection model; and comparing the safety detection score with a preset threshold value to obtain a safety detection result.

Description

Background Art

[0002] As AI (Artificial Intelligence) big model technology empowers thousands of industries, it is profoundly changing all walks of life. However, as the capabilities of AI big models continue to increase, ensuring that they truly create value for human society rather than bring threats has become an important issue in the current development of AI. As an important dimension of AI's good and sustainable development, the security, trustworthiness and responsible development of the content generated by industry big models are increasingly inseparable from effective governance and security measures.

[0003] The training data of large language models mostly comes from public data on the Internet. It is difficult to construct model values, which may lead to value problems in the output content. At the same time, the direct and indirect prompt injections from users will lead to unpredictable illegal content. Therefore, the content security review of AIGC (Artificial Intelligence Generated Content) texts has always been the focus of network platforms and regulators. It aims to filter out illegal, harmful or inappropriate content. In order to reduce the negative impact that the promotion of bad content may bring.

[0004] At present, the content review of AIGC text mainly relies on matching and identifying the content with sensitive word libraries. However, the effectiveness of this method depends on the richness of the word library, and it is easy to cause false positives due to semantic ambiguity, resulting in low accuracy of AIGC text detection. Summary of the invention

[0005] The embodiments of the present application provide a security detection method and related equipment for generative artificial intelligence AIGC text, which are used to improve the accuracy of security detection of AIGC text.

[0006] In a first aspect, the present application provides a security detection method for generative artificial intelligence AIGC text, the method comprising:

[0007] Using a preset keyword data set to perform keyword matching on the AIGC text to be detected to obtain a first score; and using a preset text segment data set to perform text segment matching on the AIGC text to be detected to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information;

[0008] Determining whether a security detection score of the AIGC text to be detected can be obtained according to the first score and the second score;

[0009] If not, determining a security detection score of the AIGC text to be detected based on the first score and the second score;

[0010] If yes, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; according to the third score, the emotion category of the AIGC to be detected and the fourth score, a safety detection score of the AIGC text to be detected is obtained; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category;

[0011] The security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0012] In the embodiment of the present application, the first score is obtained by matching keywords on the AIGC text to be detected; and the text fragments of the AIGC text to be detected are matched with a pre-set text fragment data set to obtain a second score, and then according to the first score and the second score, it is determined whether the security detection score of the AIGC text to be detected can be obtained. If not, the security detection score of the AIGC text to be detected is determined based on the first score and the second score; if so, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; and according to the third score, the emotion category of the AIGC to be detected and the fourth score, the security detection score of the AIGC text to be detected is obtained. Finally, the security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected. Therefore, the embodiment of the present application determines the security detection result of the AIGC to be detected through multi-dimensional detection, thereby improving the accuracy of the security detection result.

[0013] In a possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second value is used to represent the similarity with the text segment containing neutral information in the text segment data set;

[0014] The determining, according to the first score and the second score, whether a security detection score of the AIGC text to be detected can be obtained includes:

[0015] If the first score is a preset value, and the first value of the second score is the preset value, it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or,

[0016] If there is a score in the first value of the first score and the second score that is not the preset value, it is determined that a security detection score of the AIGC text to be detected can be obtained.

[0017] In one embodiment, the second score includes a first numerical value and a second numerical value; the first numerical value is used to represent the similarity with the text segment of negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment of neutral information in the text segment data set;

[0018] The step of determining the security detection score of the AIGC text to be detected based on the first score and the second score includes:

[0019] If the first score is greater than the first value, the first score is determined as the safety detection score; or,

[0020] If the first score is not greater than the first value, the first value is determined as the safety detection score.

[0021] In one embodiment, the emotion categories include a positive emotion category, a neutral emotion category, and a negative emotion category;

[0022] The step of obtaining a security detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC text to be detected, and the fourth score includes:

[0023] If the emotion category of the AIGC to be detected is the negative emotion category, obtaining a safety detection score of the AIGC text to be detected according to the third score and the fourth score;

[0024] If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected;

[0025] If the emotion category of the AIGC to be detected is the neutral emotion category, the safety detection score of the AIGC text to be detected is obtained according to the second score, the third score and the fourth score.

[0026] In one embodiment, the second score includes a first numerical value and a second numerical value, wherein the first numerical value is used to represent the similarity with the text segment including negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment including neutral information in the text segment data set;

[0027] The obtaining, according to the second score, the third score, and the fourth score, a security detection score of the AIGC text to be detected includes:

[0028] Using the first weights of the second value, the third score, and the fourth score, respectively, to perform a weighted summation on the second value, the third score, and the fourth score to obtain a security detection score for the AIGC text to be detected;

[0029] The obtaining, according to the third score and the fourth score, a security detection score of the AIGC text to be detected includes:

[0030] The third score and the fourth score are weightedly summed using their respective second weights to obtain a security detection score for the AIGC text to be detected.

[0031] In one embodiment, the pre-trained abnormal text detection model is obtained by:

[0032] Inputting each keyword in the keyword data set into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is a text including the keyword; and

[0033] Using each keyword in the keyword data set to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus including at least one keyword among the keywords;

[0034] Determine the first target text corpus and the second target text corpus as abnormal text corpora, and determine the third target text corpus as normal text corpus; wherein the third target text corpus is other text corpora among the preset text corpora except the second target text corpus;

[0035] Obtaining training samples according to the abnormal text corpus and the normal text corpus;

[0036] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0037] In a second aspect, the present application provides a security detection device for a generative artificial intelligence AIGC text, the device comprising:

[0038] A matching module, configured to perform keyword matching on the AIGC text to be detected using a preset keyword data set to obtain a first score; and perform text segment matching on the AIGC text to be detected using a preset text segment data set to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information;

[0039] A judgment module, configured to determine whether a security detection score of the AIGC text to be detected can be obtained according to the first score and the second score;

[0040] A first security detection score determination module, configured to, if not, determine the security detection score of the AIGC text to be detected based on the first score and the second score;

[0041] A second security detection score determination module is used for, if yes, inputting the AIGC text to be detected into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and inputting the AIGC text to be detected into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; obtaining the security detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC to be detected and the fourth score; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category;

[0042] The security detection result determination module is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0043] In a possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second value is used to represent the similarity with the text segment containing neutral information in the text segment data set;

[0044] The judgment module is specifically used for:

[0045] If the first score is a preset value, and the first value of the second score is the preset value, it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or,

[0046] If there is a score in the first value of the first score and the second score that is not the preset value, it is determined that a security detection score of the AIGC text to be detected can be obtained.

[0047] In a possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second value is used to represent the similarity with the text segment containing neutral information in the text segment data set;

[0048] The first safety detection score determination module is specifically used to:

[0049] If the first score is greater than the first value, the first score is determined as the safety detection score; or,

[0050] If the first score is not greater than the first value, the first value is determined as the safety detection score.

[0051] In a possible embodiment, the emotion category includes a positive emotion category, a neutral emotion category, and a negative emotion category;

[0052] The second safety detection score determination module is specifically used to:

[0053] If the emotion category of the AIGC to be detected is the negative emotion category, obtaining a safety detection score of the AIGC text to be detected according to the third score and the fourth score;

[0054] If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected;

[0055] If the emotion category of the AIGC to be detected is the neutral emotion category, the safety detection score of the AIGC text to be detected is obtained according to the second score, the third score and the fourth score.

[0056] In a possible embodiment, the second score includes a first value and a second value, wherein the first value is used to represent the similarity with the text segment including negative information in the text segment data set, and the second value is used to represent the similarity with the text segment including neutral information in the text segment data set;

[0057] The second security detection score determination module performs the step of obtaining the security detection score of the AIGC text to be detected according to the second score, the third score, and the fourth score, and is specifically used for:

[0058] Using the first weights of the second value, the third score, and the fourth score, respectively, to perform a weighted summation on the second value, the third score, and the fourth score to obtain a security detection score for the AIGC text to be detected;

[0059] The second security detection score determination module executes the step of obtaining the security detection score of the AIGC text to be detected according to the third score and the fourth score, and is specifically used for:

[0060] The third score and the fourth score are weightedly summed using their respective second weights to obtain a security detection score for the AIGC text to be detected.

[0061] In a possible embodiment, the device further includes:

[0062] The model training module is used to obtain the pre-trained abnormal text detection model in the following manner:

[0063] Inputting each keyword in the keyword data set into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is a text including the keyword; and

[0064] Using each keyword in the keyword data set to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus including at least one keyword among the keywords;

[0065] Determine the first target text corpus and the second target text corpus as abnormal text corpora, and determine the third target text corpus as normal text corpus; wherein the third target text corpus is other text corpora among the preset text corpora except the second target text corpus;

[0066] Obtaining training samples according to the abnormal text corpus and the normal text corpus;

[0067] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0068] In a third aspect, the present application provides an electronic device, including:

[0069] A memory for storing program instructions;

[0070] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.

[0071] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0072] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application;

[0074] Figure 2 One of the flow charts of a security detection method for a generative artificial intelligence AIGC text provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a process for training an abnormal text detection model provided in an embodiment of the present application;

[0076] Figure 4 The second flowchart of the security detection method of the generative artificial intelligence AIGC text provided in the embodiment of the present application;

[0077] Figure 5 A schematic diagram of a security detection device for a generative artificial intelligence AIGC text provided by an embodiment of the present application;

[0078] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0080] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0081] In the technical solution of this application, the collection, dissemination, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0082] Before introducing the security detection method of the generative artificial intelligence AIGC text provided by the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0083] In the prior art, the content review of AIGC texts mainly relies on matching and identifying the content in sensitive word libraries. However, the effectiveness of this method depends on the richness of the word library, and it is easy to cause false positives due to semantic ambiguity, resulting in a low accuracy rate for AIGC text detection.

[0084] Therefore, the present application provides a security detection method for generative artificial intelligence AIGC text. First, a first score is obtained by keyword matching the AIGC text to be detected; and a text segment matching is performed on the AIGC text to be detected using a pre-set text segment data set to obtain a second score, and then according to the first score and the second score, it is determined whether the security detection score of the AIGC text to be detected can be obtained. If not, the security detection score of the AIGC text to be detected is determined based on the first score and the second score; if so, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained emotion classification detection model to obtain the emotion category and fourth score of the AIGC to be detected; and according to the third score, the emotion category and fourth score of the AIGC to be detected, the security detection score of the AIGC text to be detected is obtained. Finally, the security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected. Thus, the embodiment of the present application determines the security detection result of the AIGC to be detected through multi-dimensional detection, thereby improving the accuracy of the security detection result. Below, the scheme of the present disclosure is described in detail in conjunction with the accompanying drawings.

[0085] like Figure 1 As shown, a schematic diagram of an application scenario of a security detection method for a generative artificial intelligence AIGC text is shown, and the application scenario includes a server 110 and a terminal device 120.

[0086] In a possible application scenario, the server 110 uses a preset keyword data set to perform keyword matching on the AIGC text to be detected to obtain a first score; and uses a preset text segment data set to perform text segment matching on the AIGC text to be detected to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information; then the server 110 determines whether a security detection score for the AIGC text to be detected can be obtained based on the first score and the second score; if not, the server 110 determines the security detection score of the AIGC text to be detected based on the first score and the second score; if so, the server 110 matches the AIGC text to be detected with the first score and the second score. The AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; according to the third score, the emotion category of the AIGC to be detected and the fourth score, the security detection score of the AIGC text to be detected is obtained; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category; finally, the server 110 compares the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected. And the security detection result of the AIGC to be detected is sent to the terminal device 120 for display.

[0087] in, Figure 1 The server 110 and the terminal device 120 can exchange information through a communication network, wherein the communication mode adopted by the communication network can be divided into a wireless communication mode or a wired communication mode.

[0088] Exemplarily, the server 110 can access the network through cellular mobile communication technology to communicate with the terminal device 120, wherein the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0089] Optionally, the server 110 may access the network and communicate with the terminal device 120 via short-range wireless communication, wherein the short-range wireless communication method, for example, includes Wireless Fidelity (Wi-Fi) technology.

[0090] Furthermore, the server 110 in the embodiment of the present application may be a physical server or a cloud server. And the description in the present application only describes one server 110 and one slave terminal device 120 in detail, but it should be understood by those skilled in the art that the server 110 and the slave terminal device 120 shown are intended to represent the operation of the server 110 and the terminal device 120 involved in the technical solution of the present application. It does not imply any limitation on the number, type or location of the server 110 and the slave terminal device 120. It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concept of the exemplary embodiment of the present application will not be changed.

[0091] It should be noted that the security detection method for generative artificial intelligence AIGC text proposed in this application is not only applicable to Figure 1 The application scenario shown is also applicable to any security detection device with generative artificial intelligence AIGC text.

[0092] The following describes the security detection method of generative artificial intelligence AIGC text in an exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of the present application, and the implementation methods of the present application are not limited in this respect.

[0093] like Figure 2 As shown, it is a flowchart of the security detection method of the generative artificial intelligence AIGC text, which specifically includes the following steps:

[0094] Step 201: using a preset keyword data set to perform keyword matching on the AIGC text to be detected to obtain a first score; and using a preset text segment data set to perform text segment matching on the AIGC text to be detected to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information;

[0095] Each keyword in the keyword data set in the embodiment of the present application can be collected from the Internet and news corpus, etc. In addition, each keyword in the embodiment of the present application has a label, which is used to indicate whether the keyword is a word of negative information or a word of neutral information.

[0096] In addition, in the embodiment of the present application, the method of keyword matching for the AIGC text to be detected can be performed using an AC automaton matching algorithm or a traversal method. Since the AC automaton matching algorithm or the traversal method are both methods in the prior art and do not belong to the innovation of the present invention, the embodiment of the present application will not be described in detail here.

[0097] In one embodiment, the first score is obtained by:

[0098] The AIGC text to be detected is matched with text fragments using a pre-set text fragment data set. If a keyword labeled as negative information is detected in the AIGC text to be detected, the first score is determined to be a first specified score; if a keyword labeled as neutral information is detected in the AIGC text to be detected, the first score is determined to be a second specified score; if no keyword is detected in the AIGC text to be detected, the first score is determined to be a third specified score; wherein the second specified score is greater than the first specified score and less than the first specified score.

[0099] It should be noted that the first designated score in the embodiment of the present application is 1, the second designated score is 0.5, and the third designated score is 0. However, the embodiment of the present application does not limit the first designated score, the second designated score, and the third designated score. The specific values ​​of the first designated score, the second designated score, and the third designated score in the embodiment of the present application can be set according to the specific actual situation.

[0100] In the embodiment of the present application, the pre-set text segment data set is used to match the AIGC text to be detected, and the second score can be obtained by:

[0101] Each text segment in the pre-set text segment data set is vectorized to obtain a first vector corresponding to each text segment; the AIGC text to be detected is vectorized to obtain a second vector; for any one of the text segments, based on the first vector of the any one text segment and the second vector of the AIGC text to be detected, a cosine value between the first vector and the second vector is obtained, and the cosine value is determined as the similarity between the first vector and the second vector, and the similarity with the largest value among the similarities is determined as the second score.

[0102] It should be noted that the method of vectorizing each text segment and vectorizing the AIGC text to be detected in the embodiment of the present application can be: TF-IDF (Term Frequency-Inverse Document Frequency) algorithm, Word2Vec (word to vector) algorithm and BOW (Bag of Words) algorithm, etc. The specific method can be set according to the specific actual situation, and the embodiment of the present application is not limited here.

[0103] Step 202: Determine whether the security detection score of the AIGC text to be detected can be obtained according to the first score and the second score, if not, execute step 203, if yes, execute step 204;

[0104] The second score in the embodiment of the present application includes a first numerical value and a second numerical value; the first numerical value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment containing neutral information in the text segment data set.

[0105] In one embodiment, step 202 can be specifically implemented as follows: if the first score is a preset value, and the first value of the second score is the preset value, it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or if there is a score in the first score and the first value of the second score that is not the preset value, it is determined that the security detection score of the AIGC text to be detected can be obtained.

[0106] It should be noted that the preset value in the embodiment of the present application is 0, but the preset value in the embodiment of the present application is not limited. The preset value in the embodiment of the present application can be set according to the specific actual situation.

[0107] Step 203: Determine a security detection score of the AIGC text to be detected based on the first score and the second score;

[0108] In one embodiment, step 203 may be specifically implemented as follows: if the first score is greater than the first value, the first score is determined as the security detection score; or if the first score is not greater than the first value, the first value is determined as the security detection score. The security detection score of the AIGC text to be detected can be obtained by formula (1):

[0109] S = max(s1,s 21 )……(1);

[0110] Where S is the security detection score of the AIGC text to be detected, s1 is the first score, and s 21 is the first value of the second fraction.

[0111] Step 204: input the AIGC text to be detected into a pre-trained abnormal text detection model to perform abnormal text detection, and obtain a third score; and input the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category of the AIGC to be detected and a fourth score; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the sentiment category;

[0112] Next, the training method of the abnormal text detection model in the embodiment of the present application is first introduced. Figure 3 As shown in FIG. 1 , a flow chart of training an abnormal text detection model is shown, which may specifically include the following steps:

[0113] Step 301: input each keyword in the keyword data set into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is a text including the keyword;

[0114] The label corresponding to the first target text corpus in the embodiment of the present application is the same as the label of the keyword corresponding to the first target text corpus.

[0115] For example, if the keyword is "safety", the corresponding first target text corpus is: xxxxxxxsafetyxxxxxxxxxxxxx.

[0116] Step 302: using each keyword in the keyword data set to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus including at least one keyword among the keywords;

[0117] The label of the second target text corpus in the embodiment of the present application is the same as the label of the keyword included in the second target text corpus.

[0118] The preset text corpora in the embodiment of the present application are text corpora obtained by randomly inputting words into the text generation model. The detection method in the embodiment of the present application can be a traversal method, that is, searching in the text corpora whether the keywords exist, and determining the text corpora with at least one keyword as the second text corpus.

[0119] Step 303: determining the first target text corpus and the second target text corpus as abnormal text corpora, and determining the third target text corpus as normal text corpus; wherein the third target text corpus is other text corpora among the preset text corpora except the second target text corpus;

[0120] Step 304: obtaining training samples according to the abnormal text corpus and the normal text corpus;

[0121] In the embodiment of the present application, the abnormal text corpus and each text corpus in the normal text corpus are manually annotated with corresponding scores to obtain a training sample, that is, the training sample includes each text corpus and the scores corresponding to each text corpus.

[0122] Step 305: Use the training samples to train the abnormal text detection model to obtain the pre-trained abnormal text detection model.

[0123] The embodiment of the present application does not limit the training method, and the training method can be set according to the specific actual situation. In addition, the abnormal text detection model in the embodiment of the present application can also determine the specific position of the abnormal text segment in the AIGC text to be detected. Therefore, in the embodiment of the present application, after obtaining the pre-trained abnormal text detection model, the first target text corpus and the second target text corpus can be input into the pre-trained abnormal text detection model to obtain each abnormal text segment; and the label of the keyword contained in the abnormal text segment is determined as the label of the abnormal text segment, and each abnormal text segment is determined as the pre-set text segment data set described above.

[0124] The embodiments of the present application are only used for illustration and do not limit the method for determining the text segment data set.

[0125] The emotion categories in the embodiment of the present application include positive emotion categories, neutral emotion categories, and negative emotion categories. Among them, the positive emotion category is used to represent emotions with positive values; the neutral emotion category is used to represent emotions that do not reflect values. The negative emotion category is used to represent emotions with negative values.

[0126] The following is an introduction to the training method of the sentiment classification detection model: obtaining training samples, wherein the training samples include various text corpora and sentiment categories corresponding to the various text corpora; using the training samples to train the sentiment classification detection model to obtain the pre-trained sentiment classification detection model.

[0127] The text corpora in the embodiment of the present application are the first target text corpora and the preset text corpora mentioned above.

[0128] It should be noted that the embodiment of the present application does not limit the training method of the emotion classification detection model, and the specific setting can be made according to the actual situation.

[0129] Step 205: Obtaining a safety detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC text to be detected, and the fourth score;

[0130] In one embodiment, step 205 can be specifically implemented in the following three ways:

[0131] Method 1: If the emotion category of the AIGC to be detected is the negative emotion category, a safety detection score of the AIGC text to be detected is obtained according to the third score and the fourth score.

[0132] In one embodiment, the first method can be specifically implemented as follows: using the second weights of the third score and the fourth score, weighted summing the third score and the fourth score to obtain the security detection score of the AIGC text to be detected. The security detection score of the AIGC text to be detected can be obtained by formula (2):

[0133] S = A × s3 + B × s4 ... (2);

[0134] Among them, S is the security detection score of the AIGC text to be detected, s3 is the third score, s4 is the fourth score, A is the second weight corresponding to the third score, and B is the weight corresponding to the fourth score.

[0135] It should be noted that: A in the embodiment of the present application is 0.5 and B is 0.5, but the embodiment of the present application does not limit the specific values ​​of A and B, and the specific values ​​can be set according to actual conditions.

[0136] Method 2: If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected.

[0137] The preset value in the embodiment of the present application is 0, but the specific value of the preset value in the embodiment of the present application is not limited. The preset value in the embodiment of the present application can be set according to the specific actual situation.

[0138] Method three: If the emotion category of the AIGC to be detected is the neutral emotion category, a safety detection score of the AIGC text to be detected is obtained according to the second score, the third score and the fourth score.

[0139] In one embodiment, the third method can be specifically implemented as follows: using the first weights of the second value, the third score and the fourth score, weighted summing the second value, the third score and the fourth score to obtain the security detection score of the AIGC text to be detected. The security detection score of the AIGC text to be detected can be obtained by formula (3):

[0140] S=M×s 22 +N×s3+P×s4……(3);

[0141] Where S is the security detection score of the AIGC text to be detected, s 22 is the second value, s3 is the third score, s4 is the fourth score, M is the first weight corresponding to the second value, N is the first weight corresponding to the third score, and P is the first weight corresponding to the fourth score.

[0142] It should be noted that: in the embodiment of the present application, M is 0.2, N is 0.4, and P is 0.4. However, the specific value of each first weight is not limited, and the value of each first weight can be set according to the specific actual situation in the embodiment of the present application.

[0143] Step 206: Compare the security detection score of the AIGC text to be detected with a preset threshold to obtain a security detection result of the AIGC text to be detected.

[0144] In one embodiment, step 206 can be specifically implemented as follows: if the security detection score of the AIGC text to be detected is greater than a preset threshold, the security detection result of the AIGC text to be detected is determined to be unsafe; if the security detection score of the AIGC text to be detected is not greater than the preset threshold, the security detection content of the AIGC text to be detected is determined to be safe.

[0145] In order to further understand the security detection method of the generative artificial intelligence AIGC text in this application, Figure 4 As shown, it is a flowchart of the security detection method of the generative artificial intelligence AIGC text, which specifically includes the following steps:

[0146] Step 401: Use a preset keyword data set to perform keyword matching on the AIGC text to be detected to obtain a first score;

[0147] The keyword data set includes words with negative information and / or neutral information.

[0148] Step 402: using a preset text segment data set to perform text segment matching on the AIGC text to be detected to obtain a second score;

[0149] The text segment data set includes text segments containing negative information and / or neutral information.

[0150] It should be noted that the execution sequence of step 401 and step 402 is not limited in the embodiment of the present application. Step 401 may be executed first, and then step 402; step 402 may be executed first, and then step 401; or step 401 and step 402 may be executed simultaneously.

[0151] Step 403: Determine whether the security detection score of the AIGC text to be detected can be obtained according to the first score and the second score, if not, execute step 404, if yes, execute step 405;

[0152] Step 404: Determine a security detection score of the AIGC text to be detected based on the first score and the second score;

[0153] Step 405: inputting the AIGC text to be detected into a pre-trained abnormal text detection model to perform abnormal text detection and obtain a third score;

[0154] Step 406: inputting the AIGC text to be detected into a pre-trained sentiment classification detection model to obtain the sentiment category and fourth score of the AIGC to be detected;

[0155] It should be noted that the execution sequence of step 405 and step 406 in the embodiment of the present application is not limited in the embodiment of the present application. Step 405 may be executed first, and then step 406; step 406 may be executed first, and then step 405; or step 405 and step 406 may be executed simultaneously.

[0156] Step 407: If the emotion category of the AIGC to be detected is the negative emotion category, weighted summing the third score and the fourth score using the respective second weights of the third score and the fourth score to obtain a safety detection score of the AIGC text to be detected;

[0157] Step 408: If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected;

[0158] Step 409: If the emotion category of the AIGC to be detected is the neutral emotion category, the second value, the third score and the fourth score are weightedly summed using their respective first weights to obtain a safety detection score for the AIGC text to be detected.

[0159] Based on the same inventive concept, an embodiment of the present application provides a security detection device for generative artificial intelligence AIGC text. The effect of the security detection device for generative artificial intelligence AIGC text is similar to that of the aforementioned method, which will not be repeated here.

[0160] Figure 5 The present invention is a schematic diagram of the structure of a security detection device for a generative artificial intelligence AIGC text according to an embodiment of the present disclosure.

[0161] like Figure 5 As shown, the security detection device 500 of the generative artificial intelligence AIGC text disclosed in the present invention may include a matching module 510, a judgment module 520, a first security detection score determination module 530, a second security detection score determination module 540 and a security detection result determination module 550.

[0162] The matching module 510 is used to perform keyword matching on the AIGC text to be detected using a preset keyword data set to obtain a first score; and perform text segment matching on the AIGC text to be detected using a preset text segment data set to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information;

[0163] A determination module 520, configured to determine whether a security detection score of the AIGC text to be detected can be obtained according to the first score and the second score;

[0164] A first security detection score determination module 530, configured to determine, if not, a security detection score of the AIGC text to be detected based on the first score and the second score;

[0165] The second safety detection score determination module 540 is used for, if yes, inputting the AIGC text to be detected into a pre-trained abnormal text detection model to perform abnormal text detection to obtain a third score; and inputting the AIGC text to be detected into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; obtaining the safety detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC to be detected and the fourth score; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category;

[0166] The security detection result determination module 550 is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

[0167] In a possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second value is used to represent the similarity with the text segment containing neutral information in the text segment data set;

[0168] The determination module 520 is specifically used for:

[0169] If the first score is a preset value, and the first value of the second score is the preset value, it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or,

[0170] If there is a score in the first value of the first score and the second score that is not the preset value, it is determined that a security detection score of the AIGC text to be detected can be obtained.

[0171] In a possible embodiment, the second score includes a first value and a second value; the first value is used to represent the similarity with the text segment containing negative information in the text segment data set, and the second value is used to represent the similarity with the text segment containing neutral information in the text segment data set;

[0172] The first safety detection score determination module 530 is specifically used to:

[0173] If the first score is greater than the first value, the first score is determined as the safety detection score; or,

[0174] If the first score is not greater than the first value, the first value is determined as the safety detection score.

[0175] In a possible embodiment, the emotion category includes a positive emotion category, a neutral emotion category, and a negative emotion category;

[0176] The second safety detection score determination module 540 is specifically used to:

[0177] If the emotion category of the AIGC to be detected is the negative emotion category, obtaining a safety detection score of the AIGC text to be detected according to the third score and the fourth score;

[0178] If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected;

[0179] If the emotion category of the AIGC to be detected is the neutral emotion category, the safety detection score of the AIGC text to be detected is obtained according to the second score, the third score and the fourth score.

[0180] In a possible embodiment, the second score includes a first value and a second value, wherein the first value is used to represent the similarity with the text segment including negative information in the text segment data set, and the second value is used to represent the similarity with the text segment including neutral information in the text segment data set;

[0181] The second security detection score determination module 540 performs the step of obtaining the security detection score of the AIGC text to be detected according to the second score, the third score, and the fourth score, specifically for:

[0182] Using the first weights of the second value, the third score, and the fourth score, respectively, to perform a weighted summation on the second value, the third score, and the fourth score to obtain a security detection score for the AIGC text to be detected;

[0183] The second security detection score determination module 540 performs the step of obtaining the security detection score of the AIGC text to be detected according to the third score and the fourth score, specifically for:

[0184] The third score and the fourth score are weightedly summed using their respective second weights to obtain a security detection score for the AIGC text to be detected.

[0185] In a possible embodiment, the device further includes:

[0186] The model training module 560 is used to obtain the pre-trained abnormal text detection model by:

[0187] Inputting each keyword in the keyword data set into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is a text including the keyword; and

[0188] Using each keyword in the keyword data set to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus including at least one keyword among the keywords;

[0189] Determine the first target text corpus and the second target text corpus as abnormal text corpora, and determine the third target text corpus as normal text corpus; wherein the third target text corpus is other text corpora among the preset text corpora except the second target text corpus;

[0190] Obtaining training samples according to the abnormal text corpus and the normal text corpus;

[0191] The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

[0192] After introducing a method and device for security detection of a generative artificial intelligence AIGC text according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.

[0193] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0194] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the security detection method of the generative artificial intelligence AIGC text according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201-206 shown in .

[0195] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present invention is described. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0196] like Figure 6 As shown, the components of the electronic device 600 may include but are not limited to: the at least one processor 601 mentioned above, the at least one computer storage medium 602 mentioned above, and a bus 603 connecting different system components (including the computer storage medium 602 and the processor 601).

[0197] Bus 603 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0198] The computer storage media 602 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 621 and / or cache storage media 622 , and may further include read-only computer storage media (ROM) 623 .

[0199] The computer storage medium 602 may also include a program / utility 625 having a set (at least one) of program modules 624, such program modules 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0200] The electronic device 600 may also communicate with one or more external devices 604 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 605. In addition, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 606. As shown, the network adapter 606 communicates with other modules for the electronic device 600 through a bus 603. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0201] In some possible implementations, various aspects of a method for security detection of a generative artificial intelligence AIGC text provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for security detection of a generative artificial intelligence AIGC text according to various exemplary embodiments of the present invention described above in this specification.

[0202] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A security detection method for generative artificial intelligence AIGC text, characterized in that: The method comprises: Using a preset keyword data set to perform keyword matching on the AIGC text to be detected to obtain a first score; and using a preset text segment data set to perform text segment matching on the AIGC text to be detected to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information; Determining whether a security detection score of the AIGC text to be detected can be obtained according to the first score and the second score; If not, determining a security detection score of the AIGC text to be detected based on the first score and the second score; If yes, the AIGC text to be detected is input into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and the AIGC text to be detected is input into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; according to the third score, the emotion category of the AIGC to be detected and the fourth score, a safety detection score of the AIGC text to be detected is obtained; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category; The security detection score of the AIGC text to be detected is compared with a preset threshold to obtain the security detection result of the AIGC text to be detected.

2. The method according to claim 1, characterized in that The second score includes a first numerical value and a second numerical value; the first numerical value is used to represent the similarity with the text segment of negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment of neutral information in the text segment data set; The determining, according to the first score and the second score, whether a security detection score of the AIGC text to be detected can be obtained includes: If the first score is a preset value, and the first value of the second score is the preset value, it is determined that the security detection score of the AIGC text to be detected cannot be obtained; or, If there is a score in the first value of the first score and the second score that is not the preset value, it is determined that a security detection score of the AIGC text to be detected can be obtained.

3. The method according to claim 1, characterized in that: The second score includes a first numerical value and a second numerical value; the first numerical value is used to represent the similarity with the text segment of negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment of neutral information in the text segment data set; The step of determining the security detection score of the AIGC text to be detected based on the first score and the second score includes: If the first score is greater than the first value, the first score is determined as the safety detection score; or, If the first score is not greater than the first value, the first value is determined as the safety detection score.

4. The method according to claim 1, characterized in that: The emotion categories include positive emotion categories, neutral emotion categories and negative emotion categories; The step of obtaining a security detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC to be detected, and the fourth score includes: If the emotion category of the AIGC to be detected is the negative emotion category, obtaining a safety detection score of the AIGC text to be detected according to the third score and the fourth score; If the emotion category of the AIGC to be detected is the positive emotion category, a preset value is determined as the safety detection score of the AIGC text to be detected; If the emotion category of the AIGC to be detected is the neutral emotion category, the safety detection score of the AIGC text to be detected is obtained according to the second score, the third score and the fourth score.

5. The method according to claim 4, characterized in that The second score includes a first numerical value and a second numerical value, wherein the first numerical value is used to represent the similarity with the text segment including negative information in the text segment data set, and the second numerical value is used to represent the similarity with the text segment including neutral information in the text segment data set; The obtaining, according to the second score, the third score, and the fourth score, a security detection score of the AIGC text to be detected includes: Using the first weights of the second value, the third score, and the fourth score, respectively, to perform a weighted summation on the second value, the third score, and the fourth score to obtain a security detection score for the AIGC text to be detected; The obtaining, according to the third score and the fourth score, a security detection score of the AIGC text to be detected includes: The third score and the fourth score are weightedly summed using their respective second weights to obtain a security detection score for the AIGC text to be detected.

6. The method according to claim 1, characterized in that The pre-trained abnormal text detection model is obtained by: Inputting each keyword in the keyword data set into a pre-trained text generation model to obtain a first target text corpus corresponding to each keyword, wherein the first target text corpus corresponding to any keyword is a text including the keyword; and Using each keyword in the keyword data set to detect each preset text corpus to obtain each second target text corpus, wherein any second target text corpus is a text corpus including at least one keyword among the keywords; Determine the first target text corpus and the second target text corpus as abnormal text corpora, and determine the third target text corpus as normal text corpus; wherein the third target text corpus is other text corpora among the preset text corpora except the second target text corpus; Obtaining training samples according to the abnormal text corpus and the normal text corpus; The abnormal text detection model is trained using the training samples to obtain the pre-trained abnormal text detection model.

7. A security detection device for generative artificial intelligence AIGC text, characterized in that: The device comprises: A matching module, configured to perform keyword matching on the AIGC text to be detected using a preset keyword data set to obtain a first score; and perform text segment matching on the AIGC text to be detected using a preset text segment data set to obtain a second score, wherein the keyword data set includes words with negative information and / or neutral information, and the text segment data set includes text segments with negative information and / or neutral information; A judgment module, configured to determine whether a security detection score of the AIGC text to be detected can be obtained according to the first score and the second score; A first security detection score determination module, configured to, if not, determine the security detection score of the AIGC text to be detected based on the first score and the second score; A second security detection score determination module is used for, if yes, inputting the AIGC text to be detected into a pre-trained abnormal text detection model for abnormal text detection to obtain a third score; and inputting the AIGC text to be detected into a pre-trained emotion classification detection model to obtain the emotion category of the AIGC to be detected and a fourth score; obtaining the security detection score of the AIGC text to be detected according to the third score, the emotion category of the AIGC to be detected and the fourth score; wherein the third score is used to indicate the confidence that the AIGC text to be detected is an abnormal text, and the fourth score is used to indicate the confidence that the AIGC text to be detected is the emotion category; The security detection result determination module is used to compare the security detection score of the AIGC text to be detected with a preset threshold to obtain the security detection result of the AIGC text to be detected.

8. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of claims 1-6 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images