AIGC security auditing method and device, storage medium and electronic device

Abstract

The invention discloses an AIGC security auditing method and device, a storage medium and an electronic device, and relates to the technical field of smart home, and the AIGC security auditing method comprises the steps: detecting sensitive words of user question content through a user question auditing engine; if the detected sensitive word reaches a first preset level, performing preset generalization detection on the user question content; if the violation problem is not identified after the preset generalization detection, inputting the questioning content of the user into a preset AIGC model; and performing security auditing on the generated content of the preset AIGC model through a generated content security auditing engine, and feeding back answer content to the client according to a security auditing result. According to the technical scheme, the correctness of the AIGC generation content is ensured, and potential safety hazards are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of smart home, in particular, to an AIGC security auditing method and device, a storage medium and an electronic device. BACKGROUND

[0002] At present, many domestic and foreign enterprises have invested in the research and application of artificial intelligence generated content (AIGC) technology. The text content generated by AIGC is widely used in many fields such as knowledge Q&A, social media, news media, and advertising marketing.

[0003] However, the risks and problems of the content generated by AIGC exist. The training data of AIGC comes from a large amount of text, image and other materials. The generated content may involve infringement of original works and disclosure of commercial secrets. AIGC may generate some content that violates regulations and ethics, such as politics, violence, pornography, discrimination, and hate speech. The content generated by generative artificial intelligence should be true and accurate, and measures should be taken to prevent the generation of false information. During the training process of AIGC, if the user's personal information is used, it may also lead to user privacy leakage.

[0004] The existing technology lacks an effective auditing mechanism for the questions and answers in the use process of AIGC, and often only based on simple sensitive word matching. For those questions that hit the sensitive words but had no violation intention, the existing technology is difficult to identify, which may lead to missed or false reports. SUMMARY

[0005] The embodiments of the present application provide an AIGC security auditing method, device, storage medium and electronic device, which aims to ensure the correctness of the AIGC generated content and reduce security risks.

[0006] According to an aspect of an embodiment of the present application, an AIGC security auditing method is provided, comprising:

[0007] detecting the sensitive words of the user question content through a user question auditing engine;

[0008] if the detected sensitive words reach a first preset level, performing a preset generalization detection on the user question content;

[0009] if no violation problem is identified after the preset generalization detection, inputting the user question content to a preset AIGC model;

[0010] performing security auditing on the generated content of the preset AIGC model through a generated content security auditing engine, and feeding back the answer content to the client according to the security auditing result.

[0011] According to an implementation manner of the embodiment of the present application, the sensitive word detection on the user question content by the user question auditing engine comprises:

[0012] performing sensitive word detection on the user question content based on a preset sensitive word database; wherein the preset sensitive word database comprises sensitive words and their levels;

[0013] determining the sensitive words contained in the user question content and the levels of the sensitive words.

[0014] According to an implementation manner of the embodiment of the present application, before the sensitive word detection on the user question content based on the preset sensitive word database, the method further comprises:

[0015] constructing the preset sensitive word database, and including sensitive words in the preset sensitive word database;

[0016] in response to a user labeling operation, labeling the level corresponding to the sensitive word in the preset sensitive word database.

[0017] According to an implementation manner of the embodiment of the present application, after the determination of the sensitive words contained in the user question content and the levels of the sensitive words, the method further comprises:

[0018] if the detected sensitive words reach a second preset level, determining the user question content as illegal content; wherein the first preset level is lower than the second preset level;

[0019] feeding back reply content to the client according to a preset illegal question reply mode.

[0020] According to an implementation manner of the embodiment of the present application, the sensitive word detection on the user question content based on the preset sensitive word database comprises:

[0021] performing at least one of homonym matching, homograph matching and interval word matching on the word segmentation in the user question content based on the preset sensitive word database, so as to detect the sensitive words contained in the user question content.

[0022] According to an implementation manner of the embodiment of the present application, the preset generalization detection on the user question content comprises:

[0023] performing attack instruction generalization detection on the user question content, wherein based on an instruction attack feature library, it is identified whether the user question content contains content matching specific code fragments and operation steps in the instruction attack feature library;

[0024] and / or,

[0025] The user question content is subjected to illegal question generalization detection, wherein, based on an illegal question feature library, it is determined whether illegal content exists according to the syntax structure, keyword combination and context information of the user question content.

[0026] According to an implementation manner of an embodiment of the present application, the security audit of the generated content of the preset AIGC model by the generated content security audit engine includes:

[0027] The generated content is scanned by the generated content security audit engine, at least one of content information, commercial secrets and personal information is identified, and it is determined whether abnormal content exists.

[0028] According to an implementation manner of an embodiment of the present application, the feedback of the answer content to the client according to the security audit result includes:

[0029] If abnormal content exists, the reply content is fed back to the client according to a preset illegal question reply mode;

[0030] If abnormal content does not exist, the generated content is fed back to the client.

[0031] According to another aspect of the present application, an AIGC security audit device is provided, which includes:

[0032] A sensitive word detection module is configured to detect sensitive words in user question content by a user question audit engine;

[0033] A content generalization detection module is configured to perform a preset generalization detection on the user question content if the detected sensitive words reach a first preset level;

[0034] A user question content input module is configured to input the user question content to a preset AIGC model if no illegal question is identified after the preset generalization detection;

[0035] A generated content security audit module is configured to perform a security audit of the generated content of the preset AIGC model by a generated content security audit engine, and feed back answer content to a client according to a security audit result.

[0036] According to an implementation manner of an embodiment of the present application, the sensitive word detection module includes:

[0037] A content sensitive word determination unit is configured to detect sensitive words in the user question content based on a preset sensitive word database; wherein, the preset sensitive word database includes sensitive words and their levels;

[0038] The sensitive word level determination unit is configured to determine a sensitive word contained in the user query content and a level of the sensitive word.

[0039] According to an implementation manner of an embodiment of the present application, the AIGC security auditing device further includes:

[0040] The sensitive word database construction module is configured to, before performing sensitive word detection on the user query content based on a preset sensitive word database, construct the preset sensitive word database, and collect sensitive words in the preset sensitive word database.

[0041] The sensitive word level labeling module is configured to, in response to a user labeling operation, label a level corresponding to the sensitive word in the preset sensitive word database.

[0042] According to an implementation manner of an embodiment of the present application, the AIGC security auditing device further includes:

[0043] The illegal content determination module is configured to, after determining the sensitive word contained in the user query content and the level of the sensitive word, if the detected sensitive word reaches a second preset level, determine the user query content as illegal content; wherein the first preset level is lower than the second preset level.

[0044] The reply content feedback module is configured to feed back reply content to the client according to a preset illegal question reply mode.

[0045] According to an implementation manner of an embodiment of the present application, the content sensitive word determination unit is specifically configured to:

[0046] Perform at least one of homonym matching, homograph matching, and interval word matching on the word segmentation in the user query content based on the preset sensitive word database, to detect a sensitive word contained in the user query content.

[0047] According to an implementation manner of an embodiment of the present application, the content generalization detection module includes:

[0048] The attack instruction generalization detection module is configured to perform attack instruction generalization detection on the user query content, wherein, based on an instruction attack feature library, it is determined whether the user query content contains content matching a specific code fragment and operation step in the instruction attack feature library.

[0049] And / or,

[0050] The illegal question generalization detection module is configured to perform illegal question generalization detection on the user query content, wherein, based on an illegal question feature library, it is determined whether there is illegal content according to a syntax structure, a keyword combination, and context information of the user query content.

[0051] According to an implementation form of the embodiment of the application, the generated content security audit module is specifically configured to:

[0052] The generated content is scanned by the generated content security audit engine, at least one of content information, commercial secrets and personal information is identified, and it is determined whether there is abnormal content.

[0053] According to an implementation form of the embodiment of the application, the generated content security audit module is specifically configured to:

[0054] If there is abnormal content, a preset violation problem reply mode is used to feed back reply content to the client;

[0055] If there is no abnormal content, the generated content is fed back to the client.

[0056] According to another aspect of the embodiment of the application, a computer-readable storage medium is provided, the computer-readable storage medium includes a stored program, wherein the program executes the AIGC security audit method provided by any embodiment of the application when running.

[0057] According to another aspect of the embodiment of the application, an electronic device is provided, including a memory and a processor, the memory has a computer program stored therein, and the processor is configured to execute the AIGC security audit method provided by any embodiment of the application through the computer program.

[0058] The AIGC security audit method, device, storage medium and electronic device provided by the embodiment of the application determine whether the question exists violation by judging the generalization detection of the question content of the user through the pre-set sensitive word level, further perform security audit on the generated content of the AIGC model after the question audit, solve the problem that the question and answer in the use process of AIGC only based on simple sensitive word matching leads to problem missing report or false report, and realize the correctness of the generated content of AIGC and the reduction of security risks. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0061] Figure 1 is a hardware environment schematic diagram of an AIGC security auditing method according to an embodiment of the application;

[0062] Figure 2 is a flowchart of an AIGC security auditing method according to an embodiment of the application;

[0063] Figure 3 is an interaction diagram of an AIGC security auditing system and a client according to an embodiment of the application;

[0064] Figure 4 is a structural schematic diagram of an AIGC security auditing device according to an embodiment of the application;

[0065] Figure 5 is a structural schematic diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0066] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0067] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] According to an aspect of the embodiments of the present application, a method for security auditing of AIGC of an intelligent device is provided. The method for security auditing of AIGC is widely applied to smart home, smart home, smart home device ecology, intelligence house ecology, and other whole-house intelligent digital control application scenarios. Specifically, the user can ask questions in the form of voice or text, and the AI model arranged on the server generates AIGC according to the question content, and the user's question content and the generated content are audited in the process. Alternatively, in the present embodiment, the above-mentioned method for security auditing of AIGC can be applied to the hardware environment composed of terminal device 102 and server 104 as shown in Figure 1 As shown in Figure 1 The server 104 is connected with the terminal device 102 through the network, which can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal, and a database can be set on the server or independently of the server, which is used to provide data storage services for the server 104, and cloud computing and / or edge computing services can be configured on the server or independently of the server, which is used to provide data operation services for the server 104.

[0069] The above-mentioned network can include but is not limited to at least one of the following: wired network, wireless network. The above-mentioned wired network can include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above-mentioned wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 can not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart oven, smart refrigerator, smart oven, smart oven, smart washing machine, smart water heater, smart washing device, smart dishwasher, smart projection device, smart television, smart clothesline, smart curtain, smart audio and video, smart socket, smart sound, smart sound box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart sweeping robot, smart window cleaning robot, smart mopping robot, smart air purification equipment, smart steamer, smart microwave oven, smart kitchen treasure, smart purifier, smart water dispenser, smart door lock, etc.

[0070] According to an aspect of the embodiments of the present application, a method for security auditing of AIGC is provided. As shown in Figure 2 The flowchart of the method for security auditing of AIGC includes:

[0071] Step S220, detecting sensitive words in the user's question content by the user question auditing engine.

[0072] The user question review engine is responsible for coordinating and integrating various sub-detection modules to conduct a comprehensive review of user questions to determine if any violations exist. User questions can be input by the user in the client, either through text or voice. If it's voice, it can be converted into corresponding text. Figure 3 In this process, user questions are input into the user question review engine. The user question review engine includes a sensitive word detection module. This module typically requires a pre-configured sensitive word database, which can be a dictionary containing various types of sensitive words. AIGC operations administrators can maintain this database, adding, deleting, and categorizing sensitive words. Sensitive words may include those related to politics, pornography, violence, and other sensitive terms. When a user asks a question, the sensitive word detection module matches the question text against the sensitive words in the database to determine if the question contains any words from the pre-configured database.

[0073] Step S240: If the detected sensitive words reach the first preset level, perform preset generalization detection on the user's question content.

[0074] The preset sensitive word database, a dictionary containing various types of sensitive words, allows for setting sensitivity levels. These levels are divided into two categories based on sensitivity: a first preset level and a second preset level. The classification of the preset sensitive word database can be manually assigned. The first preset level is the lowest. If the sensitive word detection module matches the user's question text with sensitive words and determines that the question contains a sensitive word from the preset sensitive word database, then the level of that sensitive word in the database can be further determined. If a sensitive word in the question text is identified as belonging to the first preset level, to reduce false positives, the user's question is submitted to the corresponding generalization detection module in the user question review engine for preset generalization detection. Figure 3 In this system, the user question review engine includes an attack command generalization detection module and / or a violation question generalization detection module. The attack command generalization detection module identifies malicious commands based on an attack command feature library, while the violation question generalization detection module performs generalization identification on user questions based on a violation question feature library. Both modules further analyze the question text of the user questions, performing detection through semantic understanding and feature generalization matching.

[0075] Step S260: If no violation is identified after the preset generalization detection, the user's question content is input into the preset AIGC model.

[0076] Among them, if no sensitive words are found after sensitive word detection, or if the sensitive word level is the first preset level and no violations are found after preset generalization detection, the user's question is identified as a normal question, such as... Figure 3 In this process, the user's question will be input into the preset AIGC model. The AIGC model is obtained by training a deep neural network in advance with training materials, and it can generate corresponding content as an answer to the user's question.

[0077] Step S280: Perform security audit on the generated content of the preset AIGC model through the generated content security audit engine, and provide feedback to the client based on the security audit results.

[0078] The system generates content from a pre-set AIGC model, which then undergoes a security audit. This engine comprehensively reviews the generated content to ensure it complies with relevant security regulations and protects the rights of the platform, users, and stakeholders. If the audit results indicate that the generated content meets compliance requirements and protects the rights of the platform, users, and stakeholders, the content is returned to the client for display, such as by showing text on a screen or playing corresponding audio through a speaker. If the audit results indicate that the generated content fails to meet compliance requirements or cannot protect the rights of the platform, users, and stakeholders, the system rejects the response, which can be sent to the client according to pre-set rejection protocols.

[0079] This application combines a multimodal user question review engine and a generated content security review engine, and can pre-label sensitive words to improve review efficiency and accuracy, while effectively reducing false positives. It comprehensively and holistically reviews user questions and generated content through multimodal fusion, promptly identifying potential violations and improving review accuracy.

[0080] According to one implementation of this application, the step of detecting sensitive words in user question content through a user question review engine includes:

[0081] Based on a preset sensitive word database, sensitive words are detected in the user's question content; wherein, the preset sensitive word database includes sensitive words and their levels;

[0082] Determine the sensitive words contained in the user's question and the level of those sensitive words.

[0083] The user query content can be preprocessed, including text normalization and / or special character removal. Normalization includes text normalization, number normalization, letter normalization, and Unicode normalization. Special character removal identifies and removes various special characters from the user query content. Then, sensitive words are extracted from a pre-set sensitive word database and compared with the user query content. If a sensitive word is matched, the specific sensitive word(s) in the user query content are determined, and their corresponding level is retrieved from the pre-set sensitive word database. Normalization helps reduce data noise in subsequent generalization detection, improving the generalization ability of the detection module. Unicode normalization unifies different characters with the same meaning into a unified character, eliminating ambiguity and confusion caused by the same text appearing in different forms.

[0084] According to one implementation of this application, before performing sensitive word detection on the user's question content based on a preset sensitive word database, the method further includes:

[0085] Construct the preset sensitive word database, and include sensitive words in the preset sensitive word database;

[0086] In response to the user's annotation operation, the level corresponding to the sensitive word in the preset sensitive word database is annotated.

[0087] The preset sensitive word database is built based on pre-collected sensitive words and can be categorized and stored in separate tables according to the domain of the sensitive words. The sensitivity word classification in the preset database can be manually marked by administrators. When a user asks a question, the detection engine matches the question text against these sensitive words. If the user's question contains any one or more of these sensitive words, and if the matched sensitive word is at the second preset level, the user's question will be marked as a violation. If the matched sensitive word is at the first preset level, to reduce false positives, it will be further analyzed and detected by the command attack generalization detection module and the violation question generalization detection module.

[0088] According to one implementation of this application, after determining the sensitive words contained in the user's question content and the level of the sensitive words, the method further includes:

[0089] If the detected sensitive words reach the second preset level, the user's question content is determined to be illegal content; wherein, the first preset level is lower than the second preset level; the second preset level can determine that the user's question content is illegal content. In this way, on the one hand, false alarms can be reduced for sensitive words of the first preset level, and on the other hand, generalized detection is not required for sensitive words of the second preset level, thereby improving the efficiency of the user question review engine.

[0090] The system sends a response to the client according to the preset response method for violation issues.

[0091] According to one implementation of this application, the step of performing sensitive word detection on the user's question content based on a preset sensitive word database includes:

[0092] Based on the preset sensitive word database, at least one of homophone matching, similar-looking word matching, and spaced word matching is performed on the word segmentation in the user's question content to detect sensitive words contained in the user's question content.

[0093] When users input their questions, whether directly into text or voice, input deviations may occur, such as due to the input method or speech-to-text conversion resulting in homophones, similar-looking words, or intervening words. The solution involves segmenting the user-input question text into words and then identifying sensitive words within the segmentation, specifically homophones, similar-looking words, and intervening words. Homophones are words with the same or similar pronunciation to the sensitive word; a pre-set sensitive word database can include homophones corresponding to sensitive words. Similar-looking words are text with similar character structures. Intervening words may be characters intentionally added by the user to the sensitive word list without substantial meaning; these can be detected by performing single-character matching between the question text and the sensitive words in the pre-set sensitive word database.

[0094] According to one implementation of an embodiment of this application, the step of performing preset generalization detection on the user's question content includes:

[0095] The user's question content is subjected to attack instruction generalization detection, wherein, based on the instruction attack feature library, it is identified whether the user's question content matches specific code snippets and operation steps in the instruction attack feature library;

[0096] And / or,

[0097] The user's question content is subjected to generalized detection of violations. Specifically, based on a violation question feature library, the presence of violation content is determined according to the grammatical structure, keyword combination, and contextual information of the user's question content.

[0098] Among them, such as Figure 3In this application, the attack command generalization detection module and the violation question generalization detection module can be set up side-by-side in the user question review engine. This means that after sensitive word detection, attack command generalization detection and violation question generalization detection can be executed in parallel, improving detection efficiency. Based on a violation question feature library, the system performs generalized identification of user question content, identifying question expressions with malicious intent. For example, by analyzing the grammatical structure, keyword combinations, contextual information, and the violation question feature library, it determines whether potential violations exist. Specifically targeting command-style questions that may cause harm to the system, platform, or other users, the system identifies offensive commands based on the command attack feature library, such as commands related to hacking techniques or maliciously damaging platform functions. This may involve detecting specific code snippets or operation steps. The system can refuse to answer questions that fail the detection. When faced with offensive command-style questions, traditional detection methods may be ineffective due to a lack of ability to identify commands related to specific attack types. The attack command detection module in this application addresses this problem by deeply understanding various attack methods and converting them into detection rules, accurately identifying attack commands and ensuring platform security.

[0099] According to one implementation of this application, the step of performing security audits on the generated content of the preset AIGC model through a content security audit engine includes:

[0100] The generated content is scanned by the generated content security audit engine to identify at least one of the content information, trade secrets, and personal information to determine whether there is any abnormal content.

[0101] Among them, such as Figure 3The generated content security review engine can include three detection modules. The first module detects the content of the system-generated text using various technologies. For the text content, it may use generalization and natural language processing techniques to check whether the grammatical structure and semantic logic are reasonable and normal, and whether they violate platform rules or laws and regulations. For example, it checks whether the generated text content contains ambiguous character combinations that may imply malicious programs, or whether the generated content contains violent, pornographic, or other harmful information. The second module detects trade secrets based on a trade secret knowledge base or specific trade secret identification algorithms. The detection engine analyzes whether the content is similar to known trade secret information. This includes comparing technical information, business models, customer data, etc., in the text. If the generated content involves unauthorized trade secret information, the detection will be considered abnormal. The third module detects personal information, which can identify various forms of personal information, including but not limited to names, ID card numbers, phone numbers, bank card numbers, and home addresses. It scans the generated content using regular expression matching, semantic analysis, and other technologies. It judges whether the appearance of this personal information conforms to legal and compliant scenarios according to preset rules. If the system generates a large number of user phone numbers in a public content generation scenario that does not require user personal information, it will be considered abnormal. Existing detection technologies for generated content have shortcomings in checking content quality and security. For example, when generating text content, they may not be able to effectively detect grammatical errors, semantic ambiguity, or the presence of malicious code. System-generated content may unintentionally disclose trade secrets, and existing technologies lack effective safeguards against this. The trade secret violation detection engine in this application solves this problem by establishing a trade secret knowledge base and using efficient comparison algorithms, preventing the leakage of trade secrets due to generated content and protecting corporate interests.

[0102] According to one implementation of an embodiment of this application, the step of feeding back the response content to the client based on the security audit result includes:

[0103] If any abnormal content is found, a response will be sent to the client according to the preset response method for violation issues;

[0104] If no abnormal content is found, the generated content is returned to the client.

[0105] For generated content without any abnormal content, the generated content can be fed back to the client for display to the user. For generated content with abnormal content, a response method for violation issues can be pre-set. For example, the corresponding response text can be set according to the nature of the violation issue as the response content to the client.

[0106] This application's embodiments combine multiple different modules (such as sensitive word detection, generalized review of violations, and attack command detection) with manual data annotation to form a comprehensive user question review system. This multi-dimensional review approach can accurately capture various forms of violations, rather than relying on a single detection method.

[0107] According to another aspect of the embodiments of this application, a security auditing device for AIGC is provided, such as... Figure 4 The diagram shown illustrates the structure of the AIGC security audit device, which includes:

[0108] The sensitive word detection module 402 is used to detect sensitive words in user question content through the user question review engine;

[0109] The content generalization detection module 404 is used to perform preset generalization detection on the user's question content if the detected sensitive words reach the first preset level.

[0110] The user question input module 406 is used to input the user question into the preset AIGC model if no violation is identified after the preset generalization detection.

[0111] The content security review module 408 is used to review the content generated by the preset AIGC model through the content security review engine, and to provide feedback to the client based on the security review results.

[0112] According to one implementation of an embodiment of this application, the sensitive word detection module includes:

[0113] The content sensitive word determination unit is used to perform sensitive word detection on the user's question content based on a preset sensitive word database; wherein, the preset sensitive word database includes sensitive words and their levels;

[0114] A sensitive word level determination unit is used to determine the sensitive words contained in the user's question content and the level of the sensitive words.

[0115] According to one implementation of an embodiment of this application, the AIGC security audit device further includes:

[0116] The sensitive word database construction module is used to construct the preset sensitive word database before performing sensitive word detection on the user's question content based on the preset sensitive word database, and to include sensitive words in the preset sensitive word database;

[0117] The sensitive word level annotation module is used to respond to user annotation operations and annotate the level corresponding to the sensitive words in the preset sensitive word database.

[0118] According to one implementation of an embodiment of this application, the AIGC security audit device further includes:

[0119] The illegal content determination module, after determining the sensitive words contained in the user's question content and the level of the sensitive words, further includes: if the detected sensitive words reach a second preset level, determining the user's question content as illegal content; wherein, the first preset level is lower than the second preset level;

[0120] The response content feedback module is used to send response content to the client according to the preset response method for violation issues.

[0121] According to one implementation of an embodiment of this application, the content sensitive word determination unit is specifically used for:

[0122] Based on the preset sensitive word database, at least one of homophone matching, similar-looking word matching, and spaced word matching is performed on the word segmentation in the user's question content to detect sensitive words contained in the user's question content.

[0123] According to one implementation of an embodiment of this application, the content generalization detection module includes:

[0124] The attack instruction generalization detection module is used to perform attack instruction generalization detection on the user's question content. Specifically, based on the instruction attack feature library, it identifies whether the user's question content matches specific code snippets and operation steps in the instruction attack feature library.

[0125] And / or,

[0126] The violation question generalization detection module is used to perform violation question generalization detection on the user's question content. It determines whether there is any violation content based on the violation question feature library, according to the grammatical structure, keyword combination and context information of the user's question content.

[0127] According to one implementation of an embodiment of this application, the content security review module is specifically used for:

[0128] The generated content is scanned by the generated content security audit engine to identify at least one of the content information, trade secrets, and personal information to determine whether there is any abnormal content.

[0129] According to one implementation of an embodiment of this application, the content security review module is specifically used for:

[0130] If any abnormal content is found, a response will be sent to the client according to the preset response method for violation issues;

[0131] If no abnormal content is found, the generated content is returned to the client.

[0132] According to another aspect of the embodiments of this application, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the electronic device can be one or more. Figure 5 Taking a processor 510 as an example; the processor 510, memory 520, input device 530, and output device 540 in the electronic device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0133] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the AIGC security auditing method in this embodiment (e.g., the sensitive word detection module 402, content generalization detection module 404, user question content input module 406, and generated content security auditing module 408 in the AIGC security auditing device). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 520, thereby implementing the aforementioned AIGC security auditing method.

[0134] The memory 520 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 520 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include memory remotely located relative to the processor 510, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] Input device 530 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the electronic device. It can also be a camera for acquiring images and a sound pickup device for acquiring audio data. Output device 540 may include display devices such as a screen, and audio devices such as a speaker. It should be noted that the specific composition of input device 530 and output device 540 can be set according to actual conditions. Processor 510 executes various functional applications and data processing of the electronic device by running software programs, instructions, and modules stored in memory 520.

[0136] According to another aspect of the embodiments of this application, the embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein the program executes the following AIGC security audit method when running:

[0137] The user question review engine detects sensitive words in user questions.

[0138] If the detected sensitive words reach the first preset level, a preset generalization detection is performed on the user's question content;

[0139] If no violation is identified after the preset generalization detection, the user's question will be input into the preset AIGC model;

[0140] The generated content of the preset AIGC model is subjected to security review by the content security review engine, and the response content is fed back to the client based on the security review results.

[0141] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the method operations described above, but can also perform related operations in the AIGC security audit method provided in any embodiment of the present invention.

[0142] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0143] It is worth noting that in the embodiments of the AIGC security audit device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0144] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A security auditing method for AIGC, characterized in that, include: The user question review engine detects sensitive words in user questions. If the detected sensitive words reach the first preset level, a preset generalization detection is performed on the user's question content; If no violation is identified after the preset generalization detection, the user's question will be input into the preset AIGC model; The generated content of the preset AIGC model is subjected to security review by the content security review engine, and the response content is fed back to the client based on the security review results.

2. The AIGC security audit method according to claim 1, characterized in that, The process of detecting sensitive words in user-submitted questions using a user question review engine includes: Based on a preset sensitive word database, sensitive words are detected in the user's question content; wherein, the preset sensitive word database includes sensitive words and their levels; Determine the sensitive words contained in the user's question and the level of those sensitive words.

3. The AIGC security audit method according to claim 2, characterized in that, After determining the sensitive words contained in the user's question and their level, the method further includes: If the detected sensitive words reach the second preset level, the user's question content will be determined as illegal content; wherein, the first preset level is lower than the second preset level; The system sends a response to the client according to the preset response method for violation issues.

4. The AIGC security audit method according to claim 2, characterized in that, The process of detecting sensitive words in the user's question based on a preset sensitive word database includes: Based on the preset sensitive word database, at least one of homophone matching, similar-looking word matching, and spaced word matching is performed on the word segmentation in the user's question content to detect sensitive words contained in the user's question content.

5. The AIGC security audit method according to claim 1, characterized in that, The preset generalization detection of the user's question content includes: The user's question content is subjected to attack instruction generalization detection, wherein, based on the instruction attack feature library, it is identified whether the user's question content matches specific code snippets and operation steps in the instruction attack feature library; And / or, The user's question content is subjected to generalized detection of violations. Specifically, based on a violation feature library, the presence of violation content is determined according to the grammatical structure, keyword combination, and contextual information of the user's question content.

6. The AIGC security audit method according to claim 5, characterized in that, The step of performing security audits on the content generated by the preset AIGC model through a content security audit engine includes: The generated content is scanned by the generated content security audit engine to identify at least one of the content information, trade secrets, and personal information to determine whether there is any abnormal content.

7. The AIGC security audit method according to claim 6, characterized in that, The step of providing feedback to the client based on the security audit results includes: If any abnormal content is found, a response will be sent to the client according to the preset response method for violation issues; If no abnormal content is found, the generated content is returned to the client.

8. A security auditing device for AIGC, characterized in that, include: The sensitive word detection module is used to detect sensitive words in user questions through the user question review engine; The content generalization detection module is used to perform preset generalization detection on the user's question content if the detected sensitive words reach the first preset level. The user question input module is used to input the user question into the preset AIGC model if no violation is identified after the preset generalization detection. The content security review module is used to review the content generated by the preset AIGC model through the content security review engine, and to provide feedback to the client based on the security review results.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the AIGC security auditing method according to any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the AIGC security audit method according to any one of claims 1 to 7 through the computer program.

Images