Security risk detection method and device for text graph large model generation content

Abstract

The invention provides a security risk detection method and device for text graph large model generation content, and the method comprises the steps: selecting a test category according to a test parameter based on a preset test database, and matching a corresponding data set from the test database according to the test category; inputting the keywords in the data set into the tested large model in batches according to the format of forward cue words; performing content security detection on a corresponding picture or video image generated by the detected large model according to the input cue word; performing watermark identification detection on a corresponding picture or video image generated by the detected large model according to the input cue word; and analyzing to obtain a detection report according to the content security detection result and the watermark identification detection result. According to the test database, the risk of the large model system is externally assessed based on the characteristics of the input prompt word, the content annotation and the output video content, and the accuracy and effectiveness of the detection and assessment method are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of picture and video image data processing, and in particular to a method and device for detecting security risks of content generated by a large text-based image model. Background Art

[0002] As a representative of generative artificial intelligence technology, text-to-image generative models have shown broad application prospects in many fields such as art creation, advertising design, and game development due to their powerful image generation capabilities.

[0003] However, with the widespread application of text-based image technology, its potential security risks have become increasingly prominent. Due to the lack of systematic supervision of such models, there is a possibility of generating inappropriate content, the spread of which will not only have a negative impact on society, but may also cause legal and moral disputes.

[0004] In response to the security risks of content generated by large text-based graph models, existing detection technologies mainly include keyword blocking, image recognition, and open source platform management technologies. However, in actual applications, image recognition technology is difficult to accurately assess the security of images generated by text-based graph models due to their complex and changeable content. At the same time, keyword blocking technology is prone to becoming outdated and requires constant updates and iterations, and there is a risk of being bypassed. In addition, with the increasing number of large text-based graph models, the cost and time overhead of manual functional testing is huge, making it difficult to meet actual needs. As model capabilities continue to increase, security detection mechanisms also need to be continuously updated to adapt to new threat forms. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for detecting security risks of content generated by a large model of a cultural graph, which solves the problem of how to improve the accuracy and effectiveness of security detection of the large model of cultural graphs.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for detecting security risks of content generated by a large text-based graph model is provided, comprising:

[0008] Based on a preset test database, a test category is selected according to test parameters, and a corresponding data set is matched from the test database according to the test category; keywords in the data set are batch-inputted into the large model under test in the format of positive prompt words;

[0009] Performing content security detection on the corresponding picture or video image generated by the tested large model according to the input prompt word;

[0010] Performing watermark detection on the corresponding picture or video image generated by the large model under test according to the input prompt word;

[0011] A test report is obtained by analyzing the content security test result and the watermark mark test result.

[0012] In a first possible implementation manner of the first aspect, the test database is a key risk vocabulary library constructed according to specific standards or specifications.

[0013] In a second possible implementation manner of the first aspect, the content security detection includes:

[0014] Use rule engines and machine learning algorithms to perform automated detection, extract the content of pictures or videos, reversely infer the reverse prompt words of the pictures or videos, and judge and mark the illegal content based on the reverse prompt words;

[0015] Manually review and confirm the illegal content marked by automated detection to obtain content risk assessment results.

[0016] In a third possible implementation manner of the first aspect, the watermark identification detection includes:

[0017] By using OCR technology, the corresponding picture or video image generated by the large model under test according to the input prompt word is subjected to display watermark detection.

[0018] Based on any possible implementation of the first aspect, in a fourth possible implementation of the first aspect, the test database also includes a vocabulary of words that should be rejected; according to the rejection rate of the prompt words in the vocabulary of words that should be rejected corresponding to the model under test, combined with the content security detection results and the watermark detection results, a detection report is analyzed to obtain.

[0019] In a second aspect, a security risk detection device for content generated by a large text-based graph model is provided, comprising:

[0020] The test data module is used to select a test category based on a preset test database according to test parameters, match a corresponding data set from the test database according to the test category; and batch input the keywords in the data set into the large model under test in the format of positive prompt words;

[0021] A content security detection module, used to perform content security detection on the corresponding picture or video image generated by the large model under test according to the input prompt word;

[0022] A logo detection module, used for performing watermark logo detection on the corresponding picture or video image generated by the large model under test according to the input prompt word;

[0023] The result analysis module is used to analyze and obtain a detection report according to the content security detection result and the watermark detection result.

[0024] In a first possible implementation manner of the second aspect, the test database is a key risk vocabulary library constructed according to specific standards or specifications.

[0025] In a second possible implementation manner of the second aspect, the content security detection module is specifically configured to:

[0026] Use rule engines and machine learning algorithms to perform automated detection, extract the content of pictures or videos, reversely infer the reverse prompt words of the pictures or videos, and judge and mark the illegal content based on the reverse prompt words;

[0027] Manually review and confirm the illegal content marked by automated detection to obtain content risk assessment results;

[0028] The identification detection module is specifically used for:

[0029] By using OCR technology, the corresponding picture or video image generated by the large model under test according to the input prompt word is subjected to display watermark detection.

[0030] According to a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for detecting security risks of content generated by a large model of a literary graph as described in the first aspect are implemented.

[0031] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the security risk detection method for generating content for the large model of the culture graph as described in the first aspect are implemented.

[0032] This application conducts an external assessment of the risks of the large model system based on the characteristics of the input prompt words, content annotations, and output video content according to the test database, thereby improving the accuracy and effectiveness of the detection and assessment method. This application is not limited to a specific large model, and will greatly accelerate the detection efficiency of content security risks for the Vincent graph large model, and proposes a Vincent graph large model content security risk detection solution with simple operation, clear process, and rapid detection. At the same time, this application contributes to the development of Vincent graph large model ecological technology and provides a reference for the regulatory development of other large model types. It not only helps to improve the security and compliance of generative large models, but also protects user rights, promotes technological innovation, and supports traceable management, which is of great value to building a healthy network ecology. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of an AIGC security assessment system deployment application provided in an embodiment of the present application;

[0034] Figure 2 A schematic flow chart of another method for detecting security risks of content generated by a large model of a text graph provided in an embodiment of the present application;

[0035] Figure 3 A schematic structural diagram of a security risk detection device for content generated by a large model of a text graph provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.

[0038] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0039] The description of the method flow in the specification of the present application and the steps of the flowchart in the drawings of the present specification do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0040] In conjunction with the accompanying drawings and preferred embodiments, the security risk detection method, device, equipment and medium for content generated by the large model of the text graph provided in the embodiments of the present application are described in detail as follows.

[0041] First, the application scenario of the security risk detection method for content generated by a large model of a text graph in an embodiment of the present application is described in detail.

[0042] In recent years, generative artificial intelligence technology has developed rapidly, and has the ability to creatively generate new content, which has promoted its widespread application. While generative artificial intelligence brings new development opportunities, it also creates new security risks.

[0043] In the process of large model generation, there are risks in data, annotation, content, model, strategy, intellectual property, etc. Models are the core of AIGC video generation, and imperfections or errors in the models may bring many risks; data is the basis for training AIGC models, and data quality and security directly affect the quality and security of generated content; data annotation is an important part of training models, and the quality of annotation directly affects the performance and output results of the model; AIGC-generated content may be used maliciously to create false information or deep fake videos (Deepfake) to mislead the public or adversely affect specific individuals and groups; the development of AIGC technology may exceed the scope of existing laws and regulations, leading to regulatory blind spots and legal risks.

[0044] Among the existing technologies for content security detection of large text-based image models: image recognition technology is difficult, and for most images generated by text-based image models, their content may be messy or even useless. Therefore, it is not easy to identify and evaluate through image recognition.

[0045] The image generation capability of the Wenshengtu model itself is also constantly developing and strengthening. Therefore, the original keyword blocking is easy to become outdated and needs to be updated and iterated quickly and continuously. In addition, the large model is also vulnerable to attacks, and the keyword blocking is easily bypassed.

[0046] According to relevant statistics, since the number of large Wenshengtu models has already exceeded 4 million, the platform's use of manual functional testing is too time-consuming and costly.

[0047] In view of the limitations and shortcomings of the above-mentioned technologies, and taking into account the large number of large models of Wensheng graphs, the difficulty in obtaining codes, and the large amount of test data, this application provides a security risk detection method for the content generated by large models of Wensheng graphs, aiming to conduct an external assessment of the risks of large model systems and improve the accuracy and effectiveness of the assessment method. Considering that the training and reasoning of generative large models are affected by many factors, including but not limited to raw data, annotation methods, model structure, computing power platform, and hyperparameter settings, the accuracy and effectiveness of security assessments of generative large models based on training data and the model itself are limited. Therefore, this application regards the large model as a black box and conducts security assessments based on the characteristics of input prompt words, content annotations, and output video content.

[0048] Specifically, see Figure 1When generative artificial intelligence operators deploy AIGC-related applications, they usually build a two-way intermediate layer between the application and the large model. Through the technical measures, the adverse effects of AIGC technology can be effectively alleviated, such as Figure 1 As shown. With the help of filtering and protection in the bidirectional middle layer, the risk control of the generative large model can be achieved: the filtering function is to use the audit classifier to filter out harmful content in the supervision channel of the large model content output. The protection function is to audit the prompt word content of the input model in the large model prompt word supervision channel. Many of the risk protection functions of the large model itself are realized based on the above filtering and protection functions, and the security assessment of the large model based on external means can be performed based on the input and output ends.

[0049] In the evaluation of the protection measures of the big model, the risk assessment of the big model itself generated by video and image can be technically done by establishing a risk warning word library of harmful content, using the risk warning word library as a prompt word, inputting the big model to be evaluated for security, and using the security risk detection system of this application to induce and detect whether it generates corresponding bad content. Alternatively, for specific prompt words, a word library that should be rejected is constructed, and the rejection rate of the big model is evaluated.

[0050] In the evaluation of filtering methods of large models, annotation feature recognition and compliance detection and content compliance detection can be used. Specifically, content compliance detection is to analyze the multi-modal features (such as visual elements, voice, text, etc.) used for video generation for security assessment. Logo compliance detection is to identify the logo information added to the video screen, and perform compliance detection and verification on it; watermark logo information in videos generated by artificial intelligence can be detected through image recognition, metadata analysis, text analysis and other means.

[0051] See also Figure 2 , the embodiment of the present application provides a method for detecting security risks of content generated by a large model of a text graph, such as Figure 2 As shown, the recommended method of the embodiment of the present application includes the following steps:

[0052] Step S1, based on a preset test database, select a test category according to test parameters, match a corresponding data set from the test database according to the test category; and batch input the keywords in the data set into the large model under test in the format of positive prompt words.

[0053] The test database is a key risk word library built according to specific standards or specifications. For example, in this embodiment, the test database is based on the "Basic Requirements for the Safety of Generative Artificial Intelligence Large Models"

[0054] The key risk vocabulary is constructed based on the 5 categories and 31 types of safety risks specified in the (TC260 Basic Safety Requirements for Large Models).

[0055] Step S2, performing content security detection on the corresponding picture or video image generated by the large model under test according to the input prompt word.

[0056] After the large model under test generates the corresponding picture or video image according to the input prompt word, it will be subjected to content security detection to determine whether there is any illegal, sensitive or inappropriate content. Specifically, content security detection includes:

[0057] Step S201, use the rule engine and machine learning algorithm to perform automated detection, extract the content of the picture or video image, reversely infer the reverse prompt words of the picture or video image content, and judge and mark the illegal content contained according to the reverse prompt words.

[0058] In this step, you can use the automatic image reverse reasoning tool to perform batch reverse reasoning tests on the large amount of image data output in the previous stage. The reverse reasoning test will reversely infer text data based on the image data, and then compare the text data with the test database to derive the risk rating of the image based on the hit rate.

[0059] Step S202: manually review and confirm the illegal content marked by the automatic detection to obtain the content risk assessment result.

[0060] Step S3, performing watermark detection on the corresponding picture or video image generated by the large model under test according to the input prompt word.

[0061] After the large model under test generates the corresponding picture or video image according to the input prompt word, the generated corresponding picture or video image should be accompanied by identification information that complies with relevant regulations and standards, and watermark detection should be performed at the same time as the content security detection. Specifically, OCR technology can be used to perform display watermark detection on the picture or video image to ensure that the video image generated by the model complies with relevant regulations and standards, and is traceable, that is, to detect the traceability of the watermark.

[0062] Step S4: Analyze and obtain a test report based on the content security test results and the watermark mark test results.

[0063] Based on the content risk assessment results and watermark detection results, an analysis is conducted to obtain a detection report, which records the detection process, results and suggestions in detail.

[0064] In some possible implementations, the test database also includes a database of words that should be rejected; and a test report is obtained by analyzing the rejection rate of prompt words in the database of words that should be rejected corresponding to the model under test, combined with content security test results and watermark detection results.

[0065] Based on the above technical solution, this application conducts an external assessment of the risks of the large model system based on the characteristics of the input prompt words, content annotations, and output video content according to the test database, thereby improving the accuracy and effectiveness of the detection and evaluation method. This application is not limited to a specific large model, and will greatly accelerate the detection efficiency of content security risks for the Vincent graph large model, and proposes a Vincent graph large model content security risk detection solution with simple operation, clear process, and rapid detection. At the same time, this application contributes to the development of Vincent graph large model ecological technology and provides a reference for the regulatory development of other large model types. It not only helps to improve the security and compliance of generative large models, but also protects user rights, promotes technological innovation, and supports traceable management, which is of great value to building a healthy network ecology.

[0066] See also Figure 3 Corresponding to the above-mentioned security risk detection method embodiment for content generated by a large model of a text graph, the embodiment of the present application provides a security risk detection device for content generated by a large model of a text graph, the detection device comprising:

[0067] The test data module 1001 is used to select a test category according to test parameters based on a preset test database, match a corresponding data set from the test database according to the test category; and batch input the keywords in the data set into the large model under test in the format of positive prompt words;

[0068] A content security detection module 1002 is used to perform content security detection on the corresponding picture or video image generated by the large model under test according to the input prompt word;

[0069] The identification detection module 1003 is used to perform watermark identification detection on the corresponding picture or video image generated by the large model under test according to the input prompt word;

[0070] The result analysis module 1004 is used to analyze and obtain a detection report according to the content security detection result and the watermark detection result.

[0071] Furthermore, the test database is a key risk vocabulary constructed according to specific standards or specifications.

[0072] Furthermore, the content security detection module is specifically used to:

[0073] Use rule engines and machine learning algorithms to perform automated detection, extract the content of pictures or videos, reversely infer the reverse prompt words of the pictures or videos, and judge and mark the illegal content based on the reverse prompt words;

[0074] Manually review and confirm the illegal content marked by automated detection to obtain content risk assessment results;

[0075] Furthermore, the identification detection module is specifically used to:

[0076] By using OCR technology, the corresponding picture or video image generated by the large model under test according to the input prompt word is subjected to display watermark detection.

[0077] The above-mentioned security risk detection device for generating content based on a large model of a text image implements the steps and various processes of the above-mentioned security risk detection method embodiment for generating content based on a large model of a text image, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0078] See also Figure 4 Corresponding to the above-mentioned embodiment of the security risk detection method for generating content by a large model of a Wensheng graph, an embodiment of the present application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the above-mentioned embodiment of the security risk detection method for generating content by a large model of a Wensheng graph and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0079] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0080] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0081] Corresponding to the above-mentioned embodiment of the security risk detection method for generating content by using a large model of a Wensheng graph, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps and various processes of the above-mentioned embodiment of the security risk detection method for generating content by using a large model of a Wensheng graph are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0082] The processor is the processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0083] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0085] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A security risk detection method for content generated by a large model of a text graph, characterized in that: include: Based on a preset test database, selecting a test category according to test parameters, and matching a corresponding data set from the test database according to the test category; Input the keywords in the data set into the tested large model in batches according to the format of positive prompt words; Performing content security detection on the corresponding picture or video image generated by the tested large model according to the input prompt word; Performing watermark detection on the corresponding picture or video image generated by the large model under test according to the input prompt word; A test report is obtained by analyzing the content security test result and the watermark mark test result.

2. The method for detecting security risks of content generated by a large model of a text graph according to claim 1 is characterized in that: The test database is a key risk word library constructed according to specific standards or specifications.

3. The method for detecting security risks of content generated by a large model of a text graph according to claim 1 is characterized in that: The content security detection includes: Use rule engines and machine learning algorithms to perform automated detection, extract the content of pictures or videos, reversely infer the reverse prompt words of the pictures or videos, and judge and mark the illegal content based on the reverse prompt words; Manually review and confirm the illegal content marked by automated detection to obtain content risk assessment results.

4. The method for detecting security risks of content generated by a large model of a text graph according to claim 1 is characterized in that: The watermark detection includes: By using OCR technology, the corresponding picture or video image generated by the large model under test according to the input prompt word is subjected to display watermark detection.

5. The method for detecting security risks of content generated by a large model of a text graph according to any one of claims 1 to 4, characterized in that: The test database also includes a database of words that should be rejected; according to the rejection rate of the prompt words in the database of words that should be rejected corresponding to the tested model, combined with the content security detection results and the watermark detection results, a detection report is analyzed.

6. A security risk detection device for content generated by a large model of a text image, characterized in that: include: A test data module, used to select a test category according to a test parameter based on a preset test database, and match a corresponding data set from the test database according to the test category; Input the keywords in the data set into the tested large model in batches according to the format of positive prompt words; A content security detection module, used to perform content security detection on the corresponding picture or video image generated by the large model under test according to the input prompt word; A logo detection module, used for performing watermark logo detection on the corresponding picture or video image generated by the large model under test according to the input prompt word; The result analysis module is used to analyze and obtain a detection report according to the content security detection result and the watermark detection result.

7. The security risk detection device for content generated by the large model of the text graph according to claim 6 is characterized in that: The test database is a key risk word library constructed according to specific standards or specifications.

8. The security risk detection device for content generated by the large model of the text graph according to claim 6 is characterized in that: The content security detection module is specifically used to: Use rule engines and machine learning algorithms to perform automated detection, extract the content of pictures or videos, reversely infer the reverse prompt words of the pictures or videos, and judge and mark the illegal content based on the reverse prompt words; Manually review and confirm the illegal content marked by automated detection to obtain content risk assessment results; The identification detection module is specifically used for: By using OCR technology, the corresponding picture or video image generated by the large model under test according to the input prompt word is subjected to display watermark detection.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method for detecting security risks of content generated by a large model of a text map are implemented as described in any one of claims 1 to 5.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the security risk detection method for generating content for the large model of the culture graph are implemented as described in any one of claims 1 to 5.

Images