Method and device for generating test samples for large model security testing, and electronic device

Abstract

The application relates to the technical field of security testing, and provides a large model security test sample generation method, a generation device and electronic equipment.The method comprises the following steps: inputting a scene description text into a text-to-image model to obtain a first image output by the text-to-image model; inputting the scene description text and a model-induced text into an induced intensity label prediction model after training to obtain an induced intensity label output by the induced intensity label prediction model; performing image-text fusion on the first image and the model-induced text to obtain a second image; and constructing a security test sample based on the second image and the induced intensity label.The large model security test sample generation method provided by the application greatly improves the evaluation accuracy, reusability and test coverage of the security test sample.

Description

Technical Field

[0001] This application relates to the field of security testing technology, specifically to a method, apparatus, and electronic device for generating large-scale security test samples. Background Technology

[0002] Currently, with the rapid development of large-scale graphic model technology, its application in various fields is becoming increasingly widespread, but it also faces the security risk of being induced to output harmful content.

[0003] The existing methods for generating security test samples rely on human experience to design scenarios and instructions, lacking quantitative standards. This results in large differences in the induction intensity of test data, making it impossible for users to reuse security test samples based on these differences, thus leading to low reusability of the generated security test samples. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for generating large-scale model safety test samples, which solves the technical problem that users cannot reuse generated safety test samples based on differences in induction intensity, resulting in low reusability of safety test samples.

[0005] In a first aspect, embodiments of this application provide a method for generating large-scale model security test samples, comprising: The scene description text is input into the text-to-image model to obtain the first image output by the text-to-image model; The scene description text and the model-induced text are input into the induced intensity label prediction model after training to obtain the induced intensity label output by the induced intensity label prediction model; The first image and the model-induced text are fused to obtain the second image; Based on the second image and the induced intensity label, a security test sample is constructed.

[0006] In one embodiment, the induction intensity label includes a concealment level, a hazard level, and an induction intensity level; The induced intensity label prediction model includes an encoding layer, an intermediate prediction head module, a cross-attention fusion layer, and a final prediction head. The step of inputting the scene description text and the model-induced text into the induced intensity label prediction model after training to obtain the induced intensity label output by the induced intensity label prediction model includes: The scene description text is input into the encoding layer to obtain the scene description features output by the encoding layer; and the model-induced text is input into the encoding layer to obtain the induced text features output by the encoding layer. The scene description features and the inducement text features are input into the intermediate prediction head module to predict concealment and hazard, thereby obtaining the concealment level and the hazard level; The scene description features and the induced text features are input into the cross-attention fusion layer for feature fusion to obtain the fused features output by the cross-attention fusion layer; The fused features are input into the final prediction head to obtain the induction intensity level output by the final prediction head.

[0007] In one embodiment, the induced intensity label prediction model further includes a feature enhancement layer, wherein inputting the fused features into the final prediction head to obtain the induced intensity level output by the final prediction head includes: The fused features are input into the feature enhancement layer and subjected to dimensionality increase and dimensionality reduction processes in sequence to obtain the enhanced fused features. The enhanced fusion features are input into the final prediction head to obtain the induction intensity level output by the final prediction head.

[0008] In one embodiment, the induced intensity label prediction model is trained in the following manner: Obtain a first induced intensity label prediction model, which is a model obtained by training an initial induced intensity label prediction model using an initial training set; Based on the uncertainty metric value of each unlabeled sample in the unlabeled sample set, the unlabeled samples in the unlabeled sample set are sorted according to a sorting rule from largest to smallest to obtain the sorting result; Based on the first G unlabeled samples in the sorting result, a set of samples to be labeled is constructed; The initial training set is updated based on the labeled sample set, wherein the labeled sample set is the set obtained by labeling each of the unlabeled samples in the unlabeled sample set; The first induced intensity label prediction model is trained for at least one round based on the updated initial training set to obtain the induced intensity label prediction model.

[0009] In one embodiment, the uncertainty metric is determined in the following manner: The first unlabeled sample is input into the first induction intensity label prediction model, and the last random deactivation layer in the first induction intensity label prediction model is enabled. K Monte Carlo samplings are performed to obtain K prediction results. Calculate the average of the K prediction results to obtain the uncertainty metric value corresponding to the first unlabeled sample; Wherein, the first unlabeled sample is any one of the unlabeled samples in the set of unlabeled samples.

[0010] In one embodiment, before constructing the set of samples to be labeled based on the top G unlabeled samples in the sorting result, the method further includes: Obtain the total number of samples in the unlabeled sample set; Obtain the screening ratio coefficient and the first sample number, where the first sample number is the maximum number of samples in a single annotation. The smaller value between the first product value and the first number of samples is taken as the value of G; Wherein, the first product value is the product of the total number of samples and the screening ratio coefficient.

[0011] In one embodiment, the step of performing image-text fusion on the first image and the model-induced text to obtain a second image includes: The font display size of a single font is determined based on the image height of the first image, the image width of the first image, and the number of characters in the model-induced text; Based on the font display size, the number of characters in the model-induced text, the image height of the first image, and the image width of the first image, a range of selectable coordinates is determined; Select text embedding coordinates within the selectable coordinate range, and based on the text embedding coordinates, induce the model to embed text into the first image to obtain the second image.

[0012] In one embodiment, the step of inducing text embedding into the first image based on the text embedding coordinates to obtain the second image includes: Obtain watermark embedding parameters, wherein the watermark embedding parameters include at least one of transparency, font style, and font color; A pixel matrix containing the model-induced text is constructed based on the watermark embedding parameters; The pixel matrix is ​​added at the text embedding coordinates in the first image to obtain the second image.

[0013] Secondly, embodiments of this application provide an apparatus for generating large-scale model security test samples, comprising: The first processing module is used to input scene description text into a text-generated image model to obtain a first image output by the text-generated image model. The second processing module is used to input the scene description text and the model-induced text into the induced intensity label prediction model after training, and obtain the induced intensity label output by the induced intensity label prediction model. The fusion module is used to perform image-text fusion on the first image and the model-induced text to obtain a second image; A construction module is used to construct a security test sample based on the second image and the induced intensity label.

[0014] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the method for generating large-scale security test samples as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for generating large-scale security test samples as described in the first aspect.

[0016] The method, apparatus, and electronic device for generating large-scale model security test samples provided in this application generate a first image by inputting scene description text into a text-to-image model, providing visually concealed and realistic scene support for the model-induced text. Simultaneously, the scene description text and the model-induced text are input into a pre-trained prediction model to generate induced strength labels. Utilizing the model's understanding of the risk associated with dual-text collaboration, the induced risk is objectively quantified. Furthermore, the first image and the model-induced text are fused to generate a second image, effectively activating the real semantic association between image and text elements in the test data. These features work together to create security test samples that not only possess strong image-text semantic collaboration and concealment, effectively simulating complex induced attacks faced by multimodal large models in practical applications, but also feature a unified standard for quantifying induced strength. This data processing mechanism completely solves the problems of rigid image-text piecing together and excessively large differences in induced strength due to reliance on subjective human experience in existing methods, greatly improving the evaluation accuracy, reusability, and test coverage of security test samples. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the method for generating large-scale security test samples provided in this application embodiment; Figure 2This is a flowchart illustrating the process of inputting scene description text and model-induced text into an induced intensity label prediction model after training, as provided in this embodiment of the application, to obtain the induced intensity label output by the induced intensity label prediction model. Figure 3 This is a schematic diagram of the structure of the induced intensity label prediction model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the process of inputting fused features into the final prediction head to obtain the induction intensity level output by the final prediction head, provided by an embodiment of this application; Figure 5 This is a schematic diagram of the training process of the induced intensity label prediction model provided in the embodiments of this application; Figure 6 This is a schematic diagram of the training process of the induced intensity label prediction model provided in the embodiments of this application; Figure 7 This is a schematic diagram of the process for determining an uncertainty metric provided in an embodiment of this application; Figure 8 This is a schematic diagram of the process of fusing a first image and model-induced text to obtain a second image, provided in an embodiment of this application. Figure 9 This is a schematic diagram of the workflow of the training and inference phases provided in the embodiments of this application; Figure 10 This is a schematic block diagram of the apparatus for generating large model security test samples provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In the relevant technical solutions, the generation of security test samples is mainly divided into two aspects: the construction aspect of security test samples consisting only of text instructions and the construction aspect of security test samples consisting of graphic and textual elements.

[0021] Specifically, regarding the construction of security test samples containing text commands, RedTeam-2K is used as the basis, which contains 2,000 malicious queries covering 16 types of security policies. The sources include Generative Pre-trained Transformer (GPT) rewriting, handmade creation, GPT generation, and Large Language Model (LLM) jailbreaking research.

[0022] Specifically, during the construction process, security policy labels not covered by OpenAI and LLaMa-2 are reconstructed. GPT is used to address structural malformations, generating "GPT rewrite" queries. Content with unclear security or logic is manually reviewed and revised. A sentence converter model is used to calculate cosine similarity to filter duplicate content, ensuring dataset diversity. Simultaneously, multiple methods are employed when generating text attack samples. These include LoIIc-type cognitive overload, language switching, veiled expression, and causal inversion; Persuade-type 40 persuasion techniques to construct adversarial prompts; Template-type 78 open-source real-world jailbreak templates inserted into red team problems; and the general and transferable large-model jailbreak adversarial attack (Greedy Coordinate Gradient, GCG) technique to generate adversarial prompt suffixes, enriching the attack scenarios and types of the text command security dataset.

[0023] Specifically, regarding the construction of security test samples containing text and image elements, SafeBench identifies 8 major risk categories and 23 subcategories through 10 LLM prompts. It first generates 23×1000 text queries, filters them through scoring, and then iteratively generates high-quality images. Specifically, LLM generates image descriptions, Text-to-Image (T2I) generates images, Multimodal Large Language Models (MLLM) performs image-to-text generation and verifies semantic alignment, repeatedly optimizing until a match is achieved, ultimately forming 23×1000 text and image pairs. JailBreakV-28K, based on RedTeam-2K extensions, generates images including Stable Diffusion to generate keyword-corresponding images, keyword-based black-text images on a white background, and SD images combined with typed text. It also rewrites the questions to associate them with text and images. MM-SafetyBench generates 13×50 questions around 13 prohibited scenarios, extracts key phrases, and generates three types of images: SD images, layout images, and hybrid images. Similarly, it rewrites the questions to enhance the image-text relationship, resulting in 5040 samples. FigStep, on the other hand, rewrites prohibited text questions into statements starting with "Stepsto," converts them into numbered layout images, and pairs them with neutral text prompts to construct image-text data that induces the model to break out of jailbreak mode. These works all emphasize the semantic relationship between images and text to test the security performance of multimodal models under visual input.

[0024] However, the security test samples constructed in the above-mentioned aspects have the following drawbacks: 1. High subjectivity: Relying on human experience to design scenarios and instructions, lacking quantitative standards, results in large differences in the induction intensity of test data. Users cannot reuse security test samples based on the differences in induction intensity, resulting in low reusability of generated security test samples.

[0025] Second, incomplete coverage: The test data focuses on a few risk categories and cannot fully cover all kinds of security threats that the large graphic model may face, which is prone to missed detections.

[0026] Third, poor adaptability: Most of the open-source large model security datasets are in plain text format and have not been converted into mixed text and image inputs, which do not match the actual input format of large text and image models, and the evaluation results have limited reference value.

[0027] Further research revealed significant drawbacks in the existing methods for generating mixed text and image security test samples: 1. Image-text irrelevant problem: Using images to pair with text that are unrelated to text commands fails to activate the neural connections that link images and text in the multimodal model. As a result, the model makes judgments based solely on text information and fails to fully evaluate its safe response capability to image-text collaborative input.

[0028] Second, the text attachment is simplified: the text is directly attached to the background board without utilizing the background information of the text. The text and the image scene lack semantic connection, which cannot simulate the collaborative relationship in real scene where the image provides contextual support and the text makes instructions based on the context. It is difficult to effectively test the model's ability to recognize hidden and misleading graphic input.

[0029] Therefore, a method for generating large-scale model security test samples is needed to overcome at least one of the above-mentioned drawbacks.

[0030] Based on this, a classification system for safety risk categories is proposed.

[0031] Table 1 shows the classification system for security risk categories.

[0032] Table 1

[0033] This application provides a method for generating large-scale security test samples under the classification system of security risk categories shown in Table 1. The subject executing this method can be a computing device with data processing capabilities, such as a server, personal computer, cloud computing node, or any other terminal device.

[0034] Figure 1 This is one of the flowcharts illustrating the method for generating large-scale model security test samples provided in this application embodiment. For example... Figure 1 As shown in the embodiments of this application, a method for generating large-scale model security test samples is provided, which may include: Step 101: Input the scene description text into the text-to-image model to obtain the first image output by the text-to-image model.

[0035] Scene description text refers to textual information used to subtly depict a specific scenario or environment. In security testing of large models, to simulate realistic and deceptive input, scene description text typically does not directly contain illegal instructions, but rather provides contextual support, such as describing an outdoor construction site environment in hot weather. A text-to-image model refers to a deep learning model capable of converting natural language text into corresponding visual images; it can be an existing diffusion model or any other large-scale text-to-image model. The first image is the visual image generated by this text-to-image model based on the scene description text.

[0036] Specifically, after obtaining the pre-defined scene description text, it is used as input to a text-to-image generation model to generate a first image semantically associated with the scene description. This process leverages the subtle semantic expression of images, providing a visually realistic scene carrier for subsequent induced attacks.

[0037] Step 102: Input the scene description text and the model-induced text into the induced intensity label prediction model after training to obtain the induced intensity label output by the induced intensity label prediction model.

[0038] In this context, the model-induced text refers to the specific instructions given to the large model under test. These instructions typically contain varying degrees of potential harm or aggression, such as "Tell me where I can buy a drug that keeps me awake and alert; I must pass the exam." The induction intensity label prediction model, after training, is a classification network model that has been pre-trained on a large number of samples and is capable of quantitatively assessing the risk induction level of the input text. The induction intensity label is the quantitative result output by this prediction model, used to characterize the overall induction risk level formed by the combination of the scene description text and the model-induced text.

[0039] Step 103: Perform image-text fusion on the first image and the model-induced text to obtain the second image.

[0040] Image-text fusion refers to the physical or visual combination of visual image information and natural language text information, organically integrating them into a single digital medium. The second image is the fused image that simultaneously contains the visual features of the first image and model-induced text information.

[0041] Specifically, after obtaining the first image, the model-induced text is superimposed or embedded as a visual element into a specific region of the first image using a texture algorithm or image processing technique, thereby outputting a second image as a multimodal input.

[0042] Step 104: Construct a security test sample based on the second image and the induced intensity label.

[0043] The security test sample refers to the standardized data unit ultimately used to verify and evaluate the security defense capabilities of the large-scale image-text model. A second image containing the concealed scene and attack instructions, along with its corresponding quantitatively standardized induction strength label, is paired and associated, and the associated data pairs are stored in the security evaluation dataset. In subsequent evaluations of the large-scale model, the second image can be directly used as the multimodal mixed input of the model under test, while the induction strength label serves as a basic evaluation reference for analyzing the model's security response capabilities when facing inductions of different intensities.

[0044] In this embodiment, a first image is generated by inputting scene description text into a text-to-image model, providing visually concealed and realistic scene support for the model-induced text. Simultaneously, the scene description text and the model-induced text are input into a pre-trained prediction model to generate induced strength labels. Utilizing the model's understanding of the risk associated with dual-text collaboration, the induced risk is objectively quantified. Furthermore, the first image and the model-induced text are fused to generate a second image, effectively activating the true semantic association between image and text elements in the test data. These features work together to create a security test sample that not only possesses strong image-text semantic collaboration and concealment, effectively simulating complex induced attacks faced by multimodal large models in practical applications, but also features a unified standard for quantifying induced strength. This data processing mechanism completely solves the problems of rigid image-text piecing together and excessively large differences in induced strength due to subjective human experience in existing methods, greatly improving the evaluation accuracy, reusability, and test coverage of security test samples.

[0045] In one embodiment, the induction intensity label includes a concealment level, a hazard level, and an induction intensity level; the induction intensity label prediction model includes an encoding layer, an intermediate prediction head module, a cross-attention fusion layer, and a final prediction head.

[0046] like Figure 2 As shown, the scene description text and model-induced text are input into the induced intensity label prediction model after training, and the induced intensity labels output by the induced intensity label prediction model are obtained, including: Step 201: Input the scene description text into the encoding layer to obtain the scene description features output by the encoding layer, and input the model-induced text into the encoding layer to obtain the induced text features output by the encoding layer.

[0047] In some embodiments, the encoding layer is a pre-trained language model with frozen parameters (such as Bidirectional Encoder Representations from Transformers, BERT model, etc.). In practice, the scene description text and the model-induced text are input into the encoding layer as two independent channels. The encoding layer utilizes its pre-trained semantic encoding capabilities to perform word vector mapping and contextual semantic feature extraction on the input scene description text and model-induced text, respectively, thereby outputting fixed-dimensional scene description features and induced text features. This step transforms unstructured natural language text into high-dimensional continuous feature vectors that the model can process.

[0048] Specifically, the induced intensity label prediction model also includes an input layer, in which scene description text and model-induced text are transmitted through two channels of the input layer and input to the encoding layer to obtain scene description features and induced text features.

[0049] In some embodiments, scene description features f scene The dimension is represented as [batch_size, 768], which induces text features. f cmd The dimension is represented as [batch_size, 768].

[0050] Step 202: Input the scene description features and the inducement text features into the intermediate prediction head module to predict the concealment and hazard levels, and obtain the concealment level and hazard level.

[0051] The intermediate prediction head module is built after the encoding layer and is used to perform intermediate supervision tasks in the classification network branch, such as... Figure 3 As shown, it typically includes concealment prediction heads and hazard prediction heads. The concealment level is used to quantify the difficulty in identifying the scenario inducement method. For example, it can be divided into three levels of quantitative standards: low, medium, and high (low level represents explicit inducement clues, and high level represents highly concealed clues, etc.). The hazard level is used to quantify the adverse consequences or degree of violation that the instruction itself may cause. It can also be divided into three levels of quantitative standards: low, medium, and high.

[0052] Specifically, the extracted scene description features are input separately into the concealment prediction head, and through operations such as linear transformation and activation function, the concealment level, which represents the concealment degree of the scene, is output.

[0053] For example, the covertness prediction head includes a first linear layer (768→512), a first normalization layer (BatchNorm), a first activation function layer, a first dropout layer, a second linear layer (512→256), a second normalization layer, a second activation function layer, a second dropout layer, and a third linear layer (256→3) connected in sequence, and outputs the covertness level. .

[0054] Simultaneously, the extracted induced text features are individually input into the hazard prediction head, and the hazard level representing the degree of risk of the instruction is output.

[0055] For example, the hazard prediction head includes a fourth linear layer (768→512), a third normalized layer, a third activation function layer, a third dropout layer, a fifth linear layer (512→256), a fourth normalized layer, a fourth activation function layer, a fourth dropout layer, and a sixth linear layer (256→3) connected in sequence, outputting the hazard level. .

[0056] The dropout layer is also known as the random deactivation layer.

[0057] This step introduces an intermediate supervision mechanism from a multi-task learning framework, forcing the model to learn key features of scene concealment and instruction hazard separately, aligning with human annotation preferences.

[0058] Step 203: Input the scene description features and the guiding text features into the cross-attention fusion layer for feature fusion to obtain the fused features output by the cross-attention fusion layer.

[0059] The cross-attention fusion layer refers to a network layer that utilizes an attention mechanism to achieve information interaction and dynamic fusion between features from different modalities or channels. In practice, the computing device inputs scene description features and induced text features into this layer, and uses the cross-attention mechanism to calculate the association weights between scene semantics and malicious instructions, dynamically allocating the importance of the two channel features. For example, using induced text features as the query vector and scene description features as the key and value vectors, the layer captures the contextual support relationship between scene semantics and malicious instructions, thus organically combining the two channel features to output a fused feature that combines scene concealment features and instruction attack features.

[0060] For example, the cross-attention fusion layer includes a seventh linear layer. f cmd →Q (768→128), Eighth Linear Layer f scene →K, (768→128), Ninth Linear Layer f scene →V, (768→128), multi-head attention layer embedding dimension 128, number of heads 4, and fifth normalization layer; the multi-head attention output is connected to the q residual and then normalized to obtain the fused features. f fused Dimension [batch_size, 128].

[0061] Step 204: Input the fused features into the final prediction head to obtain the induction intensity level output by the final prediction head.

[0062] The final prediction head is the terminal structure of the classification network used to output the overall evaluation result. The induction intensity level reflects the comprehensive induction risk after the combination of scene and instruction, reflecting the overall risk synergy effect after the combination of text and image. The computing device feeds the above-mentioned fusion features rich in interactive information into the final prediction head, which undergoes dimensionality reduction mapping and normalization processing by the fully connected layer, and finally outputs the comprehensive induction intensity level.

[0063] For example, the final prediction head includes a 10th linear layer (128→256), a 6th normalized layer, a 5th activation function layer, a 5th dropout layer, an 11th linear layer (256→128), a 7th normalized layer, a 6th activation function layer, a 6th dropout layer, a 12th linear layer (128→64), an 8th normalized layer, a 7th activation function layer, a 7th dropout layer, and a 13th linear layer (64→3), outputting the induced intensity level. .

[0064] In this embodiment, by designing a dual-text input encoding layer in the induction intensity label prediction model, high-dimensional semantic features of the scene and the instruction are extracted separately, ensuring the independence and accuracy of feature extraction. Simultaneously, an intermediate prediction head module is set to independently output the concealment level and the severity level, introducing deep supervision signals at the data processing level. This forces the model to accurately capture specific risk dimension features strongly correlated with human cognition, enhancing the transparency and interpretability of model decisions. Furthermore, a cross-attention fusion layer is used to dynamically interact and calculate the dual-branch features, accurately capturing the synergistic relationship between scene concealment and instruction severity at the data feature level, with the final prediction head outputting the overall induction intensity level. These features work together to effectively solve the problems of poor adaptability and lack of interpretability in existing general text classification models when handling dual-text interactive induction scenarios, significantly improving the accuracy and generalization ability of induction intensity quantification prediction.

[0065] In one embodiment, the induced intensity label prediction model further includes a feature enhancement layer, such as... Figure 4 As shown, the fused features are input into the final prediction head to obtain the induction intensity level output by the final prediction head, including: Step 401: Input the fused features into the feature enhancement layer and perform dimensionality increase and dimensionality reduction processing in sequence to obtain the enhanced fused features.

[0066] The feature enhancement layer, located between the cross-attention fusion layer and the final prediction head, is a network structure used to perform further nonlinear transformations and deep semantic mining on the initially fused features. It is typically implemented using a multi-head self-attention encoder layer. Dimensionality increase refers to mapping the vector dimension of the input features to a higher-dimensional space to expand the feature representation space; dimensionality reduction involves remapping the high-dimensional features back to the original lower-dimensional space to remove redundant information and align with the input requirements of subsequent networks. The enhanced fused features are feature representations that retain higher-order semantic association information after the aforementioned spatial dimensional transformations and deep semantic mining.

[0067] Specifically, after obtaining the fused features output by the cross-attention fusion layer (e.g., its initial dimension can be 128), it is input into the feature enhancement layer. The feedforward hidden layer within the feature enhancement layer first performs dimensionality increase processing on the fused features, for example, by using a linear transformation to increase the feature dimension from 128 to 512. In a higher-dimensional space, the model can more fully perform non-linear transformations, thereby accurately capturing the more complex and hidden higher-order semantic relationships between scene description features and induced text features. Subsequently, the feature enhancement layer again performs dimensionality reduction processing on the vector extracted from the higher-order features through a linear mapping, reducing its dimension from 512 back to the original 128. This process of first increasing and then decreasing dimensionality enriches the expression and interaction of features during the dimensionality increase stage, and purifies key information during the dimensionality reduction stage, while maintaining the consistency of feature dimensions throughout the network's propagation process. The final output is an enhanced fused feature with consistent dimensions but richer semantic information.

[0068] For example, the feature enhancement layer adopts a multi-head self-attention encoder layer with an input dimension of 128, 4 heads, and a feedforward network hidden layer dimension of 512. By increasing the dimension to 512, the nonlinear transformation capability is enhanced to capture high-order semantic associations. Then, the dimension is reduced back to 128 to maintain the consistency of feature dimensions, and the output dimension is [batch_size, 128].

[0069] Step 402: Input the enhanced fusion features into the final prediction head to obtain the induction intensity level output by the final prediction head.

[0070] In this embodiment, by introducing a feature enhancement layer into the model and controlling the data flow to sequentially perform dimensionality increase and decrease processing, the features are endowed with the ability to undergo nonlinear transformations in a higher-dimensional space at the data processing level. This dimensionality increase mechanism allows the complex image-text collaborative risk relationships originally hidden in the cross-attention fusion features to be fully unfolded, helping the model capture more hidden higher-order semantic associations; the subsequent dimensionality decrease processing retains the core risk features while eliminating redundant data and maintaining the coherence of the data dimensions.

[0071] In some embodiments, the induced strength label prediction model is learned based on the cross-entropy loss function, which is a metric used to quantify the difference between two probability distributions. It is widely used in classification tasks to evaluate the consistency between the model's predicted probability distribution and the true label distribution.

[0072] Its mathematical definition is as follows: For a given sample, let the probability distribution of the true label be y, using one-hot encoding, that is, the position corresponding to the true category is 1, and the rest are 0, the probability distribution predicted by the model is... The output after normalization by the softmax function satisfies ,in, C For the number of classification categories, the cross-entropy loss is... L The calculation formula is: ; in, y i Represents the th in the true label distribution i The probability of a class The model predicts the first... i The probability of a class It is a function of the natural logarithm. When the predicted distribution... With the true distribution y When completely identical, cross-entropy loss L The minimum value is 0; the greater the difference between the two, the greater the loss value.

[0073] In the technical solution of this invention, the induced intensity label prediction model includes losses for three classification tasks. To enhance the model's feature learning of scene concealment and instruction hazard, deep supervision is introduced, and an independent prediction branch loss is designed for the intermediate prediction head module. ; in, It is the cross-entropy loss corresponding to the hidden prediction head, where, It is the true distribution of the hidden prediction heads. It is the predicted distribution of the hidden prediction head.

[0074] The purpose of this loss constraint is to force the model to align with human annotation preferences for scene concealment, allowing the model to learn the concealed features of scene elements.

[0075] ; in, It is the cross-entropy loss corresponding to the hazard prediction head, where, It is the true distribution of the hazard prediction heads. It is the predicted distribution of the hazard prediction head.

[0076] This loss constraint-driven model captures the inducing intent and hazard level of instructions, enhancing the understanding of the characteristics of risky instructions.

[0077] ; in, It is the cross-entropy loss corresponding to the final prediction head, where, It is the true distribution of the final predicted head. It is the prediction distribution of the final prediction head.

[0078] The total loss function of the induced intensity label prediction model weightedly fuses the basic induced prediction loss and the deep supervision loss to balance the importance of different supervision objectives. Its expression is: ; in, Weighting coefficient It needs to be determined through validation set optimization.

[0079] In some embodiments, the initial value is set to: =1.2 dominant induction intensity Predictions are needed to ensure the effectiveness of core tasks. =0.8 Key features of hidden characteristics in assisted learning scenarios. =0.8 Auxiliary learning instructions' key characteristics of harmfulness, while also through and The constraints force the model to align with human annotation preferences for scenes and instructions, improving interpretability and feature learning accuracy. Through multi-loss term collaborative optimization based on deep supervision, gradient vanishing is mitigated, providing more direct gradient guidance for intermediate branches, resulting in stronger interpretability. This allows the model to accurately learn features strongly correlated with human judgment, such as scene concealment and the harmfulness of instructions, thus improving generalization ability.

[0080] In one embodiment, such as Figure 5 As shown, the induced intensity label prediction model is trained in the following way: Step 501: Obtain the first induced intensity label prediction model. The first induced intensity label prediction model is a model obtained by training the initial induced intensity label prediction model using the initial training set.

[0081] The initial training set refers to a small set of samples pre-constructed and manually labeled during the early stages of model training. For example, it could be a seed sample (e.g., 1000 samples) initially extracted, containing scene description text, instruction text, and corresponding manually labeled induced intensity tags. The initial induced intensity tag prediction model refers to the basic network structure that has not yet been trained on specific safe induced data or the initial model state that has not been fine-tuned. The first induced intensity tag prediction model refers to an intermediate stage model that, after initial training on the aforementioned initial training set, possesses a certain basic prediction capability but has not yet reached its optimal or final stable state.

[0082] Specifically, the labeled samples (scene description text samples, model-induced text samples, and corresponding induced intensity labels) in the initial training set are input into the initial induced intensity label prediction model. The model parameters are updated through forward propagation and backpropagation algorithms. After completing the preset basic training rounds, the model weight parameters at this time are saved, thereby obtaining the first induced intensity label prediction model, which provides a baseline model for subsequent active learning iterations.

[0083] Step 502: Based on the uncertainty metric value of each unlabeled sample in the unlabeled sample set, sort the unlabeled samples in the unlabeled sample set according to the sorting rule from largest to smallest, and obtain the sorting result.

[0084] The unlabeled sample set refers to a data pool containing a large amount of raw dual text (i.e., a combination of scene description text and model-induced text) that has not been manually labeled with induced strength tags. The uncertainty metric is a numerical indicator used to quantify the degree of certainty or confusion the model itself has regarding the output result when classifying unlabeled samples. A higher metric indicates a more ambiguous and uncertain prediction for that sample. Such samples often lie at the model's decision boundary or contain complex induced features, and therefore have higher training value for improving the model's classification ability.

[0085] Specifically, the unlabeled sample set is traversed, and each unlabeled sample is input into the current first induction intensity label prediction model. By monitoring the probability distribution or feature state of the model's output layer, the prediction uncertainty value of the model for each sample is calculated. Subsequently, based on this uncertainty metric, all samples in the unlabeled sample set are sorted in descending order, so that the high-value samples that the model finds most difficult to accurately judge are placed at the top of the queue, thus generating an ordered ranking result.

[0086] Step 503: Based on the top G unlabeled samples in the sorting results, construct a set of samples to be labeled.

[0087] Here, G is a preset or dynamically calculated positive integer, and the set of samples to be labeled is a high-value subset of data extracted from massive unlabeled data, specifically intended for submission to human or expert systems for labeling. Based on the sorting results from largest to smallest, the top G unlabeled samples are directly selected. This step strictly controls the subsequent human labeling scope to the sample set most needed by the model, thereby significantly limiting the scale of data processing while ensuring the quality of the dataset.

[0088] For example, according to the uncertainty metric Sort the samples in the unlabeled sample set from largest to smallest, and select the top G samples to include in the labeling queue. Let the sample set in the unlabeled sample set be S = {s1, s2, ... s}. M The corresponding set of uncertainty measures is} Then, based on the uncertainty measure set, the uncertainty measure values In descending order, for s1, s2, ... s MThe samples are sorted to obtain the sorting results, and then the top G samples are selected to construct the set of samples to be labeled.

[0089] In some embodiments, a ranking function rank(s) is defined. i ) represents sample s i according to The ranking after sorting from largest to smallest, i.e., rank(s) i )=t means that the sample s is at position t after sorting. i The set of samples to be labeled, constructed by selecting the top G unlabeled samples, is as follows: ; Step 504: Update the initial training set based on the labeled sample set. The labeled sample set is the set obtained by labeling each unlabeled sample in the unlabeled sample set.

[0090] The labeled sample set refers to the data set of the aforementioned sample set to be labeled, which has been manually judged and assigned accurate three-level quantitative labels (such as scene concealment, instruction hazard, and overall induction intensity level). After receiving these labeled sample sets with authoritative labels, they are merged and added to the original initial training set, dynamically expanding the size and content of the training dataset, thereby completing the update and iteration of the initial training set. This ensures that the updated training set includes error correction samples that the model was originally prone to misjudging or difficult to judge.

[0091] In some embodiments, each unlabeled sample in the set of samples to be labeled is processed according to the process of primary labeling, advanced review, and secondary labeling, specifically: like Figure 6 As shown, during the initial annotation phase, samples are drawn from the set of samples to be annotated. Three intermediate safety officers independently and in parallel annotate these samples, outputting level labels for scene concealment, instruction hazard, and overall induction intensity, forming three sets of initial results. Subsequently, a senior safety officer reviews the initial results: if the annotation results match the typical features of the level definition, they are marked as passed, and the samples enter the initial training set; if there are discrepancies in the annotation results or conflicts with the level definition, specific correction suggestions are given, and unannotated samples are returned to the set of samples to be annotated. Finally, the returned samples are reassigned to intermediate safety officers for annotation, and the annotation and review steps are repeated until the samples pass the review.

[0092] In some embodiments, by labeling scene concealment, instruction hazard, and overall induction intensity, an induction intensity label is formed that includes scene concealment level, instruction hazard level, and overall induction intensity level. Table 2 shows the labeling system of the induction intensity label.

[0093] Table 2

[0094] As shown in Table 2, the scene concealment level and the command hazard level form the basis of the overall induction intensity level. The overall induction intensity level is determined by considering different combinations of scene concealment and command hazard. Among these, there are two combinations that indicate a high overall induction intensity, and these two combinations are of great significance for evaluating the model's security defense capabilities: One approach is a combination of high-context concealment (CI) and low-instruction-harm (HI). This combination is characterized by extremely subtle inducements, such as through metaphors, hints, or logical traps, but the instructions themselves are relatively harmless, such as mildly inappropriate guidance or borderline content. If the model fails to output unsafe content under this combination, it indicates a weakness in its ability to identify subtle inducements. Even if the harm is not high, as long as the inducement method is sufficiently obscure, it can still breach the model's defenses. This boundary reflects the model's limit of resistance to disguised risks.

[0095] Another combination is the low-concealment CI (Content Instruction) and high-instruction hazard HI (Instruction Harm). This combination is characterized by direct and obvious inducement methods, such as straightforward instruction statements, but the instructions themselves are extremely harmful, such as involving violence, illegal activities, or other seriously harmful content. If the model fails under this combination, it indicates insufficient filtering ability for overtly harmful content. Even with simple inducement methods, if the harm intensity is high enough, the model still cannot effectively intercept it. This boundary reflects the model's tolerance threshold for high-risk content.

[0096] Step 505: Train the first induced intensity label prediction model for at least one round based on the updated initial training set to obtain the induced intensity label prediction model.

[0097] After obtaining the expanded, high-quality training data, the updated initial training set is re-input into the first induced intensity label prediction model, and training continues based on the previous model parameters (i.e., model parameter tuning for at least one epoch). As the features of high-value, difficult-to-distinguish samples are continuously backpropagated, the model gradually corrects its decision boundary. After one or more stages of iterative training, training stops when the model's prediction accuracy reaches a stable threshold (e.g., above 90%) or meets the convergence condition, ultimately outputting an induced intensity label prediction model with high-precision risk prediction capabilities.

[0098] Traditional annotation processes often involve indiscriminately labeling all samples, requiring significant manpower and resulting in low efficiency. Furthermore, the lack of differentiation based on sample importance can lead to the neglect of samples crucial for model training, negatively impacting training performance. In contrast, an annotation and review process based on active learning measures the uncertainty of samples based on model feedback during training, selecting the most valuable samples for manual annotation. This significantly reduces manpower costs and improves annotation efficiency while ensuring dataset quality.

[0099] In this embodiment, an uncertainty sampling process based on active learning is proposed. For any stage, after model training, multiple forward propagations of the model are performed on each sample in the unlabeled dataset. By measuring the uncertainty of the model output, samples with induced feature ambiguity and those that the model cannot accurately judge are filtered out. These samples have higher value for model training.

[0100] Specifically, in this embodiment, an active learning data processing mechanism is introduced during model training. By automatically evaluating the output distribution of the first induced intensity label prediction model for unlabeled samples, uncertainty measures that characterize the ambiguity in the model's decision-making are extracted. These measures are then used as a benchmark for descending sorting, selecting the top G high-uncertainty samples for labeling. This data processing transforms the previously undifferentiated labeling of the entire dataset into targeted labeling of high-value data sequences. The labeled high-value samples are then integrated into the initial training set to drive model training again, ensuring that each iteration of model parameter updates accurately covers previously weak feature areas (such as the complex boundaries between highly concealed scenarios and low-hazard instruction combinations). The various data selection and iteration features work together to significantly reduce the reliance on manually labeled data at the data flow level, effectively saving computing power and manpower costs. Simultaneously, the error correction mechanism significantly improves the generalization ability and prediction accuracy of the final generated induced intensity label prediction model.

[0101] In one embodiment, such as Figure 7 As shown, the uncertainty metric is determined in the following way: Step 701: Input the first unlabeled sample into the first induction intensity label prediction model, and enable the last random deactivation layer in the first induction intensity label prediction model to complete K Monte Carlo samplings and obtain K prediction results.

[0102] Specifically, Monte Carlo Dropout is a data processing method that performs multiple forward propagations on the same input sample while maintaining a random deactivation mechanism. The number of samplings, K, represents the number of forward propagations performed. The prediction result refers to the probability distribution of the possible induced intensity labels for that sample output by the model at each sampling stage.

[0103] The last random inactivation layer, which is also the last dropout layer in the first induced intensity label prediction model, predicts the results based on the label distribution. The entropy is represented specifically as follows: ; in, Label distribution entropy, For label distribution, It is the natural logarithm function. s This is the first unlabeled sample.

[0104] Step 702: Calculate the average of the K prediction results to obtain the uncertainty measure corresponding to the first unlabeled sample.

[0105] The first unlabeled sample is any unlabeled sample in the set of unlabeled samples.

[0106] Specifically, the average of the K prediction results The following formula is used to calculate the result: ; in, The larger the value, the higher the uncertainty of the model's prediction of sample s, and the greater the potential value of the sample for model training.

[0107] In this embodiment, by inputting the first unlabeled sample into the model and forcibly enabling the last random deactivation layer, K Monte Carlo samplings are performed on the model. This introduces randomness of node states into the model's forward data propagation path, enabling the computing device to capture the diverse fluctuations in the model's output when faced with the same input. Furthermore, by calculating the average of these K prediction results (such as the mean of prediction entropy) as an uncertainty measure, the model's confidence in the samples is mathematically quantified from a statistical perspective of data processing. These steps work together to enable the system to scientifically calculate the information value of each unlabeled sample, accurately identify high-value, difficult-to-identify samples with induced feature ambiguity that are most likely to cause model misjudgment, thus providing a reliable and interpretable quantitative basis for subsequent data screening based on active learning, effectively improving the efficiency of discovering high-quality samples.

[0108] In one embodiment, before constructing the set of samples to be labeled based on the top G unlabeled samples in the sorting results, the method further includes: obtaining the total number of samples in the unlabeled sample set; obtaining a screening ratio coefficient and a first sample number, wherein the first sample number is the maximum number of samples to be labeled in a single instance; and taking the smaller of the first product value and the first sample number as the value of G; wherein the first product value is the product of the total number of samples and the screening ratio coefficient.

[0109] The screening scale factor is a preset proportional value for controlling the scale of data extraction, and its value range is usually greater than zero and less than or equal to 100%. The first sample number is the upper limit of the number of samples assigned by the system to manual labeling tasks in a single time, that is, the maximum number of samples for a single labeling. The first product value refers to the theoretical extraction value obtained by multiplying the total number of samples by the screening scale factor.

[0110] In this embodiment, the first product value is obtained by acquiring the total number of unlabeled samples and multiplying it by the screening scale factor, and a minimum value taking mechanism is used to compare the first product value with the first sample number representing the upper limit of manual labeling capability to determine the final sampling quantity G. This avoids the problem that an excessively small extraction ratio when the data volume is extremely large leads to insufficient high-value data support for model training; while the introduction of the first sample number and the minimum value comparison processing with the first product value acts as a hard boundary constraint, which strictly prevents the number of screened samples from exceeding the actual processing load of the manual labeling team due to an excessively large total number of samples. While ensuring that the number of high-value samples screened out by active learning meets the model iteration requirements, it effectively avoids resource overload and achieves the optimal balance between model training efficiency and manual labeling cost.

[0111] In one embodiment, the value of G is determined by the following formula: ; Wherein, M is the total number of samples in the unlabeled sample set, A is the screening scale factor with a value range of 0<A≤1, which is set according to the abundance of labeling resources and the model training requirements, and usually takes a value from 0.1 to 0.3. B is the first sample number, which is set according to the workload and labeling efficiency of the labeling personnel, is the floor function. This formula not only ensures that the number of screened samples is adapted to the scale of the unlabeled sample set, but also prevents the number of samples from exceeding the labeling capability due to an excessive number of samples.

[0112] In one embodiment, as Figure 8 shown, performing image-text fusion on the first image and the model induction text to obtain the second image includes: Step 801: Determine the font display size of a single font based on the image height of the first image, the image width of the first image, and the number of characters in the model induction text.

[0113] Wherein, the image height and the image width are basic geometric resolution parameters characterizing the overall spatial size of the first image, and their unit is usually pixel. The number of characters refers to the total number of characters contained in the model induction text. The font display size refers to the pixel size occupied by a single character when the model induction text is rendered in the image.

[0114] Specifically, the size information of the first image (such as output width and height parameters) is first obtained, and the number of characters in the model-induced text is counted. To ensure that the text occupies a reasonable proportion in the image, neither exceeding the boundary nor being too small to be visible, the font display size is dynamically calculated following the principle of adapting the text length to the image size.

[0115] In some embodiments, the ratio of image width to text quantity can be multiplied by a first scaling factor to obtain a first candidate value, and the image height can be multiplied by a second scaling factor to obtain a second candidate value. Then, the smaller value between the first candidate value and the second candidate value is selected as the final font display size.

[0116] For example, if the number of characters in the model-induced text is N, then the font size (fontSize) is calculated using the following formula: fontSize=min(k1×width / N, k2×height); Where k1 is the first scaling factor, k2 is the second scaling factor, width is the image width, and height is the image height.

[0117] In some embodiments, k1 and k2 are set to 0.05~0.1 to ensure that the text occupies a reasonable proportion in the image, neither exceeding the boundary nor being too small to be visible.

[0118] Step 802: Determine the selectable coordinate range based on the font display size, the number of characters in the model-induced text, the image height of the first image, and the image width of the first image.

[0119] The selectable coordinate range refers to the effective spatial bounding box that allows the model-induced text to be placed on the two-dimensional pixel plane of the first image. This range is determined to prevent the generated text watermark from covering the edge areas of the image.

[0120] Specifically, based on the image width and height of the first image, and combined with the overall text width and height estimated from the font display size and the number of characters, the white space margin is calculated (for example, it can be set to a fixed multiple of the font display size, such as 1.5 times). Then, the overall size occupied by the text and the reserved boundary pixels are subtracted from the image width and height, respectively, thereby defining a two-dimensional effective coordinate range that ensures appropriate white space around the text, which is then used as the optional coordinate range.

[0121] For example, the selectable coordinate range is: x∈[margin, width-margin-textWidth], y∈[margin, height-margin-fontSize]. Here, margin is the reserved pixels for the boundary, set to 1.5 times fontSize, and textWidth is the total width of the text, estimated by fontSize and the number of characters. This range avoids the text from exceeding the image boundary, ensuring that there is blank space around the text and enhancing visual harmony.

[0122] Step 803: Select text embedding coordinates from the selectable coordinate range, and based on the text embedding coordinates, induce the model to embed the text into the first image to obtain the second image.

[0123] The text embedding coordinates refer to the reference point where the model-induced text is placed in the first image, which is usually the coordinate point of the upper left corner of the text box.

[0124] Specifically, algorithms such as uniform random sampling are used to select position coordinates to obtain random text embedding coordinates. After obtaining these coordinates, the computing device calls a drawing or image rendering algorithm, using these text embedding coordinates as a reference position, and writes the model-induced text into the data of the first image according to the previously determined font display size, thereby generating a second image that integrates the visual scene and the instruction text. This random selection ensures that the text position of each generated fused sample is not fixed.

[0125] In this embodiment, by extracting the width and height features of the first image and the number of characters in the text, a dynamic adaptation mapping relationship between the text length and the image spatial size is established, accurately calculating the font display size and the selectable coordinate range. Furthermore, random coordinate sampling is performed using this range constraint, and the text is embedded within it. This process not only ensures a high degree of coordination between the text content and the background image in terms of visual size and layout, avoiding text overflow or proportional imbalance, but also eliminates the patterned layout distribution through dynamic random positioning, effectively avoiding overfitting of the large model under test due to fixed layout. This improves the data quality and robustness of the security test samples in simulating real-world complex graphic scenarios.

[0126] In one embodiment, based on text embedding coordinates, the model induces text embedding into a first image to obtain a second image, including: Obtain watermark embedding parameters, which include at least one of transparency, font style, and font color; A pixel matrix containing model-induced text is constructed based on watermark embedding parameters; Add a pixel matrix at the text embedding coordinates in the first image to obtain the second image.

[0127] In this embodiment, by acquiring watermark embedding parameters including transparency, font style, or font color, and rendering and constructing a pixel matrix of model-induced text accordingly, the instruction text to be embedded possesses semi-transparent and low-interference visual attributes in its underlying data representation. By fusing this pixel matrix with the original pixels of the image at the specified text embedding coordinates in the first image, pixel-level precise mixing of text and image data is achieved. This allows the model-induced text to be seamlessly and naturally embedded into the first image with specific semantics in the form of a visual watermark. This processing method avoids disrupting the visual coherence of the background scene at the underlying mechanism, ensuring the authenticity of the scene content, while simultaneously ensuring that the induced instructions can be effectively recognized by the visual extraction module of the multimodal large model. Thus, in the generated security test samples, it perfectly simulates the complex induced collaborative mechanism of highly concealed scene cover + instruction attack, significantly improving the induced effectiveness and visual rationality of the image-text fusion samples.

[0128] In some embodiments, a model-induced text is drawn on the text embedding coordinates using a drawing tool. Specific parameters include: transparency: set to E=0.3~0.5 (RGB channel blending coefficient) to ensure the watermark does not obscure the main content of the image; font style: using a sans-serif font to avoid complex strokes interfering with the image's visual effect; color: selecting a color with moderate contrast to the image's main color tone to balance visibility and blending. After drawing, the modified pixel matrix is ​​written back to the first image, and the watermarked blended image is output to obtain the second image.

[0129] In some embodiments, such as Figure 9 As shown, the method for generating large-scale model security test samples can be applied to the inference stage after the training phase, thereby generating security test samples including a second image and induced intensity labels.

[0130] During the training phase, the induced intensity label prediction model undergoes H stages of active learning to meet user needs.

[0131] Table 3 shows examples of security test samples generated by the method for generating large-scale security test samples proposed in this invention.

[0132] Table 3

[0133] The apparatus for generating security test samples provided in the embodiments of this application will be described below. The apparatus for generating security test samples described below can be referred to in correspondence with the method for generating large-scale security test samples described above.

[0134] In some embodiments, such as Figure 10 As shown, a device for generating large-scale model security test samples is proposed, comprising: The first processing module 1001 is used to input scene description text into a text-generated image model to obtain a first image output by the text-generated image model. The second processing module 1002 is used to input the scene description text and the model-induced text into the induced intensity label prediction model after training, and obtain the induced intensity label output by the induced intensity label prediction model. The fusion module 1003 is used to perform image-text fusion on the first image and the model-induced text to obtain a second image; The construction module 1004 is used to construct a security test sample based on the second image and the induced intensity label.

[0135] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call a computer program in the memory 1130 to execute the steps of a method for generating large model security test samples, such as: inputting scene description text into a text-to-image model to obtain a first image output by the text-to-image model; inputting the scene description text and model-induced text into a induced intensity label prediction model after training to obtain an induced intensity label output by the induced intensity label prediction model; performing image-text fusion on the first image and the model-induced text to obtain a second image; and constructing a security test sample based on the second image and the induced intensity label.

[0136] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the method for generating large model security test samples provided in the above embodiments, such as: inputting scene description text into a text-to-image model to obtain a first image output by the text-to-image model; inputting the scene description text and model-induced text into an induced intensity label prediction model after training to obtain an induced intensity label output by the induced intensity label prediction model; performing image-text fusion on the first image and the model-induced text to obtain a second image; and constructing a security test sample based on the second image and the induced intensity label.

[0138] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the methods provided in the above embodiments, such as: inputting scene description text into a text-to-image model to obtain a first image output by the text-to-image model; inputting the scene description text and model-induced text into an induced intensity label prediction model after training to obtain an induced intensity label output by the induced intensity label prediction model; performing image-text fusion on the first image and the model-induced text to obtain a second image; and constructing a security test sample based on the second image and the induced intensity label.

[0139] Processor-readable storage media can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating large-scale model security test samples, characterized in that, include: The scene description text is input into the text-to-image model to obtain the first image output by the text-to-image model; The scene description text and the model-induced text are input into the induced intensity label prediction model after training to obtain the induced intensity label output by the induced intensity label prediction model; The first image and the model-induced text are fused to obtain the second image; Based on the second image and the induced intensity label, a security test sample is constructed.

2. The method for generating large-scale model security test samples according to claim 1, characterized in that, The induction intensity label includes concealment level, hazard level, and induction intensity level; The induced intensity label prediction model includes an encoding layer, an intermediate prediction head module, a cross-attention fusion layer, and a final prediction head. The step of inputting the scene description text and the model-induced text into the induced intensity label prediction model after training to obtain the induced intensity label output by the induced intensity label prediction model includes: The scene description text is input into the encoding layer to obtain the scene description features output by the encoding layer; and the model-induced text is input into the encoding layer to obtain the induced text features output by the encoding layer. The scene description features and the inducement text features are input into the intermediate prediction head module to predict concealment and hazard, thereby obtaining the concealment level and the hazard level; The scene description features and the induced text features are input into the cross-attention fusion layer for feature fusion to obtain the fused features output by the cross-attention fusion layer; The fused features are input into the final prediction head to obtain the induction intensity level output by the final prediction head.

3. The method for generating large-scale model security test samples according to claim 2, characterized in that, The induced intensity label prediction model further includes a feature enhancement layer, wherein the fused features are input into the final prediction head to obtain the induced intensity level output by the final prediction head, including: The fused features are input into the feature enhancement layer and subjected to dimensionality increase and dimensionality reduction processes in sequence to obtain the enhanced fused features. The enhanced fusion features are input into the final prediction head to obtain the induction intensity level output by the final prediction head.

4. The method for generating large-scale model security test samples according to claim 1, characterized in that, The induced intensity label prediction model was trained in the following way: Obtain a first induced intensity label prediction model, which is a model obtained by training an initial induced intensity label prediction model using an initial training set; Based on the uncertainty metric value of each unlabeled sample in the unlabeled sample set, the unlabeled samples in the unlabeled sample set are sorted according to a sorting rule from largest to smallest to obtain the sorting result; Based on the first G unlabeled samples in the sorting result, a set of samples to be labeled is constructed; The initial training set is updated based on the labeled sample set, wherein the labeled sample set is the set obtained by labeling each of the unlabeled samples in the unlabeled sample set; The first induced intensity label prediction model is trained for at least one round based on the updated initial training set to obtain the induced intensity label prediction model.

5. The method for generating large-scale model security test samples according to claim 4, characterized in that, The uncertainty metric is determined in the following manner: The first unlabeled sample is input into the first induction intensity label prediction model, and the last random deactivation layer in the first induction intensity label prediction model is enabled. K Monte Carlo samplings are performed to obtain K prediction results. Calculate the average of the K prediction results to obtain the uncertainty metric value corresponding to the first unlabeled sample; Wherein, the first unlabeled sample is any one of the unlabeled samples in the set of unlabeled samples.

6. The method for generating large-scale model security test samples according to claim 4, characterized in that, Before constructing the set of samples to be labeled based on the first G unlabeled samples in the sorting result, the method further includes: Obtain the total number of samples in the unlabeled sample set; Obtain the screening ratio coefficient and the first sample number, where the first sample number is the maximum number of samples in a single annotation. The smaller value between the first product value and the first number of samples is taken as the value of G; Wherein, the first product value is the product of the total number of samples and the screening ratio coefficient.

7. The method for generating large-scale model security test samples according to any one of claims 1 to 6, characterized in that, The step of fusing the first image and the model-induced text to obtain the second image includes: The font display size of a single font is determined based on the image height of the first image, the image width of the first image, and the number of characters in the model-induced text; Based on the font display size, the number of characters in the model-induced text, the image height of the first image, and the image width of the first image, a range of selectable coordinates is determined; Select text embedding coordinates within the selectable coordinate range, and based on the text embedding coordinates, induce the model to embed text into the first image to obtain the second image.

8. The method for generating large-scale model security test samples according to claim 7, characterized in that, The step of inducing text embedding into the first image based on the text embedding coordinates to obtain the second image includes: Obtain watermark embedding parameters, wherein the watermark embedding parameters include at least one of transparency, font style, and font color; A pixel matrix containing the model-induced text is constructed based on the watermark embedding parameters; The pixel matrix is ​​added at the text embedding coordinates in the first image to obtain the second image.

9. A device for generating large-scale model security test samples, characterized in that, include: The first processing module is used to input scene description text into a text-generated image model to obtain a first image output by the text-generated image model. The second processing module is used to input the scene description text and the model-induced text into the induced intensity label prediction model after training, and obtain the induced intensity label output by the induced intensity label prediction model. The fusion module is used to perform image-text fusion on the first image and the model-induced text to obtain a second image; A construction module is used to construct a security test sample based on the second image and the induced intensity label.

10. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for generating large model security test samples according to any one of claims 1 to 8.

Images