Method and device for generating adversarial attack sample, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a method and device for generating an adversarial attack sample, a storage medium and electronic equipment, and the method comprises the steps: inputting a risk picture and an initial first cue word into a graph-to-text big model, and obtaining first text description information corresponding to the risk picture output by the graph-to-text big model; performing risk obscure degree evaluation on the first text description information, and optimizing the first cue word; inputting the first text description information into a proxy text graph large model to obtain an output picture corresponding to the first text description information output by the proxy text graph large model; performing risk assessment on the output picture, and optimizing a first prompt word; determining whether the output picture has a risk or not; if yes, storing the first text description information; and re-inputting the risk picture and the optimized first cue word into the graph-to-text model, and circulating in this way to obtain a graph-to-text confrontation attack sample.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium and electronic device for generating anti-attack samples. Background Art

[0002] With the development of artificial intelligence technology, large artificial intelligence models have become an important part of our lives, work and economy, but the security issues that come with it have to attract our attention. For example, criminals take advantage of model loopholes to conduct fraudulent attacks and try to achieve illegal or unethical purposes by competing with or evading artificial intelligence models.

[0003] Especially in the case of text-generated images, the generation of pornographic, violent and other image content through model loopholes and fraudulent behavior will cause even worse social impact. Therefore, before the model is released, it is extremely important to effectively defend against the endless attacks of text-generated image models.

[0004] There are two main methods for attacking large models of text graphs. One is to manually generate attack samples of text graphs based on human expert experience. This method not only generates a small number of samples, but is also time-consuming and labor-intensive. The other is to generate attack samples based on artificial rule strategies by manually setting rules. The disadvantage is poor generalization. Summary of the invention

[0005] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for generating adversarial attack samples.

[0006] The embodiments of this specification provide a method for generating adversarial attack samples, by inputting risk images and corresponding prompt words into a large image-to-text model to generate corresponding text description information, then inputting the text description information into a large proxy image-to-text model to generate a corresponding output image, and then judging whether the output image successfully attacks the large proxy image-to-text model. If successful, the text description information is saved, otherwise it is discarded. In this process, the prompt words are optimized according to the generated text description information and the output image, and the above process is repeated to continuously generate adversarial attack samples, which can greatly increase the number of adversarial attack samples generated, so that the attack samples and risk images are more generalized while maintaining consistency in content. The prompt words are continuously optimized by evaluating the text description information and the output image, so as to generate more diverse and complex high-precision adversarial attack samples. The sample breaks the thinking limitation of conventional use of experience and rules, greatly improves the efficiency of text-generated image adversarial attack, does not require manual intervention, can reduce labor costs, and the internal process is not complicated, the magnitude is small, as long as it is running, it can continuously generate a large number of adversarial attack samples, the adversarial attack samples generated by the optimized prompt words are more aggressive, the expression form is more obscure, and after the image-generated text conversion, the generated text description information is more natural to the model, and the attack success rate will be higher. The generated adversarial attack samples do not rely on human experience and any rules. After evaluating the text description information, the prompt words are optimized, so that the generated text description information is more obscure and complex, which can improve the accuracy and success rate of the attack. The text description information comes from the large model of image-generated text. Compared with the text generated by rules, the data distribution is more natural and the generalization is better. Not only has the generation and efficiency of adversarial attack samples in the large model of image-generated text been greatly improved, but also the quality and success rate of attack samples have been significantly improved. The method includes: Inputting the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; optimizing the first prompt word by evaluating the risk obscurity of the first text description information; Input the first text description information into the proxy Wenshengtu macro model to obtain an output image corresponding to the first text description information output by the proxy Wenshengtu macro model; optimize the first prompt word by performing risk assessment on the output image; Determine whether the output image has a risk; if so, save the first text description information; The risk image and the optimized first prompt word are re-input into the image-to-text large model, and this cycle is repeated to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes a plurality of first text description information.

[0007] Furthermore, the step of optimizing the first prompt word by evaluating the risk obscurity of the first text description information includes: Determine at least one risk word in the first text description information by evaluating the risk obscurity of the first text description information, wherein the obscurity corresponding to the at least one risk word satisfies a first preset condition; The first prompt word is optimized according to the at least one risk word.

[0008] Further, the optimizing the first prompt word according to the at least one risk word includes: Obtaining a replacement word corresponding to the at least one risk word, wherein the semantic similarity between the at least one risk word and the replacement word satisfies a second preset condition, and the obscurity corresponding to the replacement word satisfies a third preset condition; The first prompt word is optimized according to the at least one risk word and the replacement word.

[0009] Further, the optimizing the first prompt word according to the at least one risk word and the replacement word includes: The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture.

[0010] Further, the optimizing the first prompt word according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture includes: Determine a first target picture description object associated with the replacement word from at least one picture description object corresponding to the risk picture; The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with the first target image description object.

[0011] Furthermore, the optimizing the first prompt word by performing risk assessment on the output image includes: By performing risk assessment on the output image, a risk assessment result corresponding to the output image is obtained, wherein the risk assessment result includes risk content information and / or risk degree information; The first prompt word is optimized according to the risk assessment result.

[0012] Further, the risk assessment result includes risk content information; The step of optimizing the first prompt word according to the risk assessment result includes: Determining risk guidance prompt information corresponding to the output image according to the risk assessment result, wherein the correlation between the risk guidance prompt information and the risk content information satisfies a fourth preset condition; The first prompt word is optimized according to the risk guidance prompt information.

[0013] Further, determining the risk guidance prompt information corresponding to the output image according to the risk assessment result includes: According to the risk assessment result, risk guidance prompt information matching at least one image description object corresponding to the output image is determined.

[0014] Further, the determining, according to the risk assessment result, risk guidance prompt information matching at least one image description object corresponding to the output image includes: Determine a second target picture description object associated with the risk content information from at least one picture description object corresponding to the output picture; According to the risk assessment result, risk guidance prompt information matching the object described in the second target image is determined.

[0015] The embodiment of this specification also provides a device for generating an adversarial attack sample, including: A first obtaining module is used to input the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; and optimize the first prompt word by evaluating the risk obscurity of the first text description information; The second obtaining module is used to input the first text description information into the proxy Wenshengtu macro model, obtain an output image corresponding to the first text description information output by the proxy Wenshengtu macro model; and optimize the first prompt word by performing risk assessment on the output image; A risk determination module, used to determine whether the output image has a risk; if so, save the first text description information; The third acquisition module is used to re-input the risk image and the optimized first prompt word into the image-to-text large model, and repeat this cycle to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes multiple first text description information.

[0016] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0017] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0018] In the embodiment of the present specification, the risk image and the corresponding prompt word are input into the image-generated text model to generate the corresponding text description information, and then the text description information is input into the proxy text-generated image model to generate the corresponding output image, and then it is determined whether the output image successfully attacks the proxy text-generated image model. If successful, the text description information is saved, otherwise it is discarded. In this process, the prompt word is optimized according to the generated text description information and the output image, and the above process is repeated to continuously generate adversarial attack samples, which can greatly increase the number of adversarial attack samples generated, so that the attack samples and risk images are more generalized while maintaining consistency in content. By evaluating the text description information and the output image, the prompt word is continuously optimized to generate more diverse and complex high-precision adversarial attack samples, breaking the conventional By utilizing the thinking limitations of experience and rules, the efficiency of text-to-image adversarial attacks is greatly improved. No manual intervention is required, which can reduce labor costs. The internal process is not complicated and the amount is small. As long as it is running, a large number of adversarial attack samples can be continuously generated. The adversarial attack samples generated by the optimized prompt words are more aggressive and the expression form is more obscure. After the image-to-text conversion, the generated text description information is more natural to the model, and the attack success rate will be higher. The generated adversarial attack samples do not rely on human experience and any rules. After evaluating the text description information, the prompt words are optimized, making the generated text description information more obscure and complex, which can improve the accuracy and success rate of the attack. The text description information comes from the large image-to-text model. Compared with the text generated by rules, the data distribution is more natural and the generalization is better. Not only has the generation and efficiency of adversarial attack samples in the large image-to-text model been greatly improved, but also the quality and success rate of the attack samples have been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for generating an adversarial attack sample provided in an embodiment of this specification.

[0020] Figure 2 A flowchart of a method for generating an adversarial attack sample is provided as an example in an embodiment of this specification.

[0021] Figure 3 A schematic diagram of the structure of a device for generating an adversarial attack sample provided in an embodiment of this specification.

[0022] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0024] See also Figure 1 , is a flowchart of a method for generating an adversarial attack sample provided in an embodiment of this specification. In an embodiment of this specification, the method for generating an adversarial attack sample is applied to a device for generating an adversarial attack sample (hereinafter referred to as "adversarial attack sample generating device") or an electronic device equipped with an adversarial attack sample generating device. Figure 1 The process shown is described in detail, and the method for generating an adversarial attack sample may specifically include the following steps: S102, input the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; and optimize the first prompt word by evaluating the risk obscurity of the first text description information.

[0025] In some embodiments, risk pictures include but are not limited to any pictures with risks, and this specification does not limit the picture content of risk pictures and the corresponding risk types. In some embodiments, in the fields of computer science and natural language processing, prompts refer to input information or instructions provided to a computer program or model. In a large language model, prompts are questions or statements provided by the user to the model, which are used to guide the model to generate relevant replies or responses. After receiving a prompt, the model will generate subsequent content or answers that are most relevant to the prompt based on its internally trained knowledge and algorithms. In some embodiments, the first prompt corresponding to the risk picture is used to guide the image-generated text model to generate text description information corresponding to the risk picture. In some embodiments, the content of the initial first prompt corresponding to the risk picture is pre-set, for example, the content of the initial first prompt can be "please use text to describe the content related to the picture", it should be noted that the content of the above initial first prompt is only for example, not for limitation, and those skilled in the art should understand that the initial first prompt of any content can be included in the protection scope of this specification.

[0026] In some embodiments, a large model (LLM) is a deep learning model trained on massive text data. It can generate natural language text or understand the meaning of language text. Such models can perform a variety of natural language processing tasks, including but not limited to text classification, question answering, and conversation. As the model size grows, LLM can generate more accurate and coherent outputs while processing more complex and longer input sequences. In addition, larger models can cover a wider range of knowledge and language contexts, thereby providing more comprehensive and targeted answers and solutions.

[0027] In some embodiments, the image-generated text model refers to a model that can convert image information into text descriptions, and is mainly used for image description generation and visual content summaries. This type of model is usually based on deep learning technology, especially the combination of convolutional neural networks (CNN) and recurrent neural networks (RNN), which can extract features from images and generate corresponding text descriptions. This specification does not limit the model structure and model parameters of the image-generated text model. In some embodiments, the risk image and its corresponding initial first prompt word are input into the image-generated text model to obtain the first text description information corresponding to the risk image output by the image-generated text model. The first text description information refers to the text information that describes the risk image input into the image-generated text model based on the first prompt word input into the image-generated text model.

[0028] In some embodiments, the text description reflection module performs a risk obscurity assessment on the first text description information output by the graph-generated text model, and assesses the risk obscurity of the first text description information, that is, assesses whether there is at least one risk word with a corresponding low degree of obscurity (that is, a corresponding high degree of semantic directness) in the first text description information, and obtains a corresponding assessment result, which includes the at least one risk word and / or the risk level corresponding to the at least one risk word. For example, the assessment is performed based on the content of the first text description information, and the assessment result indicates that sensitive words W1 and W2 are included therein. In some embodiments, the risk obscurity assessment of the first text description information may be performed manually (for example, by an operator) to obtain a corresponding assessment result input manually, or the assessment result output by the module may be obtained by inputting the first text description information into a trained risk obscurity assessment module. In some embodiments, the text description reflection module optimizes the initial first prompt word based on the obtained evaluation results. The purpose of the optimization operation is to make the large model of graph-generated text avoid using the at least one risk word as much as possible in the text description information generated by the large model of graph-generated text, and to improve the risk obscurity of the text description information generated by the large model of graph-generated text as much as possible. The optimization operation includes but is not limited to adding prompt information in the first prompt word to indicate that the large model of graph-generated text hides the at least one risk word in the generated text description information, or adding prompt information in the first prompt word to indicate that the large model of graph-generated text replaces the at least one risk word with other words in the generated text description information. In some embodiments, the text description reflection module generates a corresponding optimization scheme based on the obtained evaluation results, and optimizes the initial first prompt word according to the optimization scheme. The optimization scheme includes but is not limited to hiding the at least one risk word and replacing the at least one risk word with other words. This specification does not limit the optimization scheme. In some embodiments, the first prompt word is optimized based on the at least one risk word. For example, based on the content of the first text description information, an evaluation result is obtained indicating that sensitive words W1 and W2 are included therein. Then, an optimization scheme generated based on the evaluation result is to optimize the sensitive word W1 to word P1, and to optimize the sensitive word W2 to word P2. According to the optimization scheme, the initial first prompt word is optimized to obtain the latest first prompt word. For example, the first prompt word may be "Please use words to describe the target O1 in the picture. The text does not contain words such as W1 and W2."

[0029] In some embodiments, the prompt word storage (Memory) module is used to cache and update the first prompt word corresponding to the risk image, that is, the prompt word storage module stores the initial first prompt word corresponding to the risk image or the latest first prompt word obtained after updating the current first prompt word (the current first prompt word can be the initial first prompt word, or it can also be the updated first prompt word obtained after the most recent update of the first prompt word). In some embodiments, the image-generated text model, the text description reflection module, and the prompt word storage module together form a text description agent module, which is used to continuously optimize the prompt words stored in the prompt word storage module according to the text description information generated by the image-generated text model, wherein the agent refers to an intelligent system that can make decisions and perform actions autonomously in a certain environment to complete specific tasks or goals.

[0030] S104, inputting the first text description information into the proxy text image model, obtaining an output image corresponding to the first text description information output by the proxy text image model; optimizing the first prompt word by performing risk assessment on the output image.

[0031] In some embodiments, the Wenshengtu model is an artificial intelligence model that can generate images based on text descriptions. It belongs to the category of cross-modal models and can convert information between different sensory modalities. The main function of the Wenshengtu model is to generate images that match the provided text description. This type of model usually uses a deep learning architecture, such as a diffusion model, to generate images by gradually adding and removing noise. In some embodiments, the proxy Wenshengtu model refers to a Wenshengtu model that is different from the actual Wenshengtu model used for downstream tasks and is used to simulate the actual Wenshengtu model. This specification does not limit the model structure and model parameters of the proxy Wenshengtu model. In some embodiments, the first text description information is input into the proxy Wenshengtu model, and the output image corresponding to the first text description information output by the Wenshengtu model is obtained. The proxy Wenshengtu model is used to generate an image associated with the description through the description of the first text description information.

[0032] In some embodiments, the image reflection module performs risk assessment on the output image of the proxy text image model, that is, assesses whether the output image has risks, and obtains corresponding assessment results, which include but are not limited to indication information for indicating whether the output image has risks, risk content in the output image, and risk level corresponding to the risk content. In some embodiments, the output image can be manually (for example, an operator) to perform risk assessment to obtain the corresponding assessment result input manually, or the output image can be input into a trained risk assessment module to obtain the assessment result output by the module. In some embodiments, the image reflection module optimizes the initial first prompt word based on the obtained assessment result, and the purpose of the optimization operation is to enable the text description information generated by the image-generated text model to guide the proxy text image to generate a more risky output image. For example, if the assessment result includes that the risk level corresponding to the risk content in the output image is less than or equal to a preset threshold, the purpose of the corresponding optimization operation can be to make the text description information generated by the image-generated text model less obscure (that is, more semantically straightforward), and the optimization operation is not limited to adding prompt information in the first prompt word to indicate that the image-generated text model generates a more risky output image. In some embodiments, the image reflection module generates a corresponding optimization plan based on the obtained evaluation result, and optimizes the first prompt word according to the optimization plan. The optimization plan includes but is not limited to adding a prompt text to the first prompt word to guide the image-to-text model to generate corresponding text description information with a lower degree of obscurity (i.e., a higher degree of semantic directness) and associated with the image content of the risk image. For example, if the evaluation result corresponding to the output image is that the risk level is lower than a preset threshold, then the optimization plan generated based on the evaluation result is "adding sentence H1". The latest first prompt word obtained after optimizing the current first prompt word according to the optimization plan may be the prompt word obtained after adding sentence H1 to the original first prompt word.

[0033] In some embodiments, the prompt word storage (Memory) module is used to cache and update the first prompt word corresponding to the risk image, that is, the prompt word storage module stores the initial first prompt word corresponding to the risk image or the latest first prompt word obtained after updating the current first prompt word (the current first prompt word can be the initial first prompt word, or it can also be the updated first prompt word obtained after the most recent update of the first prompt word). In some embodiments, the agent Wensheng graph large model, the image reflection module, and the prompt word storage module together form an image intelligent agent (Agent) module, which is used to continuously optimize the prompt words stored in the prompt word storage module according to the output image generated by the agent Wensheng graph large model, wherein the agent refers to an intelligent system that can make decisions and perform actions autonomously in a certain environment to complete specific tasks or goals.

[0034] S106, determining whether the output image has risks; if so, saving the first text description information.

[0035] In some embodiments, it is determined whether an output image generated by the proxy text image model is risky. The specific determination method may be manual (for example, an operator) determination, or the output image may be input into a trained image risk detection model to obtain a model output result output by the image risk detection model to indicate whether the output image is risky. This specification does not limit the model structure and model parameters of the image risk detection model.

[0036] In some embodiments, if it is determined that the output image does not have a risk, it is further determined that the first text description information corresponding to the output image has not successfully attacked the proxy Wensheng graph model, and the first text description information is directly discarded. In some embodiments, if it is determined that the output image has a risk, it is further determined that the first text description information corresponding to the output image has successfully attacked the proxy Wensheng graph model, and the first text description information is saved.

[0037] S108, re-inputting the risk image and the optimized first prompt word into the image-to-text large model, repeating this cycle to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes a plurality of first text description information.

[0038] In some embodiments, the risk image and the optimized current latest first prompt word are re-input into the image-generated text model to obtain the current latest first text description information corresponding to the risk image output by the image-generated text model, the risk obscurity of the current latest first text description information is re-evaluated, the current latest first prompt word is optimized, the current latest first text description information is re-input into the proxy text-generated image model to obtain the current latest output image corresponding to the first text description information output by the proxy text-generated image model, the risk of the current latest output image is re-evaluated, the current latest first prompt word is optimized, and the current latest output image is re-determined. Is there a risk? If so, save the current latest output image; otherwise, directly discard the current latest output image, and repeat this process. Continuously optimize the current latest first prompt word according to the latest first text description information currently generated by the image-generated text model, and continuously optimize the current latest first prompt word according to the latest output image currently generated by the proxy image-generated text model. Continue to save the latest first text description information that successfully attacks the proxy image-generated text model (that is, the first text description information corresponding to the current latest output image with a risk), so as to obtain the image-generated text adversarial attack sample, which includes multiple saved first text description information. In the embodiments of this specification, a text-based image big model counterattack based on multi-agent (text description agent and image agent) is proposed, and two agents are used to continuously optimize the prompt words corresponding to the risk pictures, and a large number of diversified counterattack samples are continuously generated through the image-based text big model, and then the agent text-based image big model is used to test and retain the samples of successful attack, thereby obtaining text-based image counterattack samples containing multiple text description information. The agent can use more optimized technology to calculate the smallest disturbance that has the greatest impact on model prediction, improve the accuracy and success rate of the attack, and therefore the corresponding defense will become more and more difficult. By integrating multiple agents and utilizing the autonomy and responsiveness of the agents, the prompt words corresponding to the risk pictures are continuously optimized, generating more diverse and complex high-precision counterattack samples, breaking the conventional thinking limitations of using experience and rules, and greatly improving the efficiency of text-based image counterattacks.

[0039] In some embodiments, the loop end condition may be that the number of completed loops reaches a preset number threshold, or it may be that the loop duration is greater than or equal to a preset duration threshold, or it may be that the image similarity between the current latest output image generated in this loop and at least one output image generated in the most recent preset number of loops (for example, the previous loop, the previous two loops, etc.) is greater than or equal to a preset similarity threshold. It should be noted that the above-mentioned loop end conditions are only examples and not limitations. Those skilled in the art should understand that any form of loop end conditions may be included in the protection scope of this specification.

[0040] In the embodiment of the present specification, the risk image and the corresponding prompt word are input into the image-generated text model to generate the corresponding text description information, and then the text description information is input into the proxy text-generated image model to generate the corresponding output image, and then it is determined whether the output image successfully attacks the proxy text-generated image model. If successful, the text description information is saved, otherwise it is discarded. In this process, the prompt word is optimized according to the generated text description information and the output image, and the above process is repeated to continuously generate adversarial attack samples, which can greatly increase the number of adversarial attack samples generated, so that the attack samples and risk images are more generalized while maintaining consistency in content. By evaluating the text description information and the output image, the prompt word is continuously optimized to generate more diverse and complex high-precision adversarial attack samples, breaking the conventional By utilizing the thinking limitations of experience and rules, the efficiency of text-to-image adversarial attacks is greatly improved. No manual intervention is required, which can reduce labor costs. The internal process is not complicated and the amount is small. As long as it is running, a large number of adversarial attack samples can be continuously generated. The adversarial attack samples generated by the optimized prompt words are more aggressive and the expression form is more obscure. After the image-to-text conversion, the generated text description information is more natural to the model, and the attack success rate will be higher. The generated adversarial attack samples do not rely on human experience and any rules. After evaluating the text description information, the prompt words are optimized, making the generated text description information more obscure and complex, which can improve the accuracy and success rate of the attack. The text description information comes from the large image-to-text model. Compared with the text generated by rules, the data distribution is more natural and the generalization is better. Not only has the generation and efficiency of adversarial attack samples in the large image-to-text model been greatly improved, but also the quality and success rate of the attack samples have been significantly improved.

[0041] In some embodiments, the step of optimizing the first prompt word by evaluating the risk obscurity of the first text description information includes: determining at least one risk word in the first text description information by evaluating the risk obscurity of the first text description information, wherein the obscurity corresponding to the at least one risk word satisfies a first preset condition; optimizing the first prompt word according to the at least one risk word. In some embodiments, the risk obscurity evaluation refers to evaluating the risk obscurity of the first text description information, that is, evaluating whether there is at least one risk word with a low corresponding obscurity (that is, a high corresponding semantic directness) in the first text description information. In some embodiments, the risk obscurity evaluation of the first text description information obtains a corresponding evaluation result, which includes at least one risk word in the first text description information, and the obscurity corresponding to the at least one risk word satisfies a first preset condition, wherein the first preset condition may refer to that the obscurity corresponding to the at least one risk word is lower than or equal to a preset threshold, or may refer to that the at least one risk word is a preset number of risk words with the lowest corresponding obscurity among all risk words in the first text description information, and this specification does not limit the first preset condition. In some embodiments, based on the one less risk word, the current latest first prompt word is optimized. The purpose of the optimization operation is to make the large graph-generated text model avoid using the at least one risk word as much as possible in the text description information it generates, and to maximize the risk obscurity of the text description information generated by the large graph-generated text model. The optimization operation includes but is not limited to adding prompt information in the first prompt word to instruct the large graph-generated text model to hide the at least one risk word in the generated text description information, or adding prompt information in the first prompt word to instruct the large graph-generated text model to replace the at least one risk word with other words in the generated text description information. In some embodiments, a corresponding optimization scheme is first generated based on the at least one risk word, and then the current latest first prompt word is optimized according to the optimization scheme. The optimization scheme includes but is not limited to hiding the at least one risk word, replacing the at least one risk word with other words, etc. This specification does not limit the first preset condition.

[0042] In some embodiments, optimizing the first prompt word according to the at least one risk word includes: obtaining a replacement word corresponding to the at least one risk word, wherein the semantic similarity between the at least one risk word and the replacement word satisfies a second preset condition, and the degree of obscurity corresponding to the replacement word satisfies a third preset condition; optimizing the first prompt word according to the at least one risk word and the replacement word. In some embodiments, based on the at least one risk word, a replacement word corresponding to the at least one risk word is first determined. The replacement word corresponding to the risk word may be determined for each risk word in the at least one risk word, or the replacement word corresponding to the risk word may be determined only for some of the at least one risk word, that is, one or more risk words in the at least one risk word have not determined a corresponding replacement word, wherein the semantic similarity between the risk word and the corresponding replacement word satisfies a second preset condition, which may refer to that the semantic similarity between the risk word and the corresponding replacement word is greater than or equal to a preset threshold, and the present specification does not impose any restriction on the second preset condition, wherein the degree of obscurity corresponding to the replacement word satisfies a third preset condition, which may refer to that the degree of obscurity corresponding to the replacement word is higher than or equal to a preset threshold, or the degree of obscurity corresponding to the replacement word is higher than the degree of obscurity of the risk word corresponding to the replacement word, and the present specification does not impose any restriction on the third preset condition. In some embodiments, the first prompt word is optimized based on the at least one risk word and the replacement word. For a risk word for which a corresponding replacement word exists in the at least one risk word, the optimization operation on the first prompt word includes adding prompt information in the first prompt word for instructing the graph-generated-text model to replace the risk word with its corresponding replacement word in the generated text description information. For a risk word for which a corresponding replacement word does not exist in the at least one risk word, the optimization operation on the first prompt word includes adding prompt information in the first prompt word for instructing the graph-generated-text model to hide the risk word in the generated text description information.

[0043] In some embodiments, the optimization of the first prompt word according to the at least one risk word and the replacement word includes: optimizing the first prompt word according to the at least one risk word and the replacement word, and in combination with at least one picture description object corresponding to the risk picture. In some embodiments, the risk picture is subjected to content recognition to determine at least one picture description object in the risk picture (i.e., the object described in the risk picture). In some embodiments, for each risk word in the at least one risk word, a plurality of words whose semantic similarity with the risk word is less than or equal to a preset threshold can be obtained based on the semantics of the risk word, and then the words with the highest degree of association with the at least one picture description object can be selected from the plurality of words as the replacement words corresponding to the risk word, and then the corresponding optimization operation includes adding prompt information in the first prompt word for instructing the image-generated text model to replace the risk word with its corresponding replacement word in the generated text description information. In some embodiments, for each risk word of the at least one risk word, after obtaining the replacement word corresponding to the risk word, the replacement word is modified according to the at least one picture description object, and prompt information associated with the at least one picture description object is obtained, which is used to instruct the picture-to-text model to replace the risk word with its corresponding replacement word in the generated text description information, and then the corresponding optimization operation includes adding the prompt information to the first prompt word.

[0044] In some embodiments, the optimizing the first prompt word according to the at least one risk word and the replacement word, in combination with at least one picture description object corresponding to the risk picture, includes: determining a first target picture description object associated with the replacement word from the at least one picture description object corresponding to the risk picture; optimizing the first prompt word according to the at least one risk word and the replacement word, in combination with the first target picture description object. In some embodiments, the risk picture is content identified to determine at least one picture description object in the risk picture (i.e., the object described in the risk picture), and for each risk word in the at least one risk word, after obtaining the replacement word corresponding to the risk word, the first target picture description object associated with the replacement word is screened out from the at least one picture description object, i.e., the correlation between the first target picture description object and the replacement word meets a preset condition, and the preset condition may be that the first target picture description object is the picture description object with the highest correlation with the replacement word among the at least one picture description object, and this specification does not limit the preset condition. In some embodiments, for each risk word in the at least one risk word, after obtaining the replacement word corresponding to the risk word, the replacement word is modified according to the first target picture description object associated with the replacement word in the at least one picture description object, and prompt information associated with the first target picture description object is obtained, which is used to instruct the picture-to-text model to replace the risk word with its corresponding replacement word in the generated text description information, and then the corresponding optimization operation includes adding the prompt information to the first prompt word.

[0045] In some embodiments, the optimizing the first prompt word by performing risk assessment on the output image includes: performing risk assessment on the output image to obtain a risk assessment result corresponding to the output image, wherein the risk assessment result includes risk content information and / or risk degree information; optimizing the first prompt word according to the risk assessment result. In some embodiments, risk assessment refers to evaluating whether there is a risk in the output image, obtaining a risk assessment result corresponding to the output image, and the risk assessment result includes the risk content in the output image and / or the risk degree corresponding to the risk content. In some embodiments, the first prompt word is optimized according to the risk assessment result. The purpose of the optimization operation is to enable the text description information generated by the large picture-generated text model to guide the proxy text-generated image to generate a more risky output image (for example, to make the text description information generated by the large picture-generated text model less obscure). The optimization operation is not limited to adding prompt information in the first prompt word to indicate that the large picture-generated text model can generate a more risky output image, wherein the prompt information matches the risk content and / or the risk level. The matching of the prompt information and the risk content means that the semantic correlation between the two is greater than a preset threshold, and the matching of the prompt information and the risk level means that the prompt information is important for generating The semantic guidance degree of the risk output image matches the risk degree. For example, the two are inversely proportional. The lower the risk degree, the higher the semantic guidance degree of the corresponding prompt information for generating the risk output image. The higher the risk degree, the lower the semantic guidance degree of the corresponding prompt information for generating the risk output image. Alternatively, the prompt information matches the risk degree may also refer to determining whether it is necessary to add corresponding prompt information to the first prompt word based on the risk degree. If the risk degree is less than or equal to a preset threshold, it is necessary to add corresponding prompt information to the first prompt word, that is, it is necessary to optimize the first prompt word. If the risk degree is greater than the preset threshold, it is not necessary to add corresponding prompt information to the first prompt word, that is, it is not necessary to optimize the first prompt word.

[0046] In some embodiments, the risk assessment result includes risk content information; wherein, optimizing the first prompt word based on the risk assessment result includes: determining the risk guidance prompt information corresponding to the output picture based on the risk assessment result, wherein the correlation between the risk guidance prompt information and the risk content information satisfies a fourth preset condition; optimizing the first prompt word based on the risk guidance prompt information. In some embodiments, the risk assessment result includes the risk content present in the output image and / or the risk degree corresponding to the risk content. According to the risk assessment result, the risk guidance prompt information corresponding to the output image is determined, and the risk guidance prompt information is used to guide the image-to-text model to generate text description information that can make the proxy text-to-image generate a more risky output image (that is, guide the image-to-text model to generate less obscure text description information), wherein the correlation between the risk guidance prompt information and the risk content satisfies a fourth preset condition, and the fourth preset condition may refer to the correlation between the two being greater than or equal to a preset threshold. This specification does not limit the fourth preset condition, and then the first prompt word is optimized according to the risk guidance prompt information. The optimization operation includes but is not limited to adding the risk guidance prompt information to the first prompt word, adding the risk guidance prompt information to the first prompt word and adaptively adjusting the first prompt word based on the newly added content, etc. In some embodiments, if the risk assessment result only includes the risk content existing in the output image, risk guidance prompt information associated with the semantics of the risk content is generated according to the semantics of the risk content. If the risk assessment result also includes the risk level corresponding to the risk content, it is first determined whether to generate the corresponding risk guidance prompt information according to the risk level. Only when the risk level is less than or equal to a preset threshold value will the corresponding risk guidance prompt information be generated. Then, according to the semantics of the risk content, risk guidance prompt information associated with the semantics of the risk content is generated.

[0047] In some embodiments, the step of determining the risk guidance prompt information corresponding to the output image according to the risk assessment result includes: determining the risk guidance prompt information matching at least one image description object corresponding to the output image according to the risk assessment result. In some embodiments, the output image is subjected to content recognition to determine at least one image description object in the output image (i.e., the object described in the output image). In some embodiments, the risk guidance prompt information corresponding to the output image generated according to the risk assessment result matches at least one image description object in the output image, and the matching means that the correlation between the risk guidance prompt information and the at least one image description object is greater than or equal to a preset threshold. For example, the image description object in the output image includes a person, and the risk guidance prompt information corresponding to the output image generated may be "she / he is turning around", and the risk guidance prompt information matches the image description object, i.e., the correlation between the two is greater than or equal to a preset threshold. In some embodiments, the generated risk guidance prompt information not only matches at least one image description object corresponding to the output image, but also satisfies the fourth preset condition in terms of the correlation between the risk guidance prompt information and the risk content.

[0048] In some embodiments, the step of determining the risk guidance prompt information matching at least one picture description object corresponding to the output picture according to the risk assessment result includes: determining a second target picture description object associated with the risk content information from the at least one picture description object corresponding to the output picture; and determining the risk guidance prompt information matching the second target picture description object according to the risk assessment result. In some embodiments, content recognition is performed on the output picture to determine at least one picture description object in the output picture, and then the picture description object associated with the risk content (i.e., the degree of association between the two is greater than or equal to a preset threshold) is screened out from the at least one picture description object as the second target picture description object, and then the risk guidance prompt information matching the second target picture description object is generated, and the matching means that the degree of association between the risk guidance prompt information and the second target picture description object is greater than or equal to a preset threshold. In some embodiments, the generated risk guidance prompt information not only matches the second target picture description object, but also satisfies the fourth preset condition in terms of the degree of association between the risk guidance prompt information and the risk content.

[0049] Figure 2 A flowchart of a method for generating an adversarial attack sample is provided as an example in an embodiment of this specification.

[0050] like Figure 2As shown, the risk image and the prompt word (prompt) in the prompt word storage module are input into the image-generated text model to obtain the text description information (query) output by the image-generated text model, and the text description information is input into the proxy text-generated image model to obtain the output image output by the proxy text-generated image model, and it is judged whether the output image successfully attacks the proxy text-generated image model (i.e., it is judged whether the output image has risks). If the attack is successful, the text description information is saved as a text-generated image adversarial attack sample, otherwise, the text description information is directly discarded. In this process, the prompt word in the prompt word storage module is updated according to the generated text description information through the text description reflection module, and the prompt word in the prompt word storage module is updated according to the generated output image through the image reflection module. The above process is repeated to continuously generate text-generated image adversarial attack samples.

[0051] Figure 3 A schematic diagram of the structure of a device for generating an adversarial attack sample provided in an embodiment of this specification, the device for generating an adversarial attack sample (hereinafter referred to as "the adversarial attack sample generating device 1") can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the adversarial attack sample generating device 1 includes a first obtaining module 11, a second obtaining module 12, a risk determination module 13 and a third obtaining module 14.

[0052] The first obtaining module 11 is used to input the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; and optimize the first prompt word by evaluating the risk obscurity of the first text description information; The second obtaining module 12 is used to input the first text description information into the proxy Wenshengtu model, obtain the output image corresponding to the first text description information output by the proxy Wenshengtu model; and optimize the first prompt word by performing risk assessment on the output image; A risk determination module 13, used to determine whether the output image has a risk; if so, save the first text description information; The third obtaining module 14 is used to re-input the risk image and the optimized first prompt word into the image-to-text large model, and repeat this cycle to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes multiple first text description information.

[0053] In some embodiments, optimizing the first prompt word by evaluating the risk obscurity of the first text description information includes: Determine at least one risk word in the first text description information by evaluating the risk obscurity of the first text description information, wherein the obscurity corresponding to the at least one risk word satisfies a first preset condition; The first prompt word is optimized according to the at least one risk word.

[0054] In some embodiments, optimizing the first prompt word according to the at least one risk word includes: Obtaining a replacement word corresponding to the at least one risk word, wherein the semantic similarity between the at least one risk word and the replacement word satisfies a second preset condition, and the obscurity corresponding to the replacement word satisfies a third preset condition; The first prompt word is optimized according to the at least one risk word and the replacement word.

[0055] In some embodiments, optimizing the first prompt word according to the at least one risk word and the replacement word includes: The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture.

[0056] In some embodiments, optimizing the first prompt word according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture includes: Determine a first target picture description object associated with the replacement word from at least one picture description object corresponding to the risk picture; The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with the first target image description object.

[0057] In some embodiments, optimizing the first prompt word by performing risk assessment on the output image includes: By performing risk assessment on the output image, a risk assessment result corresponding to the output image is obtained, wherein the risk assessment result includes risk content information and / or risk degree information; The first prompt word is optimized according to the risk assessment result.

[0058] In some embodiments, the risk assessment result includes risk content information; The step of optimizing the first prompt word according to the risk assessment result includes: Determining risk guidance prompt information corresponding to the output image according to the risk assessment result, wherein the correlation between the risk guidance prompt information and the risk content information satisfies a fourth preset condition; The first prompt word is optimized according to the risk guidance prompt information.

[0059] In some embodiments, determining the risk guidance prompt information corresponding to the output image according to the risk assessment result includes: According to the risk assessment result, risk guidance prompt information matching at least one image description object corresponding to the output image is determined.

[0060] In some embodiments, determining risk guidance prompt information matching at least one image description object corresponding to the output image according to the risk assessment result includes: Determine a second target picture description object associated with the risk content information from at least one picture description object corresponding to the output picture; According to the risk assessment result, risk guidance prompt information matching the object described in the second target image is determined.

[0061] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0062] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0063] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0064] The embodiments of this specification also provide Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of the embodiment of this specification.

[0065] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0066] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0071] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0072] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0073] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for generating an adversarial attack sample, comprising: Input the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; By evaluating the risk obscurity of the first text description information, the first prompt word is optimized; Input the first text description information into the proxy Wenshengtu macro model to obtain an output image corresponding to the first text description information output by the proxy Wenshengtu macro model; optimize the first prompt word by performing risk assessment on the output image; Determine whether the output image has risks; If yes, save the first text description information; The risk image and the optimized first prompt word are re-input into the image-to-text large model, and this cycle is repeated to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes a plurality of first text description information.

2. The method according to claim 1, wherein optimizing the first prompt word by evaluating the risk obscurity of the first text description information comprises: Determine at least one risk word in the first text description information by evaluating the risk obscurity of the first text description information, wherein the obscurity corresponding to the at least one risk word satisfies a first preset condition; The first prompt word is optimized according to the at least one risk word.

3. The method according to claim 2, wherein optimizing the first prompt word according to the at least one risk word comprises: Obtaining a replacement word corresponding to the at least one risk word, wherein the semantic similarity between the at least one risk word and the replacement word satisfies a second preset condition, and the obscurity corresponding to the replacement word satisfies a third preset condition; The first prompt word is optimized according to the at least one risk word and the replacement word.

4. The method according to claim 3, wherein optimizing the first prompt word according to the at least one risk word and the replacement word comprises: The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture.

5. The method according to claim 4, wherein optimizing the first prompt word according to the at least one risk word and the replacement word and in combination with at least one picture description object corresponding to the risk picture comprises: Determine a first target picture description object associated with the replacement word from at least one picture description object corresponding to the risk picture; The first prompt word is optimized according to the at least one risk word and the replacement word and in combination with the first target image description object.

6. The method according to claim 1, wherein optimizing the first prompt word by performing risk assessment on the output image comprises: By performing risk assessment on the output image, a risk assessment result corresponding to the output image is obtained, wherein the risk assessment result includes risk content information and / or risk degree information; The first prompt word is optimized according to the risk assessment result.

7. The method according to claim 6, wherein the risk assessment result includes risk content information; in, Optimizing the first prompt word according to the risk assessment result includes: Determining risk guidance prompt information corresponding to the output image according to the risk assessment result, wherein the correlation between the risk guidance prompt information and the risk content information satisfies a fourth preset condition; The first prompt word is optimized according to the risk guidance prompt information.

8. The method according to claim 7, wherein determining the risk guidance prompt information corresponding to the output image according to the risk assessment result comprises: According to the risk assessment result, risk guidance prompt information matching at least one image description object corresponding to the output image is determined.

9. The method according to claim 8, wherein determining risk guidance prompt information matching at least one image description object corresponding to the output image according to the risk assessment result comprises: Determine a second target picture description object associated with the risk content information from at least one picture description object corresponding to the output picture; According to the risk assessment result, risk guidance prompt information matching the object described in the second target image is determined.

10. A device for generating an adversarial attack sample, comprising: A first obtaining module is used to input the risk picture and the initial first prompt word into the image-to-text model to obtain the first text description information corresponding to the risk picture output by the image-to-text model; and optimize the first prompt word by evaluating the risk obscurity of the first text description information; The second obtaining module is used to input the first text description information into the proxy Wenshengtu macro model, obtain an output image corresponding to the first text description information output by the proxy Wenshengtu macro model; and optimize the first prompt word by performing risk assessment on the output image; A risk determination module, used to determine whether the output image has risks; If yes, save the first text description information; The third acquisition module is used to re-input the risk image and the optimized first prompt word into the image-to-text large model, and repeat this cycle to obtain a text-to-image adversarial attack sample, wherein the text-to-image adversarial attack sample includes multiple first text description information.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 9.

13. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

Images