Black-box controllable text-to-image oriented multi-modal privacy image generation method and device

Abstract

The application relates to the technical field of artificial intelligence security and privacy protection, and provides a multi-modal private image generation method and device for black-box controllable text-to-image. The application generates an obfuscated prompt word and an obfuscated reference image, thereby ensuring the consistency of the text and the image in the semantic level through the joint action of the prompt word camouflage and the visual barrier trick, and preventing the privacy inference leakage across the modalities. The context prompt word and the obfuscated prompt word are combined with the corresponding context reference image and the obfuscated reference image, and are respectively sent to a black-box controllable text-to-image model, so that the multi-branch distributed request is utilized, the single service provider cannot obtain the complete user intention, and the privacy security is significantly improved. Through the strategy of local generation and late fusion, the image quality decline caused by the traditional noise adding method is avoided, so that the application can generate high-quality images with accurate structure and unified style.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence security and privacy protection technology, and in particular to a method and apparatus for generating multimodal privacy images for black-box controllable text-to-image processing. Background Technology

[0002] With the rapid development of deep learning technology, controllable text-to-image generation technology has been widely applied in fields such as artistic creation and industrial design. This type of technology allows users to precisely control the content and layout of the generated image by inputting prompts and reference images (such as sketches and edge maps). High-performance models are typically provided as "black-box" APIs (Application Programming Interfaces). However, this model carries a serious risk of privacy leakage. User input often contains sensitive information. Existing privacy protection methods mainly suffer from the following limitations: most methods rely on the white-box assumption, making them unsuitable for commercial APIs; existing research focuses primarily on single-modal protection, failing to address the cross-modal leakage problem caused by the high coupling between text and images; and simple noise-addition methods often lead to a significant decrease in generation quality. Therefore, a technical solution that balances multimodal privacy protection and generation quality in a black-box environment is urgently needed. Summary of the Invention

[0003] Therefore, it is necessary to provide a method and apparatus for generating multimodal privacy images for black-box controllable text-to-image processing, addressing the aforementioned technical problems.

[0004] A method for generating multimodal privacy-preserving images for black-box controllable raw images includes the following steps:

[0005] Obtain the original prompt words and original reference image, extract the sensitive keywords from the original prompt words, and decompose the original reference image into the background context region and the key visual region corresponding to the sensitive keywords;

[0006] Generate obfuscated keywords that are semantically unrelated to sensitive keywords and have reasonable style descriptions; use the original prompt words with sensitive keywords removed as context prompt words; use the original prompt words with only sensitive keywords retained and the rest replaced with obfuscated keywords as obfuscated prompt words; and construct context reference images and obfuscated reference images corresponding to the context prompt words and obfuscated prompt words based on the original reference images;

[0007] The contextual cue words and obfuscation cue words are combined with the corresponding contextual reference images and obfuscation reference images, and sent to the black-box controllable text-to-image model to generate the corresponding context-generated image and obfuscation-generated image.

[0008] Extract the generated regions corresponding to sensitive keywords from the obfuscated generated image, and merge the generated regions into the corresponding positions in the context generated image to obtain the target generated image.

[0009] In one embodiment, the obfuscated keywords are generated according to the following steps:

[0010] By spatial search, words orthogonal to the text encoding feature vectors of the sensitive keywords are obtained as obfuscated keywords.

[0011] In one embodiment, constructing context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words based on the original reference image includes:

[0012] Use the original reference image with key visual regions removed as the context reference image;

[0013] Based on the obfuscated keywords, obfuscated visual elements are generated and then merged into the non-critical visual regions of the original reference image, which retains only the critical visual regions, to obtain an obfuscated reference image.

[0014] In one embodiment, generating obfuscated visual elements based on the obfuscated keywords, and fusing the obfuscated visual elements into non-critical visual regions of an original reference image that retains only the critical visual regions, to obtain an obfuscated reference image, further includes:

[0015] A structured visual representation is generated based on the obfuscated keywords, and the structured visual representation is aligned and filled into the non-critical visual regions of the original reference image, which retains only the critical visual regions.

[0016] In one embodiment, a multi-endpoint distribution mechanism is used to combine contextual prompts and obfuscation prompts with corresponding contextual reference images and obfuscation reference images, and send them to black-box controllable text-to-image models at different server endpoints.

[0017] In one embodiment, the fusion algorithm that fuses the generated region to the corresponding position in the context-generated image includes a Poisson fusion algorithm or a gradient domain fusion algorithm.

[0018] In one embodiment, using the original prompt words (with sensitive keywords removed) as context prompt words, the method further includes:

[0019] If the sentence structure is incomplete after removing sensitive keywords, fill it with a neutral, random noun.

[0020] In one embodiment, the original reference image is decomposed into a background context region and key visual regions corresponding to sensitive keywords, including:

[0021] Based on sensitive keywords, a word target detection algorithm is used to search in the original reference image to locate the region corresponding to the sensitive keywords as the key visual region, and the original reference image with the key visual region removed is used as the background context region.

[0022] In one embodiment, the spatial search is based on a large language model or word vectors.

[0023] A multimodal privacy image generation device for black-box controllable raw images includes:

[0024] The data acquisition module is used to acquire the original prompt words and the original reference image, extract the sensitive keywords from the original prompt words, and decompose the original reference image into the background context region and the key visual region corresponding to the sensitive keywords.

[0025] The branch construction module is used to generate obfuscated keywords that are semantically unrelated to sensitive keywords and have reasonable style descriptions; the original prompt words with sensitive keywords removed are used as context prompt words; the original prompt words with only sensitive keywords retained and the rest replaced with obfuscated keywords are used as obfuscated prompt words; and the module constructs context reference images and obfuscated reference images corresponding to the context prompt words and obfuscated prompt words based on the original reference images.

[0026] The image generation module is used to combine the context prompts and obfuscation prompts with the corresponding context reference images and obfuscation reference images, and send them to the black-box controllable text-to-image model to generate the corresponding context-generated image and obfuscation-generated image.

[0027] The image fusion module is used to extract the generation region corresponding to the sensitive keywords in the obfuscated generated image and fuse the generated region into the corresponding position in the context generated image to obtain the target generated image.

[0028] The aforementioned method and apparatus for generating multimodal privacy images for black-box controllable text-to-image models ensures semantic consistency between text and images through the combined effects of "cue word camouflage" and "visual illusion," preventing cross-modal privacy inference leaks. By generating obfuscated cue words and obfuscated reference images, and sending them separately to the black-box controllable text-to-image model, multi-branch distributed requests prevent a single service provider from obtaining complete user intent, significantly improving privacy security. By employing a strategy of local generation and post-fusion, the image quality degradation caused by traditional noise-adding methods is avoided, enabling the invention to generate high-quality images with accurate structure and consistent style.

[0029] This invention can protect the user's creative intent and privacy information by decoupling and obfuscating the input data without accessing the internal parameters of the black-box controllable text image model. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a multimodal privacy image generation method for black-box controllable raw images in one embodiment;

[0031] Figure 2 This is a structural block diagram of a multimodal privacy image generation device for black-box controllable text-to-image generation in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The multimodal privacy image generation method for black-box controllable raw images provided in this application is deployed on a user terminal. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0034] Since it is deployed on the user terminal, this application does not require access to or modification of the internal parameters of the generated model, and can perfectly adapt to various commercial closed-source APIs.

[0035] In one embodiment, such as Figure 1 As shown, a multimodal privacy image generation method for black-box controllable raw images is provided, including the following steps:

[0036] Step 201: Obtain the original prompt words and the original reference image, extract the sensitive keywords from the original prompt words, and decompose the original reference image into the background context region and the key visual region corresponding to the sensitive keywords.

[0037] It should be noted that the original prompts and original reference images were input by the user. Specifically, for the extraction of sensitive keywords, natural language processing techniques were used to break down the original prompts into style descriptions (e.g., "oil painting style"), background context (e.g., "in a park"), and sensitive keywords (i.e., the core objects that the user wants to highlight and keep confidential, such as "a specific unreleased product").

[0038] Step 202: Generate obfuscated keywords that are semantically unrelated to the sensitive keywords and have reasonable style descriptions; use the original prompt words with the sensitive keywords removed as context prompt words; use the original prompt words with only the sensitive keywords retained and the rest replaced by obfuscated keywords as obfuscated prompt words; and construct context reference images and obfuscated reference images corresponding to the context prompt words and obfuscated prompt words based on the original reference images.

[0039] Understandably, the original prompts, which retain only the sensitive keywords and replace the rest with obfuscated keywords, are used as obfuscated prompts. While retaining the sensitive keywords (i.e. the user's true intent), the surrounding background context is completely replaced with obfuscated keywords. This ensures that although the subsequent black-box controllable text-to-image model generates the target object, it is in a completely fake semantic environment, thus hiding and protecting the user's true intent.

[0040] Step 203: Combine the context prompts and obfuscation prompts with the corresponding context reference images and obfuscation reference images, and send them to the black-box controllable text-to-image model to generate the corresponding context-generated images and obfuscation-generated images.

[0041] Step 204: Extract the generation region corresponding to the sensitive keyword in the obfuscated generated image, and merge the generated region into the corresponding position in the context generated image to obtain the target generated image.

[0042] Understandably, the generated target image is a target image with a consistent style and complete content, and the image is visually highly consistent with the image generated directly using the original data, but the complete original data is not leaked throughout the process.

[0043] The aforementioned multimodal privacy image generation method for black-box controllable text-to-image processing ensures semantic consistency between text and images through the combined effects of "cue word camouflage" and "visual illusion," preventing cross-modal privacy inference leaks. By combining contextual and obfuscated cue words with their corresponding contextual and obfuscated reference images and sending them separately to the black-box controllable text-to-image model, multi-branch distributed requests prevent a single service provider from obtaining complete user intent, significantly improving privacy and security. Furthermore, by employing a strategy of local generation and post-fusion, the method avoids image quality degradation caused by traditional noise-adding methods, generating high-quality images with accurate structure and consistent style.

[0044] In one embodiment, step 201 involves decomposing the original reference image into a background context region and key visual regions corresponding to sensitive keywords, including:

[0045] Based on sensitive keywords, a word target detection algorithm is used to search in the original reference image to locate the region corresponding to the sensitive keywords as the key visual region, and the original reference image excluding the key visual region is used as the background context region.

[0046] The vocabulary target detection algorithm includes, but is not limited to, Grounding DINO.

[0047] In one embodiment, the obfuscated keywords are generated according to the following steps:

[0048] By spatial search, words orthogonal to the text encoding feature vectors of the sensitive keywords are obtained as obfuscated keywords.

[0049] Specifically, if the sensitive keyword is "sports car", the confusing keyword could be "rock".

[0050] In this embodiment, a set of words orthogonal to the text encoding feature vectors of sensitive keywords are generated as obfuscated keywords, so that the obfuscated keywords are semantically unrelated to the sensitive keywords, but are visually reasonable.

[0051] In one embodiment, the spatial search is based on a large language model or word vectors.

[0052] In one embodiment, using the original prompt words (with sensitive keywords removed) as context prompt words, the method further includes:

[0053] If the sentence structure is incomplete after removing sensitive keywords, fill it with a neutral, random noun.

[0054] In this embodiment, grammatical fluency is ensured by filling in neutral random nouns.

[0055] In one embodiment, step 202, constructing context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words based on the original reference image, includes:

[0056] Use the original reference image with key visual regions removed as the context reference image;

[0057] Based on the obfuscated keywords, obfuscated visual elements are generated and then merged into the non-critical visual regions of the original reference image, which retains only the critical visual regions, to obtain an obfuscated reference image.

[0058] It should be noted that removing key visual regions specifically can involve removal or masking. The original reference image from which key visual regions are removed in this step is an image that has undergone structured visual representation. Understandably, the context reference image and the obfuscated reference image are also images that have undergone structured visual representation.

[0059] Specifically, for the keyword "rock", the visual element used to obfuscate the keyword could be the texture of a rock.

[0060] In this embodiment, by fusing the generated obfuscated visual elements into the key visual regions of the original reference image, the resulting obfuscated reference image maintains a high degree of consistency with the semantics of the obfuscated prompts in terms of visual structure, thus preventing the generation of the generated image from failing due to modal conflicts.

[0061] In one embodiment, generating obfuscated visual elements based on the obfuscated keywords, and fusing the obfuscated visual elements into non-critical visual regions of an original reference image that retains only the critical visual regions, to obtain an obfuscated reference image, further includes:

[0062] A structured visual representation is generated based on the obfuscated keywords, and the structured visual representation is aligned and filled into the non-critical visual regions of the original reference image, which retains only the critical visual regions.

[0063] It should be noted that the structured visual representation is generated using an edge detection model or a structure generation model.

[0064] In this embodiment, by aligning and filling the non-critical visual regions of the original reference image that retains only the critical visual regions with the structured visual representation, the generated obfuscated reference image is made to maintain consistency with the semantics of the obfuscated prompts in terms of structural layout, thereby achieving dual camouflage of text and image at the input end.

[0065] In one embodiment, a multi-endpoint distribution mechanism is used to combine contextual prompts and obfuscation prompts with corresponding contextual reference images and obfuscation reference images, and send them to black-box controllable text-to-image models at different server endpoints.

[0066] It's understandable that different server endpoints can also be different API service accounts.

[0067] Specifically, the first task involves combining contextual cue words with a contextual reference image and sending this as the first task to a black-box controllable text-to-image (TPEA) model. The TPEA model generates a context-generated image containing only the background environment and lacking the core sensitive object. The second task involves combining obfuscated cue words with an obfuscated reference image and sending this as the second task to the TPEA model. The TPEA model then generates an obfuscated image containing the core sensitive object, but with a completely incorrect and disguised background. This results in cloud service providers only seeing fragmented and misleading snippets, unable to reconstruct the user's complete and true intent.

[0068] In this embodiment, a black-box controllable text-to-image model is used to send the combination of prompt words and images to different server endpoints through a multi-endpoint distribution mechanism, thereby blocking the path for the server to restore the user's original intent through association analysis, thus protecting the user's privacy intent.

[0069] In one embodiment, the fusion algorithm that fuses the generated region to the corresponding position in the context-generated image includes a Poisson fusion algorithm or a gradient domain fusion algorithm.

[0070] In this embodiment, the lighting and hue of the image edges can be automatically adjusted by using the Poisson fusion algorithm or the gradient domain fusion algorithm, so that the generated region can be seamlessly integrated into the corresponding position of the context-generated image, and the two parts can be perfectly merged.

[0071] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0072] In one embodiment, such as Figure 2 As shown, a multimodal privacy image generation apparatus for black-box controllable raw images is provided, comprising:

[0073] The data acquisition module 901 is used to acquire the original prompt words and the original reference image, extract the sensitive keywords in the original prompt words, and decompose the original reference image into the background context area and the key visual area corresponding to the sensitive keywords.

[0074] Branch construction module 902 is used to generate obfuscated keywords that are semantically unrelated to sensitive keywords and have reasonable style descriptions; the original prompt words with sensitive keywords removed are used as context prompt words; the original prompt words with only sensitive keywords retained and the rest replaced by obfuscated keywords are used as obfuscated prompt words; and the context reference image and obfuscated reference image corresponding to the context prompt words and obfuscated prompt words are constructed based on the original reference image.

[0075] The image generation module 903 is used to combine the context prompts and obfuscation prompts with the corresponding context reference images and obfuscation reference images, and send them to the black-box controllable text-to-image model to generate the corresponding context-generated image and obfuscation-generated image.

[0076] The image fusion module 904 is used to extract the generation region corresponding to the sensitive keyword in the obfuscated generated image and fuse the generated region into the corresponding position in the context generated image to obtain the target generated image.

[0077] Specific limitations regarding the multimodal privacy image generation device for black-box controllable raw images can be found in the limitations of the multimodal privacy image generation method for black-box controllable raw images mentioned above, and will not be repeated here. Each module in the aforementioned multimodal privacy image generation device for black-box controllable raw images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating multimodal privacy-preserving images for black-box controllable raw images, characterized in that, Includes the following steps: Obtain the original prompt words and original reference image, extract the sensitive keywords from the original prompt words, and decompose the original reference image into the background context region and the key visual region corresponding to the sensitive keywords; Generate obfuscated keywords that are semantically irrelevant to sensitive keywords and have reasonable style descriptions; Use the original suggestion words (after removing sensitive keywords) as context suggestion words; Use the original suggestion words, retaining only the sensitive keywords and replacing the rest with obfuscated keywords, as the obfuscated suggestion words; And based on the original reference image, context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words are constructed; Construct context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words based on the original reference images, including: Use the original reference image with key visual regions removed as the context reference image; Based on the obfuscated keywords, obfuscated visual elements are generated, and the obfuscated visual elements are merged into the non-critical visual regions of the original reference image that retains only the key visual regions to obtain an obfuscated reference image; Based on the obfuscated keywords, obfuscated visual elements are generated, and these obfuscated visual elements are then merged into the non-critical visual regions of the original reference image, which retains only the critical visual regions, to obtain an obfuscated reference image. The process also includes: A structured visual representation is generated based on the obfuscated keywords, and the structured visual representation is aligned and filled into the non-critical visual regions of the original reference image that retains only the critical visual regions. The contextual cue words and obfuscation cue words are combined with the corresponding contextual reference images and obfuscation reference images, and sent to the black-box controllable text-to-image model to generate the corresponding context-generated image and obfuscation-generated image. Extract the generated regions corresponding to sensitive keywords from the obfuscated generated image, and merge the generated regions into the corresponding positions in the context generated image to obtain the target generated image.

2. The multimodal privacy image generation method for black-box controllable text-based images according to claim 1, characterized in that, The obfuscated keywords are generated according to the following steps: By spatial search, words orthogonal to the text encoding feature vectors of the sensitive keywords are obtained as obfuscated keywords.

3. The multimodal privacy image generation method for black-box controllable text-based images according to claim 1, characterized in that, A multi-endpoint distribution mechanism is adopted to combine contextual prompts and obfuscation prompts with their corresponding contextual reference images and obfuscation reference images, and send them to different server endpoints in a black-box controllable text-to-image model.

4. The multimodal privacy image generation method for black-box controllable text-based images according to claim 1, characterized in that, Fusion algorithms that fuse the generated regions to the corresponding locations in the context-generated image include Poisson fusion algorithms or gradient domain fusion algorithms.

5. The multimodal privacy image generation method for black-box controllable text-based images according to claim 1, characterized in that, Using the original suggestion words (with sensitive keywords removed) as contextual suggestion words also includes: If the sentence structure is incomplete after removing sensitive keywords, fill it with a neutral, random noun.

6. The multimodal privacy image generation method for black-box controllable text-based images according to claim 1, characterized in that, The original reference image is decomposed into background context regions and key visual regions corresponding to sensitive keywords, including: Based on sensitive keywords, a word target detection algorithm is used to search in the original reference image to locate the region corresponding to the sensitive keywords as the key visual region, and the original reference image with the key visual region removed is used as the background context region.

7. The multimodal privacy image generation method for black-box controllable text-based images according to claim 2, characterized in that, The spatial search is based on a large language model or word vectors.

8. A multimodal privacy image generation device for black-box controllable text-based images, characterized in that, include: The data acquisition module is used to acquire the original prompt words and the original reference image, extract the sensitive keywords from the original prompt words, and decompose the original reference image into the background context region and the key visual region corresponding to the sensitive keywords. The branch building module is used to generate obfuscated keywords that are semantically unrelated to sensitive keywords and have reasonable stylistic descriptions. Use the original suggestion words (after removing sensitive keywords) as context suggestion words; Use the original suggestion words, retaining only the sensitive keywords and replacing the rest with obfuscated keywords, as the obfuscated suggestion words; And based on the original reference image, context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words are constructed; Construct context reference images and obfuscation reference images corresponding to context cue words and obfuscation cue words based on the original reference images, including: Use the original reference image with key visual regions removed as the context reference image; Based on the obfuscated keywords, obfuscated visual elements are generated, and the obfuscated visual elements are merged into the non-critical visual regions of the original reference image that retains only the key visual regions to obtain an obfuscated reference image; Based on the obfuscated keywords, obfuscated visual elements are generated, and these obfuscated visual elements are then merged into the non-critical visual regions of the original reference image, which retains only the critical visual regions, to obtain an obfuscated reference image. The process also includes: A structured visual representation is generated based on the obfuscated keywords, and the structured visual representation is aligned and filled into the non-critical visual regions of the original reference image that retains only the critical visual regions. The image generation module is used to combine the context prompts and obfuscation prompts with the corresponding context reference images and obfuscation reference images, and send them to the black-box controllable text-to-image model to generate the corresponding context-generated image and obfuscation-generated image. The image fusion module is used to extract the generation region corresponding to the sensitive keywords in the obfuscated generated image and fuse the generated region into the corresponding position in the context generated image to obtain the target generated image.

Images