Image generation method and system based on diffusion personalized generation model, and storage medium

Abstract

The invention provides an image generation method and system based on a diffusion personalized generation model, and a storage medium, and the method comprises the following steps: obtaining an adversarial disturbance image, building a diffusion personalized generation model, taking the adversarial disturbance image as the input of the diffusion personalized generation model, and generating an adversarial disturbance image; acquiring potential features of the anti-disturbance image according to an auto-encoder of the diffusion personalized generation model; performing calculation based on a first preset rule according to the potential features; calculating based on a second preset rule according to random noise of the diffusion personalized generation model; performing denoising processing on the potential features according to the alignment loss and the potential diffusion loss; and cutting the de-noised features and decoding the de-noised features through a decoder which diffuses the personalized generation model. Compared with the prior art, the method has the advantages that the damaged mapping relation between the confrontation disturbance image and the potential space is repaired through the alignment loss, the potential diffusion loss and the dynamic weight changing along with the time step, and the generation quality of the image is improved.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of image processing, and particularly relates to an image generation method and system based on a diffusion personalization generation model and a storage medium. BACKGROUND

[0002] With the rapid iteration of generative artificial intelligence technology, diffusion personalization generation models have been widely used in digital content creation, visual design, virtual asset generation, and other fields due to their powerful image generation capabilities. Among them, diffusion personalization generation models, as a branch of targeted optimization technology, can achieve "style customization" functions by learning the style and features of specific input images. For example, the DreamBooth model can learn the core features of target objects or styles through a small number of reference images, and the Textual Inversion model can achieve precise migration of specific styles by optimizing text embedding vectors. Such models greatly meet the user's demand for personalized digital content creation and promote the practical application of generative AI technology.

[0003] However, in the process of large-scale application of diffusion personalization generation models, their security and stability face serious challenges, especially the problem of functional failure caused by adversarial perturbation attacks. Adversarial perturbation refers to the addition of small pixel perturbations by malicious users to input reference images that are difficult to detect by human vision but can be recognized by the model. After careful design, such perturbations can mislead the feature learning and generation logic of diffusion personalization generation models, resulting in a significant difference between the model output and the style and features of the input reference image, and completely failing to achieve the expected style customization goal. This malicious attack not only destroys the normal user experience, but also causes the generated content to deviate significantly from the requirements of the application scenario, limiting the application of diffusion personalization generation models in fields with high security requirements such as professional design and brand content generation.

[0004] Although there are existing methods for adversarial attacks and defenses for general generation models in the prior art, applying them to diffusion personalization generation models still has significant limitations and cannot effectively solve the above security problems: Firstly, the current mainstream adversarial attack method relies heavily on destroying the mapping relationship of the input image to the model latent space. One of the core logics of the diffusion personalized generative model is to accurately map the style and features of the input image to the latent space and perform feature encoding. The existing attack method directly destroys the stability of the mapping process through targeted perturbation, so that the model cannot extract effective features from the input image, and thus the generated result is distorted. Although this attack method has strong pertinence, it also exposes the passivity of existing defense techniques in dealing with "mapping relationship destruction attacks", and it is difficult to restore the destroyed feature mapping from the root.

[0005] Secondly, the existing adversarial attack method often constructs adversarial perturbation only by the loss function of the diffusion personalized generative model itself. This kind of method only aims to maximize the model generation error, and does not fully consider the core requirement of "personalized feature transfer" in the diffusion personalized generative model, resulting in that the constructed adversarial perturbation can achieve the attack effect, but lacks deep utilization of the personalized learning mechanism of the model, and also makes the robustness of the attack insufficient, and the existing defense method lacks effective adaptive response strategy for this kind of single loss driven perturbation.

[0006] Finally, the current adversarial defense method is difficult to balance between removing adversarial perturbation and maintaining the generation quality of the diffusion personalized generative model. The existing defense means mostly try to eliminate adversarial perturbation by image denoising and feature correction, but due to the strong dependence of the diffusion personalized generative model on the detailed features of the input image, excessive denoising or feature correction operation will cause the loss of personalized style information in the input image, and thus the generated content of the model loses the customized characteristics; and if the defense strength is reduced to preserve the style information, the adversarial perturbation cannot be completely removed, and the model still faces the risk of functional failure.

[0007] In summary, the existing diffusion personalized generative model lacks effective response scheme when facing adversarial perturbation attack, and the above limitations of the existing adversarial attack and defense methods seriously restrict the security and reliability of the diffusion personalized generative model and the expansion of application scenarios.

[0008] Therefore, there is an urgent need for a new image generation method, system and storage medium based on diffusion personalized generative model to solve the above technical problems. SUMMARY The present application provides an image generation method, system and storage medium based on diffusion personalized generative model, aiming to reconstruct the mapping relationship of the image destroyed by malicious adversarial attack, ensure the image generation ability of the diffusion personalized generative model not to be affected by the adversarial perturbation, and improve the quality of image generation.

[0009] In a first aspect, the present application provides an image generation method based on diffusion personalized generative model, the image generation comprising the following steps: S1, an adversarial perturbation image is obtained, a diffusion personalized generation model is established, the adversarial perturbation image is taken as an input of the diffusion personalized generation model, and a latent feature of the adversarial perturbation image is obtained according to an autoencoder of the diffusion personalized generation model; S2, an alignment loss of the adversarial perturbation image is obtained by calculating based on a first preset rule according to the latent feature; S3, a latent diffusion loss is obtained by calculating based on a second preset rule according to random noise of the diffusion personalized generation model; S4, a denoising feature is obtained by denoising the latent feature according to the alignment loss and the latent diffusion loss; S5, the denoising feature is cropped and decoded by a decoder of the autoencoder to obtain an optimized image.

[0010] Preferably, the first preset rule is to make the alignment loss satisfy the following condition: ; wherein, denotes the alignment loss, denotes the decoder, denotes the adversarial perturbation image, denotes the latent feature, denotes a similarity measure index of the adversarial perturbation image.

[0011] Preferably, the second preset rule is to make the latent diffusion loss satisfy the following condition: ; wherein, denotes the latent diffusion loss, denotes random Gaussian noise added to the latent feature by the diffusion personalized generation model in each forward process, denotes the random noise, denotes a parameter of the diffusion personalized generation model, denotes a latent feature generated by the diffusion personalized generation model in the first t forward process, denotes a category of an image generated by the diffusion personalized generation model.

[0012] Preferably, step S4 further includes the following substeps: S41, a total loss is obtained by weighted calculation of the alignment loss and the latent diffusion loss according to a time step of the diffusion personalized generation model; S42, the denoising feature is obtained by denoising the latent feature according to the total loss.

[0013] Preferably, the total loss satisfies the following condition: ; ; wherein, denotes the current weight coefficient, denotes the initial weight coefficient, denotes the final weight coefficient, denotes the total loss, denotes the time step, denotes the total time step.

[0014] Preferably, step S5 further comprises the following sub-steps: S51, decoding the denoised feature according to the decoder to obtain a decoded image; S52, calculating the pixel difference between the decoded image and the adversarial interference image, and when the pixel difference is greater than a preset threshold, cropping the decoded image; S53, encoding the cropped decoded image according to the encoder of the autoencoder to obtain an encoded feature; S54, decoding the encoded feature again according to the decoder to obtain the optimized image.

[0015] Preferably, the decoded feature satisfies the following condition ; wherein, denotes the decoded feature, denotes the encoder, denotes the cropping range.

[0016] In a second aspect, the present application also provides an image generation system based on a diffusion personalized generation model, and the adversarial defense system comprises: an input module, configured to obtain an adversarial interference image, establish a diffusion personalized generation model, take the adversarial interference image as an input of the diffusion personalized generation model, and obtain a latent feature of the adversarial interference image according to an autoencoder of the diffusion personalized generation model; an alignment loss module, configured to calculate an alignment loss of the adversarial interference image based on a first preset rule according to the latent feature; a diffusion loss module, configured to obtain a latent diffusion loss based on a second preset rule according to random noise of the diffusion personalized generation model; a denoising module, configured to perform denoising processing on the latent feature according to the alignment loss and the latent diffusion loss to obtain a denoised feature; An optimization module is configured to crop the denoised feature and decode the denoised feature through a decoder of the diffusion personalized generative model to obtain an optimized image.

[0017] In a third aspect, the present application also provides a computer device, comprising a memory, a processor, and an image generation program based on a diffusion personalized generative model stored in the memory and executable on the processor, wherein the processor implements the steps of the image generation method based on the diffusion personalized generative model when executing the image generation program based on the diffusion personalized generative model.

[0018] In a fourth aspect, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores an image generation program based on a diffusion personalized generative model, and the image generation program based on the diffusion personalized generative model implements the steps of the image generation method based on the diffusion personalized generative model when executed by a processor.

[0019] Compared with the prior art, the image generation method based on the diffusion personalized generative model comprises the following steps: obtaining an adversarial perturbation image, establishing a diffusion personalized generative model, inputting the adversarial perturbation image into the diffusion personalized generative model, and obtaining latent features of the adversarial perturbation image according to an autoencoder of the diffusion personalized generative model; calculating the latent features based on a first preset rule to obtain an alignment loss of the adversarial perturbation image; calculating a latent diffusion loss based on a second preset rule according to random noise of the diffusion personalized generative model; performing denoising processing on the latent features according to the alignment loss and the latent diffusion loss to obtain denoised features; cropping the denoised features and decoding the denoised features through a decoder of the diffusion personalized generative model to obtain an optimized image. The present application repairs the mapping relationship between the adversarial perturbation image and the latent space that is destroyed by the alignment loss, the latent diffusion loss, and the dynamic weight that changes with time steps, and further improves the generation quality of the image by cropping the denoised features, so that the diffusion personalized generative model can still normally generate a style customized image similar to the input image when facing malicious adversarial attacks, and the generation quality of the image is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other aspects of the present application will become more apparent and more readily appreciated by referring to the following detailed description in conjunction with the accompanying drawings, in which: Figure 1 is a flow block diagram of the image generation method based on the diffusion personalized generative model provided by the embodiments of the present application; Figure 2 is a structural schematic diagram of the image generation system based on the diffusion personalized generative model provided by the embodiments of the present application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Example 1 Please refer to Figure 1 This invention provides an image generation method based on a diffusion-based personalized generation model, wherein the image generation includes the following steps: S1. Obtain an adversarial perturbation image, establish a diffusion-based personalized generation model, use the adversarial perturbation image as input to the diffusion-based personalized generation model, and obtain the latent features of the adversarial perturbation image based on the autoencoder of the diffusion-based personalized generation model.

[0023] In this embodiment of the invention, adversarial perturbation images refer to images generated by malicious users by adding adversarial perturbations to the input images of the diffusion personalization generation model, causing the model to generate images with styles that are very different from the input images, thus making it impossible to achieve the purpose of style customization.

[0024] S2. Based on the latent features and the first preset rule, calculate the alignment loss of the adversarial perturbation image.

[0025] In this embodiment of the invention, for an adversarial perturbation image, the latent features generated by the autoencoder of the diffusion-based personalized generative model have a significant semantic gap with the original adversarial perturbation image. To reduce this gap, this invention optimizes the latent features through alignment loss to align their semantics with the input adversarial perturbation image.

[0026] The first preset rule is: to make the alignment loss satisfy the following conditions: ; in, This represents the alignment loss. The decoder represents the autoencoder of the diffusion-based personalized generation model. This refers to the image that resists perturbation. This indicates the potential feature. This represents the similarity metric of the images against the perturbation.

[0027] The similarity metric is obtained by: comparing the adversarial perturbation images... Latent features in autoencoders One extra decoder input ,get ; after that and As input, deep features and semantic embeddings of the pre-trained visual encoder ViT and semantic encoder CLIP are extracted respectively. Then, a cross-modal attention fusion module is introduced, which uses the semantic embedding as the query and the deep features as the key-value pair to calculate the attention-weighted fusion features. The parameters involved in this process include the weights of the pre-trained model, the attention fusion matrix, and a parameter derived from the current time step. and text prompts Adaptive scaling factor dynamically generated by a small MLP network The adaptive scaling factor dynamically recalibrates the weights of different feature channels by multiplying the channel attention mechanism with the backbone features of the semantic awareness network. For example, at higher time steps, it prioritizes amplifying the responses of contour-related feature channels, while at lower time steps, it enhances the contribution of texture feature channels, thereby achieving contextualized adaptive adjustment of the network parameter utility. Finally, the similarity metric... The Wasserstein distance between the fused features obtained after processing two images through the above process is calculated, and this distance can better characterize the differences in the distribution level.

[0028] The above method achieves a leap from pixel-level alignment to semantic and perceptual-level alignment. Its dynamic characteristics ensure that the alignment emphasis of the diffusion-based personalized generation model can be adaptively adjusted at different denoising stages and under different generation instructions (e.g., emphasizing contour structure in the early stages and texture details in the later stages), thereby more accurately repairing semantic breaks caused by adversarial perturbations. Adaptive scaling factor. As the core of dynamic regulation, it can adjust the intensity of attention to different semantic feature dimensions within the network in real time based on the time step of the denoising process and the text prompts of the generation task.

[0029] S3. Based on the random noise of the diffusion personalized generation model, the potential diffusion loss is calculated according to the second preset rule.

[0030] In this embodiment of the invention, the random noise of the diffusion-based personalized generation model is predicted based on a neural network model. Specifically, an improved U-Net++ architecture is used as the backbone network, and a perturbation-aware attention gating mechanism is added to it. This network feeds forward... Noisy latent features of the step Time step Text conditions As input, the processing comprises three levels: feature extraction, perturbation awareness, and conditional generation. Specifically, the feature extraction level extracts multi-scale features through the encoder downsampling path and injects conditional information into each residual block using AdaGN. The perturbation awareness level, located at the bottleneck layer, utilizes an auxiliary perturbation region prediction subnetwork to generate a soft mask based on the current features. This mask identifies important regions in the feature map that may be affected by adversarial perturbations. The conditional generation level, located in the decoder upsampling path, uses the aforementioned mask as an attention gating mechanism, guiding the network to focus more precisely on estimating suspected perturbation regions when reconstructing noise. Finally, the U-Net++ network outputs conditionally focused predicted noise.

[0031] By predicting random noise using the aforementioned neural network model, noise prediction is no longer a blind global estimation, but rather a targeted, locally reinforced perturbation deconstruction process. Through explicit modeling and focusing on perturbation regions, the network can more effectively disrupt the structural features of adversarial perturbations while preserving the clean content of the image to the maximum extent. This network design, which couples "localization" and "repair," demonstrates deep optimization for adversarial defense scenarios using diffusion-based personalized generative models.

[0032] The second preset rule is: to ensure that the potential diffusion loss satisfies the following conditions: ; in, This represents the potential diffusion loss. This represents the random Gaussian noise added to the latent features by the diffusion-based personalized generation model at each forward step. This represents random noise predicted based on a neural network model. The parameters represent the diffusion-based personalized generation model. This represents the latent features generated by the diffusion-based personalized generation model during the t-th forward pass. This indicates the category of the image generated by the diffusion-based personalized generation model.

[0033] By optimizing the potential diffusion loss This can further repair the mapping relationship that has been damaged by malicious adversarial attacks.

[0034] S4. Denoise the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features.

[0035] In this embodiment of the invention, step S4 further includes the following sub-steps: S41. The alignment loss and potential diffusion loss are weighted and calculated according to the time step of the diffusion personalized generation model to obtain the total loss; S42. Denoise the latent features based on the total loss to obtain the denoised features.

[0036] In this embodiment of the invention, the total loss satisfies the following condition: ; ; in, Indicates the current weight coefficient. Indicates the initial weighting coefficients. This represents the final weighting coefficient. This represents the total loss. Indicates the time step, Indicates the total time step.

[0037] A time step of the diffusion-based personalized generation model was constructed. Linearly correlated dynamic loss weights This weight, acting as a precise scheduler, divides the denoising process for latent features into two distinct stages and achieves a smooth transition between them. At the beginning of the diffusion process (high... (Value phase) Set to a higher initial value This makes the potential diffusion loss in the total loss function... It occupies a relatively dominant position. However, as the denoising process continues, the time step... Gradually decrease, The value of the latent space-image alignment loss decreases linearly thereafter. It will take the dominant position.

[0038] This design enables high noise reduction in the early stages (high... In the value phase, the main objective of the optimization process is to minimize the potential diffusion loss. In this stage, noise constitutes a significant proportion of the potential features, making the structure relatively vulnerable to disturbances and hidden within the noise. Therefore, focused optimization is necessary. It can efficiently utilize its gradient to directly attack and dismantle perturbed structures. When the process enters the later stages of denoising (low... (Value phase) It has dropped to a low level. The proportion of noise in the total loss increases significantly, and the optimization focus shifts accordingly. At this point, a large amount of noise has been removed, the semantic content of the image is clarified, and the image is enhanced. Optimization can use its perceptual similarity metric to finely correct semantic biases or detail distortions that may be introduced by previous denoising, ensuring that the output is highly consistent with the expected content.

[0039] S5. The denoised features are cropped and decoded by the decoder of the diffusion personalized generation model to obtain an optimized image.

[0040] In this embodiment of the invention, step S5 further includes the following sub-steps: S51. Decode the denoising features according to the decoder to obtain a decoded image; S52. Calculate the pixel difference between the decoded image and the anti-interference image. When the pixel difference is greater than a preset threshold, crop the decoded image. By constraining the pixel level of the decoded image, the quality of subsequent image generation is improved.

[0041] S53. Encode the cropped decoded image according to the encoder of the autoencoder to obtain the encoded features; S54. The encoded features are decoded again according to the decoder to obtain the optimized image.

[0042] In this embodiment of the invention, the decoding feature satisfies the following conditions. ; in, This indicates the decoding feature. This refers to the encoder. This refers to the decoder. Indicates the area to be clipped.

[0043] Compared with existing technologies, the image generation method based on a diffusion-based personalized generation model proposed in this invention includes the following steps: acquiring an adversarial perturbation image, establishing a diffusion-based personalized generation model, using the adversarial perturbation image as input to the diffusion-based personalized generation model, acquiring latent features of the adversarial perturbation image based on the autoencoder of the diffusion-based personalized generation model; calculating the alignment loss of the adversarial perturbation image based on the latent features according to a first preset rule; calculating the latent diffusion loss based on the random noise of the diffusion-based personalized generation model according to a second preset rule; denoising the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features; cropping the denoised features and decoding them through the decoder of the diffusion-based personalized generation model to obtain an optimized image. This invention repairs the disrupted mapping relationship between the adversarial perturbation image and the latent space through alignment loss, latent diffusion loss, and dynamic weights that change with time step, and further improves the image generation quality by cropping the denoised features. This allows the diffusion-based personalized generation model to still generate style-customized images that are similar to the input image even when facing malicious adversarial attacks, thus improving the image generation quality. Example 2 This invention also provides an image generation system based on a diffusion-based personalized generation model. Please refer to [link / reference].Figure 2 , Figure 2 This is a schematic diagram of the structure of an image generation system 200 based on a diffusion-based personalized generation model provided in an embodiment of the present invention, which includes: 201. Input module, used to acquire an adversarial perturbation image, establish a diffusion personalized generation model, use the adversarial perturbation image as input to the diffusion personalized generation model, and acquire the latent features of the adversarial perturbation image according to the autoencoder of the diffusion personalized generation model; 202. Alignment loss module, used to calculate the alignment loss of the adversarial perturbation image based on the latent features and a first preset rule; 203. Diffusion loss module, used to calculate the potential diffusion loss based on the random noise of the personalized diffusion generation model according to the second preset rule; 204. A denoising module, used to denoise the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features; 205. Optimization module, used to crop the denoised features and decode them through the decoder of the diffusion personalized generation model to obtain an optimized image.

[0044] The image generation system 200 based on the diffusion-based personalized generation model can implement the steps in the image generation method based on the diffusion-based personalized generation model in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0045] Example 3 This invention also provides a computer device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and an image generation program based on a diffusion-based personalized generation model stored in the memory 302 and capable of running on the processor 301.

[0046] The processor 301 calls the image generation program based on the diffusion-based personalized generation model stored in the memory 302, and executes the steps in the image generation method based on the diffusion-based personalized generation model provided in this embodiment of the invention. Please refer to... Figure 1 Specifically, it includes the following steps: S1. Obtain an adversarial perturbation image, establish a diffusion-based personalized generation model, use the adversarial perturbation image as input to the diffusion-based personalized generation model, and obtain the latent features of the adversarial perturbation image based on the autoencoder of the diffusion-based personalized generation model.

[0047] In this embodiment of the invention, adversarial perturbation images refer to images generated by malicious users by adding adversarial perturbations to the input images of the diffusion personalization generation model, causing the model to generate images with styles that are very different from the input images, thus making it impossible to achieve the purpose of style customization.

[0048] S2. Based on the latent features and the first preset rule, calculate the alignment loss of the adversarial perturbation image.

[0049] In this embodiment of the invention, for an adversarial perturbation image, the latent features generated by the autoencoder of the diffusion-based personalized generative model have a significant semantic gap with the original adversarial perturbation image. To reduce this gap, this invention optimizes the latent features through alignment loss to align their semantics with the input adversarial perturbation image.

[0050] The first preset rule is: to make the alignment loss satisfy the following conditions: ; in, This represents the alignment loss. The decoder represents the autoencoder of the diffusion-based personalized generation model. This refers to the image that resists perturbation. This indicates the potential feature. This represents the similarity metric of the images against the perturbation.

[0051] The similarity metric is obtained by: comparing the adversarial perturbation images... Latent features in autoencoders One extra decoder input ,get ; after that and As input, deep features and semantic embeddings of the pre-trained visual encoder ViT and semantic encoder CLIP are extracted respectively. Then, a cross-modal attention fusion module is introduced, which uses the semantic embedding as the query and the deep features as the key-value pair to calculate the attention-weighted fusion features. The parameters involved in this process include the weights of the pre-trained model, the attention fusion matrix, and a parameter derived from the current time step. and text prompts Adaptive scaling factor dynamically generated by a small MLP network The adaptive scaling factor dynamically recalibrates the weights of different feature channels by multiplying the channel attention mechanism with the backbone features of the semantic awareness network. For example, at higher time steps, it prioritizes amplifying the responses of contour-related feature channels, while at lower time steps, it enhances the contribution of texture feature channels, thereby achieving contextualized adaptive adjustment of the network parameter utility. Finally, the similarity metric... The Wasserstein distance between the fused features obtained after processing two images through the above process is calculated, and this distance can better characterize the differences in the distribution level.

[0052] The above method achieves a leap from pixel-level alignment to semantic and perceptual-level alignment. Its dynamic characteristics ensure that the alignment emphasis of the diffusion-based personalized generation model can be adaptively adjusted at different denoising stages and under different generation instructions (e.g., emphasizing contour structure in the early stages and texture details in the later stages), thereby more accurately repairing semantic breaks caused by adversarial perturbations. Adaptive scaling factor. As the core of dynamic regulation, it can adjust the intensity of attention to different semantic feature dimensions within the network in real time based on the time step of the denoising process and the text prompts of the generation task.

[0053] S3. Based on the random noise of the diffusion personalized generation model, the potential diffusion loss is calculated according to the second preset rule.

[0054] In this embodiment of the invention, the random noise of the diffusion-based personalized generation model is predicted based on a neural network model. Specifically, an improved U-Net++ architecture is used as the backbone network, and a perturbation-aware attention gating mechanism is added to it. This network feeds forward... Noisy latent features of the step Time step Text conditions As input, the processing comprises three levels: feature extraction, perturbation awareness, and conditional generation. Specifically, the feature extraction level extracts multi-scale features through the encoder downsampling path and injects conditional information into each residual block using AdaGN. The perturbation awareness level, located at the bottleneck layer, utilizes an auxiliary perturbation region prediction subnetwork to generate a soft mask based on the current features. This mask identifies important regions in the feature map that may be affected by adversarial perturbations. The conditional generation level, located in the decoder upsampling path, uses the aforementioned mask as an attention gating mechanism, guiding the network to focus more precisely on estimating suspected perturbation regions when reconstructing noise. Finally, the U-Net++ network outputs conditionally focused predicted noise.

[0055] By predicting random noise using the aforementioned neural network model, noise prediction is no longer a blind global estimation, but rather a targeted, locally reinforced perturbation deconstruction process. Through explicit modeling and focusing on perturbation regions, the network can more effectively disrupt the structural features of adversarial perturbations while preserving the clean content of the image to the maximum extent. This network design, which couples "localization" and "repair," demonstrates deep optimization for adversarial defense scenarios using diffusion-based personalized generative models.

[0056] The second preset rule is: to ensure that the potential diffusion loss satisfies the following conditions: ; in, This represents the potential diffusion loss. This represents the random Gaussian noise added to the latent features by the diffusion-based personalized generation model at each forward step. This represents random noise predicted based on a neural network model. The parameters represent the diffusion-based personalized generation model. This represents the latent features generated by the diffusion-based personalized generation model during the t-th forward pass. This indicates the category of the image generated by the diffusion-based personalized generation model.

[0057] By optimizing the potential diffusion loss This can further repair the mapping relationship that has been damaged by malicious adversarial attacks.

[0058] S4. Denoise the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features.

[0059] In this embodiment of the invention, step S4 further includes the following sub-steps: S41. The alignment loss and potential diffusion loss are weighted and calculated according to the time step of the diffusion personalized generation model to obtain the total loss; S42. Denoise the latent features based on the total loss to obtain the denoised features.

[0060] In this embodiment of the invention, the total loss satisfies the following condition: ; ; in, Indicates the current weight coefficient. Indicates the initial weighting coefficients. This represents the final weighting coefficient. This represents the total loss. Indicates the time step, Indicates the total time step.

[0061] A time step of the diffusion-based personalized generation model was constructed. Linearly correlated dynamic loss weights This weight, acting as a precise scheduler, divides the denoising process for latent features into two distinct stages and achieves a smooth transition between them. At the beginning of the diffusion process (high... (Value phase) Set to a higher initial value This makes the potential diffusion loss in the total loss function... It occupies a relatively dominant position. However, as the denoising process continues, the time step... Gradually decrease, The value of the latent space-image alignment loss decreases linearly thereafter. It will take the dominant position.

[0062] This design enables high noise reduction in the early stages (high... In the value phase, the main objective of the optimization process is to minimize the potential diffusion loss. In this stage, noise constitutes a significant proportion of the potential features, making the structure relatively vulnerable to disturbances and hidden within the noise. Therefore, focused optimization is necessary. It can efficiently utilize its gradient to directly attack and dismantle perturbed structures. When the process enters the later stages of denoising (low... (Value phase) It has dropped to a low level. The proportion of noise in the total loss increases significantly, and the optimization focus shifts accordingly. At this point, a large amount of noise has been removed, the semantic content of the image is clarified, and the image is enhanced. Optimization can use its perceptual similarity metric to finely correct semantic biases or detail distortions that may be introduced by previous denoising, ensuring that the output is highly consistent with the expected content.

[0063] S5. The denoised features are cropped and decoded by the decoder of the diffusion personalized generation model to obtain an optimized image.

[0064] In this embodiment of the invention, step S5 further includes the following sub-steps: S51. Decode the denoising features according to the decoder to obtain a decoded image; S52. Calculate the pixel difference between the decoded image and the anti-interference image. When the pixel difference is greater than a preset threshold, crop the decoded image. By constraining the pixel level of the decoded image, the quality of subsequent image generation is improved.

[0065] S53. Encode the cropped decoded image according to the encoder of the autoencoder to obtain the encoded features; S54. The encoded features are decoded again according to the decoder to obtain the optimized image.

[0066] In this embodiment of the invention, the decoding feature satisfies the following conditions. ; in, This indicates the decoding feature. This refers to the encoder. This refers to the decoder. Indicates the area to be clipped.

[0067] The computer device 300 provided in this embodiment of the invention can implement the steps in the image generation method based on the diffusion personalized generation model as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0068] Example 4 This invention also provides a computer-readable storage medium storing an image generation program based on a diffusion-based personalized generation model. When executed by a processor, the image generation program implements the various processes and steps of the image generation method based on a diffusion-based personalized generation model provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0069] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer programs or instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0070] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0072] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form under the guidance of the present invention without departing from the spirit and scope of the claims. All such changes are within the protection scope of the present invention.

Claims

1. An image generation method based on a diffusion-based personalized generation model, characterized in that, The image generation method includes the following steps: S1. Obtain an adversarial perturbation image, establish a diffusion-based personalized generation model, use the adversarial perturbation image as input to the diffusion-based personalized generation model, and obtain the latent features of the adversarial perturbation image based on the autoencoder of the diffusion-based personalized generation model. S2. Calculate the alignment loss of the adversarial perturbation image based on the latent features and the first preset rule; S3. Calculate the potential diffusion loss based on the random noise of the personalized diffusion generation model according to the second preset rule; S4. Denoise the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features; S5. The denoising features are cropped and decoded by the decoder of the autoencoder to obtain an optimized image.

2. The image generation method based on a diffusion-based personalized generation model as described in claim 1, characterized in that, The first preset rule is: to make the alignment loss satisfy the following conditions: ； in, This represents the alignment loss. This refers to the decoder. This refers to the image that resists perturbation. This indicates the potential feature. This represents the similarity metric of the images against the perturbation.

3. The image generation method based on a diffusion-based personalized generation model as described in claim 2, characterized in that, The second preset rule is: to ensure that the potential diffusion loss satisfies the following conditions: ； in, This represents the potential diffusion loss. This represents the random Gaussian noise added to the latent features by the diffusion-based personalized generation model at each forward step. This represents the random noise. The parameters represent the diffusion-based personalized generation model. The diffusion-based personalized generation model indicates that in the first... t Potential features generated during the forward pass. This indicates the category of the image generated by the diffusion-based personalized generation model.

4. The image generation method based on the diffusion-based personalized generation model as described in claim 3, characterized in that, Step S4 also includes the following sub-steps: S41. The alignment loss and potential diffusion loss are weighted and calculated according to the time step of the diffusion personalized generation model to obtain the total loss; S42. Denoise the latent features based on the total loss to obtain the denoised features.

5. The image generation method based on the diffusion-based personalized generation model as described in claim 4, characterized in that, The total loss satisfies the following condition: ； ； in, Indicates the current weight coefficient. Indicates the initial weighting coefficients. This represents the final weighting coefficient. This represents the total loss. Indicates the time step, Indicates the total time step.

6. The image generation method based on the diffusion-based personalized generation model as described in claim 5, characterized in that, Step S5 also includes the following sub-steps: S51. Decode the denoising features according to the decoder to obtain a decoded image; S52. Calculate the pixel difference between the decoded image and the anti-interference image. When the pixel difference is greater than a preset threshold, crop the decoded image. S53. Encode the cropped decoded image according to the encoder of the autoencoder to obtain the encoded features; S54. The encoded features are decoded again according to the decoder to obtain the optimized image.

7. The image generation method based on a diffusion-based personalized generation model as described in claim 6, characterized in that, The decoding feature satisfies the following conditions: ； in, This indicates the decoding feature. This refers to the encoder. Indicates the area to be clipped.

8. An image generation system based on a diffusion-based personalized generation model, characterized in that, The aforementioned countermeasures and defense system includes: The input module is used to acquire an adversarial perturbation image, establish a diffusion personalized generation model, use the adversarial perturbation image as input to the diffusion personalized generation model, and acquire the latent features of the adversarial perturbation image according to the autoencoder of the diffusion personalized generation model. The alignment loss module is used to calculate the alignment loss of the adversarial image based on the latent features and a first preset rule. The diffusion loss module is used to calculate the potential diffusion loss based on the random noise of the diffusion personalized generation model according to a second preset rule. The denoising module is used to denoise the latent features based on the alignment loss and the latent diffusion loss to obtain denoised features; An optimization module is used to crop the denoised features and decode them through the decoder of the diffusion-based personalized generation model to obtain an optimized image.

9. A computer device, characterized in that, include: The image generation method based on a diffusion-based personalized generation model includes a memory, a processor, and an image generation program stored in the memory and executable on the processor. When the processor executes the image generation program based on the diffusion-based personalized generation model, it implements the steps of the image generation method based on a diffusion-based personalized generation model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image generation program based on a diffusion-based personalized generation model, which, when executed by a processor, implements the steps of the image generation method based on a diffusion-based personalized generation model as described in any one of claims 1-7.

Images