A Secure Optimization Method and Device for Diffusion Generation Based on Multimodal Region Semantic Alignment

Abstract

This application provides a method and apparatus for secure optimization of diffusion generation based on multimodal region semantic alignment. In one example, the method includes: acquiring an image to be processed and text instructions; inputting the image to be processed and text instructions into a trained multimodal diffusion model; and receiving the processed image output by the trained multimodal diffusion model. This method can achieve precise control over sensitive regions while ensuring the image quality and naturalness of non-sensitive regions, thus optimizing the overall visual effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a diffusion generation security optimization method and device based on multimodal region semantic alignment. Background Technology

[0002] With the rapid development of generative artificial intelligence, especially conditional image generation models, ensuring the security and compliance of its output content has become a major challenge. Existing security control schemes for generative models all have significant limitations, making it difficult to achieve a good balance between accuracy, efficiency, and flexibility. Summary of the Invention

[0003] In view of this, this application provides a diffusion generation security optimization method and device based on multimodal region semantic alignment.

[0004] Specifically, this application is implemented through the following technical solution:

[0005] According to a first aspect of the embodiments of this application, a diffusion generation security optimization method based on multimodal region semantic alignment is provided, comprising:

[0006] Obtain the image to be processed and the text instructions;

[0007] The image to be processed and the text command are input into a trained multimodal diffusion model. The trained multimodal diffusion model, responding to the semantics of the text command, automatically identifies sensitive regions in the image to be processed and performs region-level content control on these sensitive regions. The multimodal diffusion model is trained by injecting a multi-branch dLoRA module, which includes at least a general branch and a task branch. The task branch is used to automatically identify the sensitive regions based on the semantics of the text command and perform region-level content control on these sensitive regions.

[0008] Receive the processed image output by the trained multimodal diffusion model.

[0009] According to a second aspect of the embodiments of this application, an electronic device is provided, including a processor and a memory, wherein...

[0010] Memory, used to store computer programs;

[0011] The processor, when executing a program stored in memory, implements the method provided in the first aspect.

[0012] According to a third aspect of the embodiments of this application, a computer program product is provided, wherein the computer program product stores a computer program, and the computer program, when executed by a processor, implements the method provided in the first aspect.

[0013] The diffusion generation security optimization method based on multimodal region semantic alignment in this application acquires the image to be processed and text instructions, and inputs the acquired image to be processed and text instructions into a trained multimodal diffusion model. The trained multimodal diffusion model automatically identifies sensitive regions in the image to be processed based on the semantics in the text instructions, and performs region-level content control on the identified sensitive regions. Then, it receives the processed image output by the trained multimodal diffusion model, realizing end-to-end automated sensitive region identification and region-level content control. It does not rely on manual annotation or external segmentation models, significantly improving the efficiency of image security processing and reducing application costs. By injecting a multi-branch dLoRA module for training, the task branch automatically identifies and controls sensitive regions based on semantics, while the general branch maintains the basic image quality, thus decoupling general generation capabilities from specific security tasks. While achieving precise control over sensitive regions, it ensures the image quality and naturalness of non-sensitive regions, optimizing the overall visual effect. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a diffusion generation security optimization method based on multimodal region semantic alignment, as shown in an exemplary embodiment of this application.

[0015] Figure 2 This is a schematic diagram illustrating a sensitive area mask generation process, as shown in an exemplary embodiment of this application.

[0016] Figure 3 This is a schematic diagram illustrating a model training process in the case of injecting a dLoRA module, as an exemplary embodiment of this application.

[0017] Figure 4 This is a schematic diagram illustrating a model training implementation process as shown in an exemplary embodiment of this application;

[0018] Figure 5 This is a schematic diagram illustrating a complete implementation process of a diffusion generation security optimization scheme based on multimodal region semantic alignment, as shown in an exemplary embodiment of this application.

[0019] Figure 6 This is a schematic diagram illustrating the structure of a diffusion generation security optimization device based on multimodal region semantic alignment, as shown in an exemplary embodiment of this application.

[0020] Figure 7 This is a schematic diagram of the hardware structure of an electronic device as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be explained below.

[0022] 1. Image segmentation model: This is a type of model that divides an input image into several regions with semantic or visual coherence. Its basic task is to identify the location and boundaries of different targets or structures in the image.

[0023] Among them, the SAM (Segment Anything Model) is an image segmentation model with generalization capabilities. Its core idea is to build a highly robust segmentation engine that supports "segmentation with any cue". By pre-training on large-scale and diverse image data, this model learns a general image understanding ability, enabling it to perform fast and high-precision segmentation of any region in an image without relying on specific category labels.

[0024] 2. Multimodal Region Semantic Alignment: This is a deep representation learning technique that integrates image and text information. Its core idea is to map local regions in an image to their corresponding linguistic descriptions into the same semantic space, thereby achieving cross-modal information understanding and alignment. This technique not only focuses on the overall semantic consistency of the image but also emphasizes region-level semantic expression, enabling the model to recognize the semantics expressed by specific regions in the image and to establish a one-to-one correspondence with words, phrases, or sentences in human language.

[0025] Significant breakthroughs have been achieved in multimodal region semantic alignment in terms of accuracy, generalization ability, and zero-shot adaptability. In particular, without relying on manual annotation, the CLIP (Contrastive Language-Image Pre-training) model can compare and match any image region with natural language labels to achieve automated semantic recognition and classification.

[0026] 3. Diffusion Generation: This technique provides a powerful generative paradigm for the deep fusion of images and text. Its basic idea is to generate high-quality images or other modal data from random noise through a gradual "denoising" process. In the context of image segmentation and multimodal semantic alignment, diffusion generation not only retains the advantages of high-quality generation but also demonstrates improved precision in semantic control and region guidance.

[0027] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0028] Please see Figure 1This is a flowchart illustrating a diffusion generation security optimization method based on multimodal region semantic alignment provided in an embodiment of this application. Figure 1 As shown, the diffusion generation security optimization method based on multimodal region semantic alignment may include:

[0029] Step S100: Obtain the image to be processed and the text instructions.

[0030] For example, the image to be processed refers to an image that has regional content control requirements, which may include, but is not limited to, images actively uploaded by users, publicly available images obtained from the Internet, or images generated by other systems.

[0031] For example, text instructions are control commands expressed in natural language that explicitly indicate what operation to perform on a specific region of an image.

[0032] For example, text instructions could include "specify a target in a blurred image" or "blur the license plate".

[0033] For example, the aforementioned regional content control may include, but is not limited to, one or more of the following: partial modification of a specific region, de-identification reconstruction for privacy or compliance protection, and semantic-based content filtering.

[0034] In one example, the image to be processed and the text command can be entered simultaneously; the text command is a text instruction specific to the image to be processed.

[0035] In another example, text commands can be pre-configured. The system can perform unified region-level content control on single or batch-acquired images based on a preset text command (such as "reduce license plates on all images").

[0036] It should be noted that, in the embodiments of this application, all data involving personal information used in the implementation of the solution are used with the explicit authorization of the relevant personnel and in accordance with the relevant privacy policies and regulations. The relevant processing of the data during use strictly complies with the applicable data protection policies and regulations, and the relevant processing behavior has obtained the unambiguous authorization of the data subject based on the principle of explicit knowledge for the purpose of using the data for the purposes described in this application.

[0037] Step S110: Input the image to be processed and the text command into the trained multimodal diffusion model; wherein, the trained multimodal diffusion model responds to the semantics in the text command, automatically determines the sensitive regions in the image to be processed, and performs region-level content control on the sensitive regions; the multimodal diffusion model is trained by injecting a multi-branch dLoRA module, which includes at least a general branch and a task branch, the task branch being used to automatically identify sensitive regions based on the semantics of the text command and perform region-level content control on the sensitive regions.

[0038] In this embodiment, a trained multimodal diffusion model can be used, with the image to be processed and text instructions as multimodal inputs, and based on the semantic understanding of the text instructions, regional content control can be performed on the image to be processed.

[0039] In this embodiment, a trained multimodal diffusion model can be used, with the image to be processed and text instructions as multimodal inputs, and based on the semantic understanding of the text instructions, regional content control can be performed on the image to be processed.

[0040] For example, a well-trained multimodal diffusion model can align and interact the semantic embedding vector of text instructions with the visual features of the image in a deep semantic space through its internal multimodal fusion mechanism (such as a cross-attention layer). Thus, it can automatically focus on and locate specific semantic regions in an image without relying on external segmentation models or manually labeled region masks.

[0041] To improve the accuracy and efficiency of regional content control while ensuring image generation quality, a multi-branch dLoRA (Dynamic Low-Rank Adaptation) module can be injected into the multimodal diffusion model during training, and the multimodal diffusion model with the injected multi-branch dLoRA module can be trained.

[0042] For example, the multi-branch dLoRA module may include at least two functionally decoupled branches: a general branch and a task branch; wherein:

[0043] The general branch is responsible for maintaining the model's general generative capabilities and ensuring the basic quality of the images. For example, this branch can focus on learning general image structure and texture knowledge that is independent of specific tasks.

[0044] Task branches are used to learn semantic control capabilities for specific security concepts. For example, this branch is specifically designed to capture and respond to feature changes related to specific semantic concepts such as security and compliance.

[0045] For example, through training, the task branch can internalize the mapping relationship between specific semantic concepts (such as "license plate", "violent weapon" etc.) and the appearance features of image regions, thereby automatically locking the corresponding target region in the image space according to the guidance of text instructions.

[0046] Based on this, in the trained multimodal diffusion model, the task branch can automatically identify sensitive regions in the image to be processed according to the semantics of the text instructions, and perform region-level content control on the identified sensitive regions.

[0047] For example, to obtain the image to be processed and the text instruction, a visual encoder can be used to extract image features from the image to be processed (e.g., extract multi-scale features from the image to be processed), and the text instruction can be converted into a semantic embedding vector using an in-text encoder (such as the CLIP text encoder).

[0048] The task branch can automatically identify sensitive regions based on the semantics of the current text instruction and perform specific content control processing on those regions during the diffusion process. The general branch works in parallel to ensure the overall quality and naturalness of the generated image.

[0049] After the above process, the model finally outputs a high-quality image, in which sensitive areas related to text instructions have been precisely controlled as required (such as blurring, censoring, replacement, etc.), while non-sensitive areas have been preserved to the greatest extent possible.

[0050] Step S120: Receive the processed image output by the trained multimodal diffusion model.

[0051] In this embodiment of the application, when the acquired image to be processed and the text command are input into the trained multimodal diffusion model, the processed image output by the trained multimodal diffusion model can be received (regional content control is performed on the sensitive area).

[0052] For example, the received processed image can be returned to the user (such as when the user submits an image request for processing), stored in a specified location, or published to the corresponding platform.

[0053] For example, in content moderation scenarios, processed images can be automatically marked as "compliant" and entered into the publishing queue; in privacy protection scenarios, processed images can be used for data exchange or public sharing.

[0054] It can be seen that, in Figure 1The method described above acquires the image to be processed and text instructions, and inputs these into a trained multimodal diffusion model. The trained model automatically identifies sensitive regions in the image based on the semantics of the text instructions, and performs region-level content control on these identified sensitive regions. Finally, it receives the processed image output by the trained multimodal diffusion model, achieving end-to-end automated sensitive region identification and region-level content control. This eliminates the need for manual annotation or external segmentation models, significantly improving the efficiency of image security processing and reducing application costs. By injecting a multi-branch dLoRA module for training, the task branch automatically identifies and controls sensitive regions based on semantics, while the general branch maintains the basic image quality. This decouples general generation capabilities from specific security tasks, ensuring precise control over sensitive regions while maintaining image quality and naturalness in non-sensitive areas, thus optimizing the overall visual effect.

[0055] In some embodiments, training the multimodal diffusion model described above may include:

[0056] The weight matrix of the target linear transformation layer in the pre-trained multimodal diffusion model is decomposed by SVD, and the parameters of the general branch and the task branch are initialized according to the decomposition results; wherein, the target linear transformation layer is a linear transformation layer injected with a multi-branch dLoRA module;

[0057] Different dynamic learning rate strategies are used to train the initialized general branch and task branch.

[0058] For example, in order to achieve precise region-level content control in a parametrically efficient manner while maintaining the generative capabilities of the pre-trained multimodal diffusion model, multi-branch dLoRA modules can be injected into a specified linear layer (which may be called the target linear layer) of the pre-trained multimodal diffusion model.

[0059] For example, multi-branch dLoRA can be injected into the cross-attention module of a pre-trained multimodal diffusion model.

[0060] For example, during training, in order to ensure that the parameter update direction is consistent with the principal components of the pre-trained multimodal diffusion model, accelerate training convergence and improve stability, the weight matrix of the target linear layer can be decomposed by SVD (Singular Value Decomposition), and the parameters of the general branch and the task branch can be initialized based on the decomposition results.

[0061] In one example, the initialization parameters of the general branch are derived from the principal component directions after the SVD decomposition of the weight matrix of the target linear transformation layer;

[0062] The initialization parameters of the task branch come from the secondary component directions after the SVD decomposition of the weight matrix of the target linear transformation layer.

[0063] For example, assuming the weight matrix of the target linear layer is W, it can be decomposed using SVD:

[0064]

[0065] in, ,and U is a left singular vector. It is the transpose of the right singular vector.

[0066] Based on the SVD decomposition results, the parameters of the general branch and task branch can be initialized as follows:

[0067]

[0068] and The initialization parameters for the general branch come from the principal component directions after SVD decomposition, i.e.:

[0069]

[0070]

[0071] in, The front of matrix U List, For the front The most important singular values ​​and their corresponding The product of parts, where r is the total rank of dLoRA. The rank of the general branch.

[0072] The above initialization method ensures that the general branch can focus on maintaining the general feature representation of the model.

[0073] and The initialization parameters for the task branch come from the direction of the minor components after SVD decomposition, namely:

[0074]

[0075]

[0076] in, For the U matrix, the first Column to column r, For the first up to the r-th singular value and its corresponding Partial product.

[0077] The above initialization method provides a "potential space" for task branches to learn specific security concepts, avoiding interference with general knowledge.

[0078] To collaboratively optimize the two functionally decoupled branches during training and improve the expressive power of important parameters and the generalization ability of the model, different dynamic learning rate strategies can be used to train the initialized general branch and task branch.

[0079] In one example, an exponentially decaying learning rate can be used for the general branch, and a periodic cosine restart learning rate can be used for the task branch.

[0080] For example, an exponentially decaying learning rate can be used for the general branch:

[0081]

[0082] in, The learning rate of the general branch when training steps t. Let be the initial learning rate for the general branch, λ be the decay rate used to control the speed at which the learning rate decays, and t be the number of training steps. This is the lower bound of the learning rate.

[0083] This strategy allows the learning rate of the general branch to be high in the early stages of training, enabling rapid learning of general knowledge; then it is gradually reduced for fine-tuning, and finally stabilized at a base value to avoid over-adjustment.

[0084] For the general branch, the learning rate is periodically restarted using a cosine-based method.

[0085]

[0086] in, The learning rate of the task branch when training steps t. This represents the maximum value of the learning rate. The minimum learning rate is T, the length of one cycle (the number of steps in one cycle), and t mod T is the remainder when t is divided by T.

[0087] This strategy involves periodically reducing and restarting the learning rate for task branches, which helps the model escape local optima, maintain its exploratory ability in complex and variable "task-specific" learning, and improve generalization.

[0088] It should be noted that, in the embodiments of this application, the task branches can be flexibly expanded according to actual security needs. Each task branch is independently designed to learn a specific security concept or semantic category (for example, "license plate fuzzing branch", "violent content filtering branch", "inappropriate symbol recognition branch", etc. can be set up respectively).

[0089] For example, in the case of multiple task branches, these multiple task branches can share the aforementioned secondary component directions (as described above). and (i.e., all tasks have the same initialization parameters), or, according to specific rules, the above secondary component directions can be divided. For example, assuming there are K (K≥2) task branches, the above secondary component directions can be divided into N segments and assigned to the N task branches as initialization parameters.

[0090] During training, each task branch can be configured with its own dynamic learning rate parameters. For example, different cosine restart periods and / or maximum learning rates can be set for task branches with different complexities, enabling more refined optimization control.

[0091] For example, during the inference process, the system dynamically activates one or more related task branches based on the semantics of the input text instructions. The activated branches work together in the generation process to identify and control different types of sensitive regions in the image, thereby achieving complex security and compliance processing.

[0092] For example, suppose the task branches include "license plate blurring branch", "violent content filtering branch", and "inappropriate symbol recognition branch", and the text instruction is "please desensitize the license plate and violent content in the image", then the system can dynamically activate two corresponding task branches during the inference process: "license plate blurring branch" and "violent content filtering branch".

[0093] During the reasoning process, the two activated task branches work in parallel, each searching for a semantically matching target in the image:

[0094] License plate blurring branch: Locate all vehicle license plates in the image.

[0095] Violent content filtering branch: Analyze the image content to determine whether it contains "violent content" mentioned in the instructions.

[0096] The model ultimately outputs a processed image. In this image:

[0097] License plates are blurred (e.g., pixelated), and parts identified as violent content are removed or replaced with harmless patterns.

[0098] The rest of the image, such as streets, buildings, and trees, retains its original high quality and natural appearance.

[0099] In some embodiments, training the multimodal diffusion model described above may include:

[0100] During backpropagation, the general branch updates parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss;

[0101] The task branch updates parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss, or the task branch updates parameters based on the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss.

[0102] Among them, diffusion reconstruction loss is used to ensure the basic quality of the generated image, region semantic supervision loss is used to ensure the semantic consistency between the generated content and the text instructions, and mask consistency loss is used to constrain the model's precise control over the specified region.

[0103] For example, in order to ensure the semantic consistency between the generated content and the text instructions while maintaining the quality of the generated image, the loss function during the training of the multimodal diffusion model can include diffusion reconstruction loss and region semantic supervision loss.

[0104] Among them, diffusion reconstruction loss can be used to ensure the basic quality of the generated image, and region semantic supervision loss can be used to ensure the semantic consistency between the generated content and the text instructions.

[0105] For example, during backpropagation, the parameters of the general branch can be updated based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss, and the parameters of the task branch can be updated based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss.

[0106] For example, in order to enable task branches to more accurately identify sensitive regions in an image and improve the accuracy of region-level content control, the loss function can also include mask consistency loss during the training of the multimodal diffusion model.

[0107] Among them, mask consistency loss can be used to constrain the model to accurately control a specified region.

[0108] During backpropagation, the parameters of the task branches can be updated based on the gradient of the mask consistency loss. That is, the parameters of the task branches can be updated based on the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss.

[0109] In one example, the diffusion reconstruction loss can be obtained from the mean square error between the model's predicted noise and the actual noise.

[0110] In one example, the region semantic supervision loss is determined based on the cosine similarity between the generated image and the text instruction in the shared semantic space.

[0111] In one example, the mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region.

[0112] For example, the specific methods for determining the aforementioned losses can be explained below with specific examples.

[0113] In some embodiments, the total loss of training the multimodal diffusion model is a weighted sum of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss;

[0114] The weighting coefficient of each loss is dynamically adjusted based on the proportion of the original loss value of each loss in the total original loss value.

[0115] For example, considering that different loss functions correspond to different learning tasks with different convergence characteristics and difficulty levels during training, in order to prevent the performance of other tasks from deteriorating due to over-optimization of one task, and to ensure that the model improves in a balanced way in the three dimensions of image quality, semantic alignment and region control, a dynamic weight adjustment strategy based on the proportion of the original loss value (the ratio of the original loss value to the total original loss value) can be adopted to achieve stable convergence of training and collaborative optimization of multiple objectives.

[0116] In one example, dynamic adjustments can be made based on the deviation of the original loss value of each loss from the proportion of the total original loss value to the mean proportion.

[0117] Specifically, when the deviation is positive, the weighting coefficient of the corresponding loss increases; when the deviation is negative, the weighting coefficient of the corresponding loss decreases.

[0118] The sum of the weighting coefficients for each loss remains at a preset value.

[0119] For example, by monitoring the relative proportion of each original loss value, the system can automatically identify the "bottleneck task" in the current training. If the proportion of the original loss value of a certain loss is relatively high, it can indicate that the corresponding task is currently difficult to train and needs more attention.

[0120] For example, tasks with poor performance (tasks with a higher proportion of the original loss value) can automatically receive higher weight coefficients and gain greater influence in gradient updates, thereby achieving intelligent allocation of training resources through a dynamic adjustment mechanism.

[0121] In addition, to avoid gradient explosion, a summation constraint can be introduced during the dynamic adjustment of the weight coefficients of each loss, that is, to ensure that the sum of the weight coefficients of each loss remains at a preset value.

[0122] For example, the total loss function of a multimodal diffusion model can be as follows:

[0123]

[0124] The initial weights can be set to λ1=1.0, λ2=0.3, and λ3=0.5.

[0125] The weighting coefficients of each loss can be dynamically adjusted in the following ways:

[0126]

[0127]

[0128]

[0129] in, This represents the original loss value for the k-th loss at time step t. The sum of the original loss values ​​of the three losses at time step t (i.e., the total original loss value). The weighting coefficient for the k-th loss at time step t. The weighting coefficient for the k-th loss at time step t+1. This is the sum of the weighting coefficients for the three losses.

[0130] Based on the above dynamic adjustment strategy, for a loss, if the proportion of the original loss value in the total original loss value is less than the proportional mean (1 / 3≈0.33), that is, the deviation between the proportion of the original loss value in the total original loss value and the proportional mean is negative, that is, the original loss value is relatively small (less than the average of the original loss values), the weight coefficient of the loss is reduced; if the deviation between the proportion of the original loss value in the total original loss value and the proportional mean is positive, that is, the original loss value is relatively large (greater than the average of the original loss values), the weight coefficient of the loss is increased.

[0131] In addition to determining the adjusted weight coefficients of each loss according to the above strategy, it is also necessary to ensure that the sum of the weight coefficients of each loss remains unchanged.

[0132] For example, normalization can be used to keep the sum of the weighted coefficients of each loss unchanged.

[0133] For example, suppose the total weight coefficient is preset to 1.8. Based on the deviation of each original loss value from the target mean, the pre-adjusted weight coefficients are calculated. The ratio of the sum of these pre-adjusted weight coefficients to the preset value of 1.8 can be used as a normalization coefficient. Each pre-adjusted weight coefficient is then multiplied by this normalization coefficient to obtain the final weight coefficients, ensuring that their sum equals 1.8.

[0134] In some embodiments, the mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region;

[0135] The actual mask area is determined in the following way:

[0136] For any sample image, the SAM model is used to divide the sample image into initial candidate regions, and the N initial candidate regions with the highest scores are determined as the final candidate regions based on the salience scores of each initial candidate region; where N≥2.

[0137] Based on the similarity between the image features of the original sample image block corresponding to each final candidate region and the text features of the preset text security label, the final candidate region whose image features are more than the same as the text features of any preset text security label are determined as a sensitive region.

[0138] Based on the identified sensitive area, determine the true mask area of ​​the sample image.

[0139] For example, during the training of a multimodal diffusion model, a region recognition mechanism combining salient region decomposition and semantic-driven screening can be used to automatically identify sensitive regions in sample images.

[0140] For example, the SAM (Segment Anything Model) model can be used to divide the sample image into salient regions (also known as initial candidate regions) to generate an initial candidate region mask.

[0141] Among them, the SAM model has zero-shot segmentation capability, which can automatically extract salient regions in images on unlabeled data and obtain initial candidate regions without manual annotation.

[0142] To improve the efficiency and reliability of subsequent semantic judgment, for the regions corresponding to each initial candidate region mask in the sample image (which can be called initial candidate regions), each initial candidate region can be scored according to its saliency intensity (such as region area, boundary gradient change and / or image contrast) (which can be called saliency score) and sorted. The initial candidate regions with the top N scores are selected as the final candidate regions.

[0143] For example, for any final candidate region, the image features of the original sample image block corresponding to the candidate region (i.e., the image block at the location of the candidate region in the sample image) can be extracted, and the similarity between the image features of each sample image block and the text features of the preset text security label can be determined.

[0144] The text features of the preset text security label can be obtained by extracting the text features of the preset text security label.

[0145] If the similarity between the image features of the original sample image block and the text features of any preset text security label exceeds a preset similarity threshold, the final candidate region can be identified as a sensitive region, where the mask corresponding to the sensitive region is the sensitive region mask.

[0146] For example, the mask region (which may be called the real mask region) of the sample image can be determined based on the sensitive region determined in the manner described above.

[0147] For example, in the training configuration of a multimodal diffusion model, for any sample image, the mask consistency loss can be determined based on the overlap between the mask region of the sample image predicted by the multimodal diffusion model (i.e., the sensitive region in the sample image predicted by the multimodal diffusion model) and the real mask region in the sample image.

[0148] To enable those skilled in the art to better understand the technical solutions provided in this application, the technical solutions provided in this application are described below with reference to specific examples.

[0149] In this embodiment, a security control scheme for diffusion generative models is provided, integrating automatic region segmentation, region-level multimodal semantic alignment, multi-branch low-rank structure injection, and multi-level adaptive compliance determination. Its implementation process may include:

[0150] 1. Automated Region Segmentation: The SAM model is used to automatically decompose candidate regions in any input image without manual annotation, and multiple candidate region masks are obtained.

[0151] 2. Region-level multimodal alignment: For each candidate region (the region in the input image corresponding to the candidate region mask), the CLIP model is used to extract the region image features and security text semantic features respectively, and the semantic space similarity is used to determine whether the region has sensitive risks;

[0152] 3. Multi-branch dLoRA structure injection: Inject a multi-branch (e.g., dual-branch) dLoRA structure based on SVD initialization into the cross-attention module of the U-Net backbone, and perform independent parameter optimization for general tasks and sensitive regions respectively, so as to ensure efficient parameter decoupling and dynamic adaptation.

[0153] 4. Adaptive multi-level semantic judgment and feedback: During the training phase, a region-level loss function and an adaptive weight adjustment mechanism are introduced to achieve continuous feedback optimization and safety control for different risk levels.

[0154] 5. Highly automated throughout the entire process: adaptable to large-scale task expansion in unsupervised / weakly supervised scenarios.

[0155] The implementation process described above will be explained in detail below.

[0156] I. Automated region segmentation.

[0157] For example, for a sample image, the sample image can be input into the SAM model, and the SAM model can output multiple candidate sensitive region masks.

[0158] For example, for the multiple candidate region masks (which can be called initial candidate region masks) output by the SAM model, the regions corresponding to each initial candidate region mask can be ranked according to their salience strength. The initial candidate region masks corresponding to the Top N regions with the highest scores can be selected as the final candidate region masks to reduce the consumption of computing resources by low-value regions and improve the efficiency and reliability of subsequent semantic judgment.

[0159] II. Regional-level multimodal alignment.

[0160] For example, for any candidate region mask (such as the final candidate region mask), the corresponding region image block can be cropped from the sample image based on the candidate region mask, and after normalization, it is sent to the image-text alignment semantic judgment module. By comparing it with the preset multi-label security semantic vector set, its semantic relevance vector is obtained.

[0161] For example, candidate region masks can be traversed, and corresponding region images in the original sample image can be cropped according to the candidate region masks to obtain region image patches. These patches are then scaled to the CLIP standard size (224*224) and input into the CLIP image encoder to extract image features, resulting in an image feature vector (also known as a region image vector). i .

[0162] Furthermore, for a predefined set of text security tags, such as a set of security-sensitive words S = ("violent", "Dangerous articles", "unsafe for work"), CLIP text encoding can be performed to obtain a text feature vector (also known as a text semantic vector) t. j .

[0163] Normalize the region image vector and the text semantic vector respectively:

[0164]

[0165]

[0166] For example, the semantic similarity between each region's image vector and the text semantic vector can be calculated (e.g., using a cosine similarity function) and compared with a set threshold θ.

[0167] For example, the similarity matrix can be as follows:

[0168]

[0169] Where d represents the dimension index of the vector, the image feature vector and text feature vector output by the CLIP model are usually 512-dimensional. This summation operation is used to calculate the inner product of the two in a dimension-wise manner, thereby obtaining the semantic cosine similarity between the region image patch and the text security label.

[0170] For example, for any candidate region mask, if the similarity between the region image block corresponding to the candidate region mask and any security-sensitive word (or text security label) is greater than a threshold θ, then the candidate region mask is marked as a sensitive region mask. The implementation flowchart is as follows: Figure 2 As shown.

[0171] For example, the masks of each sensitive region can be merged to obtain the final sensitive region mask.

[0172] For example, all sensitive region masks can be merged pixel-by-pixel in the image space to construct the final sensitive region mask M of the image. gt .

[0173] M gt As a real-world sensitive region supervision signal during the training phase, it is used to calculate the mask consistency loss L. mask .

[0174] III. Injection of multi-branch dLoRA structures.

[0175] For example, a multi-branch dLoRA module is injected into the U-Net backbone of a multimodal diffusion model (such as the Stable Diffusion model).

[0176] For example, the dual-branch dLoRA module includes a general branch and a task branch, which can adapt to changes in common structures and local regions, respectively.

[0177] For example, a dual-branch dLoRA module can be injected into the cross-attention layer.

[0178] The pre-trained weight matrix can be decomposed using SVD, and the parameters of the general branch and the task branch can be initialized based on the SVD decomposition results.

[0179] Based on the SVD decomposition results, the parameters of the general branch and task branch can be initialized as follows:

[0180]

[0181] and The initialization parameters for the general branch come from the principal component directions after SVD decomposition, i.e.:

[0182]

[0183]

[0184] in, The front of matrix U List, For the front The most important singular values ​​and their corresponding The product of parts, where r is the total rank of dLoRA. The rank of the general branch.

[0185] The above initialization method ensures that the general branch can focus on maintaining the general feature representation of the model.

[0186] and The initialization parameters for the task branch come from the direction of the minor components after SVD decomposition, namely:

[0187]

[0188]

[0189] in, For the U matrix, the first Column to column r, For the first up to the r-th singular value and its corresponding Partial product.

[0190] The above initialization method provides a "potential space" for task branches to learn specific security concepts, avoiding interference with general knowledge.

[0191] To collaboratively optimize the two functionally decoupled branches during training and improve the expressive power of important parameters and the generalization ability of the model, different dynamic learning rate strategies can be used to train the initialized general branch and task branch.

[0192] In one example, an exponentially decaying learning rate can be used for the general branch, and a periodic cosine restart learning rate can be used for the task branch.

[0193] For example, an exponentially decaying learning rate can be used for the general branch:

[0194]

[0195] in, The learning rate of the general branch when training steps t. Let be the initial learning rate for the general branch, λ be the decay rate used to control the speed at which the learning rate decays, and t be the number of training steps. This is the lower bound of the learning rate.

[0196] This strategy allows the learning rate of the general branch to be high in the early stages of training, enabling rapid learning of general knowledge; then it is gradually reduced for fine-tuning, and finally stabilized at a base value to avoid over-adjustment.

[0197] For the general branch, the learning rate is periodically restarted using a cosine-based method.

[0198]

[0199] in, The learning rate of the task branch when training steps t. This represents the maximum value of the learning rate. The minimum learning rate is T, the length of one cycle (the number of steps in one cycle), and t mod T is the remainder when t is divided by T.

[0200] For example, the model training process with the dLoRA module injected can be as follows: Figure 3 As shown.

[0201] This strategy involves periodically reducing and restarting the learning rate for task branches, which helps the model escape local optima, maintain its exploratory ability in complex and variable "task-specific" learning, and improve generalization.

[0202] IV. Adaptive multi-level semantic judgment and feedback.

[0203] For example, during model training, the loss function is the fusion diffusion reconstruction loss (L... recon ), region semantic supervision loss (L) clip ) and optional mask consistency loss (L mask ( ), jointly optimize image generation quality and secure semantic alignment.

[0204] For example, the region mask and sample image can be input into the diffusion model separately (the region mask and sample image are input into the diffusion model through different input channels, and the region mask can be used as the real mask region to achieve region mask supervision).

[0205] 1. Enter M gt and sample images.

[0206] 2. Obtain prediction noise using UNet Calculate L recon :

[0207]

[0208] Where, x t The noisy image at time step t is generated through a forward diffusion process.

[0209] It is real Gaussian noise, following a standard normal distribution N(0, I), with a mean of 0 and a variance of 1.

[0210] The noise predicted by UNet is represented by model parameters θ, which include the parameters of the dLoRA module being trained.

[0211] The model learns the restoration and denoising mapping by minimizing the L2 norm squared error between the two.

[0212] 3. Via CLIP encoder E I and E T Calculate the cosine similarity of image-text pairs:

[0213]

[0214] Among them, E I For CLIP image encoders, the output is a 512-dimensional normalized vector; E T For CLIP text encoder, output in the same dimension; x gen The image generated for the model.

[0215] In the above "image-text pair", the image refers to the generated whole frame image, which is used to calculate the semantic consistency between the whole image and the text. The text refers to the text instruction. The cosine similarity of the above image-text pair is used to constrain the overall semantic consistency of the generated result.

[0216] 4. Calculate the Dice loss in the presence of a region mask:

[0217]

[0218] in, The distribution of the sensitive region mask predicted by the diffusion model. This represents the mask distribution for the actual sensitive areas.

[0219] 5. Total loss function:

[0220]

[0221] The initial weights can be set to λ1=1.0, λ2=0.3, and λ3=0.5.

[0222] The weighting coefficients of each loss can be dynamically adjusted in the following ways:

[0223]

[0224]

[0225]

[0226] in, This represents the original loss value for the k-th loss at time step t. The sum of the original loss values ​​of the three losses at time step t (i.e., the total original loss value). The weighting coefficient for the k-th loss at time step t. The weighting coefficient for the k-th loss at time step t+1. This is the sum of the weighting coefficients for the three losses.

[0227] The system's final multi-objective joint training mechanism includes three sub-loss functions: L recon (Diffusion Reconstruction), L clip (Semantic alignment), L mask (Mask smoothing), through region masking The weighted aggregation is the total loss L total It is used for backpropagation to optimize the model, ultimately generating desensitized, compliant, and high-quality images. The training implementation flowchart is as follows: Figure 4 As shown.

[0228] For example, the complete implementation process of the diffusion generation security optimization scheme based on multimodal region semantic alignment can be as follows: Figure 5 As shown.

[0229] As can be seen, based on the above implementation process, the potential semantic regions in the image are automatically determined by the mechanism based on multimodal semantic alignment, and the determined regions are matched and controlled end-to-end with the corresponding text security concepts. This achieves fine-grained interactive alignment of multimodal information at the region level, breaking through the single and coarse process of "full-image labeling - full-image optimization" in the existing technology. This enables content security control to have decision interpretability (able to clearly trace the text semantics on which the control of a specific region is based) and precise regional orientation (ensuring that security operations are strictly limited to the target semantic region and retaining non-sensitive content to the greatest extent). Therefore, it is suitable for large-scale unmanned compliant image generation tasks with extremely high automation requirements.

[0230] Secondly, compared with traditional solutions that rely on static text matching or single-stage filtering, this application's embodiments construct a multi-stage, multi-modal dynamic compliance control system deeply integrated into the diffusion generation process. This transforms security judgment from a single check before and after generation to continuous semantic verification throughout each denoising step. By dynamically fusing visual features and deep semantics of text instructions during the generation process for joint evaluation, and adaptively adjusting the generation path based on verification results, it abandons the traditional coarse-grained control method that relies on fixed blocking rules and establishes a dynamic intervention mechanism based on cross-modal deep semantics. Specifically, the judgment mechanism based on cross-modal deep semantics effectively resists adversarial attacks such as character substitution, significantly improving the system's accuracy and robustness. The multi-stage, continuously iterative judgment system constitutes a defense-in-depth, greatly reducing the false negative rate of illegal content. Simultaneously, the fine-grained regional control mechanism avoids false negatives on compliant images, significantly improving the usability and controllability of the generated results while ensuring security, providing a reliable technical foundation for large-scale, unattended compliant image generation.

[0231] Furthermore, compared to the full-parameter fine-tuning methods of traditional CLIP or basic diffusion models, this application introduces a parameter-efficient LoRA method and innovatively adopts a multi-branch structure: by setting multiple independent LoRA branches for different security concepts or semantic categories, each branch only needs to train a small number of low-rank matrices while keeping the backbone parameters of the pre-trained model completely unchanged. This technical approach breaks through the traditional single adapter adjustment method. While strictly preserving the original model knowledge from being forgotten, it achieves efficient learning of multiple security semantics through branch specialization, significantly improving the ability to identify diverse illegal content. Since only a very small number of additional parameters need to be trained, training time and computational resources are greatly reduced, demonstrating excellent parameter efficiency. The independent branching design provides flexible and controllable capabilities for the generation process, supporting the dynamic enabling or disabling of specific branches according to actual needs, greatly enhancing the security controllability of the generated content.

[0232] Finally, by performing fine-grained semantic checks layer by layer from global to regional levels, and dynamically adjusting control strategies based on real-time risks, the accuracy and robustness of compliance judgments were systematically improved, and fine-grained control over generated content was achieved. This multi-layered verification system enables the model to more accurately distinguish between safe and illegal content, effectively reducing the risk of misjudgments and omissions.

[0233] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0234] Please see Figure 6 This is a schematic diagram of a diffusion generation security optimization device based on multimodal region semantic alignment provided in an embodiment of this application. Figure 6As shown, the diffusion generation security optimization device based on multimodal region semantic alignment may include:

[0235] The acquisition unit is used to acquire the image to be processed and text instructions.

[0236] An output unit is used to input the image to be processed and the text instruction into a trained multimodal diffusion model; wherein, the trained multimodal diffusion model responds to the semantics in the text instruction, automatically determines the sensitive regions in the image to be processed, and performs region-level content control on the sensitive regions; the multimodal diffusion model is trained by injecting a multi-branch dynamic low-rank adaptive dLoRA module, the multi-branch dLoRA module including at least a general branch and a task branch, the task branch being used to automatically identify the sensitive regions based on the semantics of the text instruction, and perform region-level content control on the sensitive regions;

[0237] The receiving unit is used to receive the processed image output by the trained multimodal diffusion model.

[0238] For example, the specific implementation process of the diffusion generation security optimization scheme based on multimodal region semantic alignment in each unit of the above-mentioned diffusion generation security optimization device can be found in the relevant description in the above embodiments, and will not be repeated here in the embodiments of this application.

[0239] This application also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; the processor is used to execute the program stored in the memory to implement the diffusion generation security optimization method based on multimodal region semantic alignment described above.

[0240] Please see Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 701 and a memory 702 storing machine-executable instructions. The processor 701 and the memory 702 can communicate via a system bus 703. Furthermore, by reading and executing the machine-executable instructions corresponding to the diffusion generation security optimization logic based on multimodal region semantic alignment in the memory 702, the processor 701 can execute the diffusion generation security optimization method based on multimodal region semantic alignment described above.

[0241] The memory 702 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0242] In some embodiments, a machine-readable storage medium, such as Figure 7 The memory 702 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the diffusion generation security optimization method based on multimodal region semantic alignment described above. For example, the machine-readable storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0243] This application also provides a computer program product storing a computer program, which, when executed by a processor, causes the processor to execute the diffusion generation security optimization method based on multimodal region semantic alignment described above.

Claims

1. A diffusion generation security optimization method based on multimodal region semantic alignment, characterized in that, include: Obtain the image to be processed and the text instructions; The image to be processed and the text command are input into a trained multimodal diffusion model. The trained multimodal diffusion model responds to the semantics of the text command, automatically identifies sensitive regions in the image to be processed, and performs region-level content control on the sensitive regions. The multimodal diffusion model is trained by injecting a multi-branch dLoRA module. The multi-branch dLoRA module includes at least a general branch and a task branch. The task branch is used to automatically identify the sensitive regions based on the semantics of the text command and perform region-level content control on the sensitive regions. The general branch is responsible for maintaining the model's general generation capability and ensuring the basic image quality. Receive the processed image output by the trained multimodal diffusion model; The training of the multimodal diffusion model includes: The weight matrix of the target linear transformation layer in the pre-trained multimodal diffusion model is decomposed by singular value decomposition (SVD), and the parameters of the general branch and the task branch are initialized according to the decomposition results; wherein, the target linear transformation layer is a linear transformation layer injected with the multi-branch dLoRA module; Different dynamic learning rate strategies are used to train the initialized general branch and task branch; The initialization of the parameters of the general branch and the task branch based on the decomposition results includes: The initialization parameters of the general branch are derived from the principal component directions after the SVD decomposition of the weight matrix of the target linear transformation layer; The initialization parameters of the task branch are derived from the secondary component directions after the SVD decomposition of the weight matrix of the target linear transformation layer.

2. The method according to claim 1, characterized in that, The process of training the initialized general branch and task branch using different dynamic learning rate strategies includes: An exponentially decaying learning rate is applied to the general branch; The learning rate is periodically restarted using a cosine function for the task branch.

3. The method according to claim 1, characterized in that, The training of the multimodal diffusion model includes: During backpropagation, the general branch updates its parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss. The task branch updates its parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss, or the task branch updates its parameters based on the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss. The diffusion reconstruction loss is used to ensure the basic quality of the generated image, the region semantic supervision loss is used to ensure the semantic consistency between the generated content and the text instructions, and the mask consistency loss is used to constrain the model's precise control over the specified region.

4. The method according to claim 3, characterized in that, The diffusion reconstruction loss is obtained based on the mean square error between the model-predicted noise and the actual noise. And / or, The region semantic supervision loss is determined based on the cosine similarity between the generated image and the text instruction in the shared semantic space; And / or, The mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region.

5. The method according to claim 3, characterized in that, The total loss during training of the multimodal diffusion model is a weighted sum of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss. The weighting coefficient of each loss is dynamically adjusted based on the proportion of the original loss value of each loss in the total original loss value.

6. The method according to claim 5, characterized in that, The dynamic adjustment is based on the deviation between the proportion of each loss's original loss value in the total original loss value and the average proportion. Specifically, when the deviation is positive, the weighting coefficient of the corresponding loss increases; when the deviation is negative, the weighting coefficient of the corresponding loss decreases. The sum of the weighting coefficients for each loss remains at a preset value.

7. The method according to claim 3, characterized in that, The mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region; The actual mask region is determined in the following way: For any sample image, the Segmentation All Model (SAM) is used to divide the sample image into initial candidate regions. Based on the saliency score of each initial candidate region, the N salient regions with the highest scores are determined as the final candidate regions; where N≥2. Based on the similarity between the image features of the original sample image block corresponding to each final candidate region and the text features of the preset text security label, the final candidate region whose image features are more than the same as the text features of any preset text security label are determined as a sensitive region. Based on the sensitive area, the true mask area of ​​the sample image is determined.

8. An electronic device, characterized in that, Including processor and memory, among which, Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1 to 7.

9. A computer program product, characterized in that, The computer program product contains a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

Images