Diffusion generation security optimization method and device based on multi-modal region semantic alignment

Abstract

The invention provides a diffusion generation security optimization method and device based on multi-modal region semantic alignment. In one example, the method comprises the steps of obtaining a to-be-processed image and a text instruction; inputting the to-be-processed image and the text instruction into a trained multi-modal diffusion model; and receiving a processed image output by the trained multi-modal diffusion model. According to the method, the image quality and naturalness of the non-sensitive area can be guaranteed while precise control over the sensitive area is achieved, and the overall visual effect is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a diffusion generation security optimization method and device based on multi-modal region semantic alignment. BACKGROUND

[0002] With the rapid development of generative artificial intelligence, especially conditional image generation models, ensuring the safety and compliance of their output content has become a major challenge. In the prior art, the safety control schemes for generation models all have obvious limitations, making it difficult to achieve a good balance between precision, efficiency and flexibility. SUMMARY

[0003] Therefore, the present application provides a diffusion generation security optimization method and device based on multi-modal region semantic alignment.

[0004] Specifically, the present application is implemented through the following technical solutions: According to a first aspect of an embodiment of the present application, a diffusion generation security optimization method based on multi-modal region semantic alignment is provided, comprising: obtaining a to-be-processed image and a text instruction; inputting the to-be-processed image and the text instruction into a trained multi-modal diffusion model; wherein the trained multi-modal diffusion model automatically determines a sensitive region in the to-be-processed image in response to the semantics in the text instruction, and performs region-level content control on the sensitive region; the multi-modal diffusion model is trained by injecting a multi-branch dLoRA module, and the multi-branch dLoRA module at least includes a general branch and a task branch, the task branch is used to automatically identify the sensitive region according to the semantics of the text instruction, and perform region-level content control on the sensitive region; receiving a processed image output by the trained multi-modal diffusion model.

[0005] According to a second aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein, the memory is used to store a computer program; the processor is used to execute the program stored on the memory, and realize the method provided in the first aspect.

[0006] According to a third aspect of an embodiment of the present application, a computer program product is provided, and the computer program product stores a computer program, and the computer program is executed by a processor to realize the method provided in the first aspect.

[0007] The diffusion generation safety optimization method based on multi-modal region semantic alignment provided by the embodiment of the present application can obtain an image to be processed and a text instruction, input the obtained image to be processed and the text instruction into a trained multi-modal diffusion model, automatically identify a sensitive region in the image to be processed according to semantics in the text instruction by using the trained multi-modal diffusion model, and perform region-level content control on the identified sensitive region. Then, the processed image output by the trained multi-modal diffusion model is received, and end-to-end automatic sensitive region identification and region-level content control are realized. The method does not need to rely on manual labeling or an external segmentation model, significantly improves the efficiency of image safety processing, and reduces application cost. The multi-branch dLoRA module is injected for training, the sensitive region is automatically identified and controlled by the task branch according to semantics, and the general branch maintains the basic quality of the image. The general generation capability and the specific safety task are decoupled, the image quality and naturalness of the non-sensitive region are ensured while the sensitive region is accurately controlled, and the overall visual effect is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A flowchart of a diffusion generation safety optimization method based on multi-modal region semantic alignment is shown for an exemplary embodiment of the present application. Figure 2 A sensitive region mask generation flowchart is shown for an exemplary embodiment of the present application. Figure 3 A model training flowchart in the case of injecting a dLoRA module is shown for an exemplary embodiment of the present application. Figure 4 A model training implementation flowchart is shown for an exemplary embodiment of the present application. Figure 5 A complete diffusion generation safety optimization scheme implementation flowchart based on multi-modal region semantic alignment is shown for an exemplary embodiment of the present application. Figure 6 A structure diagram of a diffusion generation safety optimization device based on multi-modal region semantic alignment is shown for an exemplary embodiment of the present application. Figure 7 A hardware structure diagram of an electronic device is shown for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0009] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms related to the embodiments of the present application will be described first.

[0010] 1. Image segmentation model: a class of models that divide input images into regions with semantic or visual coherence. The basic task is to identify the location and boundary of different targets or structures in the image.

[0011] Among them, the SAM (Segment Anything Model) model is a generalizable image segmentation model. Its core idea is to build a high-robustness segmentation engine that supports "arbitrary prompt segmentation". Through pre-training on large-scale diverse image data, the model learns a general image understanding ability and can quickly and accurately segment any region in the image without relying on specific class labels.

[0012] 2. Multi-modal region semantic alignment: a deep representation learning technique that fuses image and text information. Its core idea is to map local regions in an image to the corresponding language description in the same semantic space, thereby achieving cross-modal information understanding and alignment. This technique not only focuses on the overall semantic consistency of the image, but also emphasizes the semantic expression at the regional level, allowing the model to identify the specific semantics expressed by a region in the image and correspond to words, phrases, or sentences in human language.

[0013] Multi-modal region semantic alignment has made significant breakthroughs in precision, generalization ability, and zero-shot adaptability. In particular, without relying on human annotation, the CLIP (Contrastive Language-Image Pre-training) model can match any image region with natural language labels, achieving automated semantic recognition and classification.

[0014] 3. Diffusion generation: This technology provides a powerful generation paradigm for deep fusion of images and text. Its basic idea is to generate high-quality images or other modal data from random noise through a gradual "denoising" process. In the context of image segmentation and multi-modal semantic alignment, diffusion generation not only retains the advantages of generation quality, but also embodies the precision improvement of semantic control and regional guidance.

[0015] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the drawings.

[0016] Please refer to Figure 1 , the flowchart of a diffusion generation security optimization method based on multi-modal region semantic alignment provided by the embodiments of the present application, as Figure 1 shown, the diffusion generation security optimization method based on multi-modal region semantic alignment can include: Step S100, obtaining an image to be processed and a text instruction.

[0017] For example, the image to be processed is an image with regional content control requirements, which can include but is not limited to images uploaded by users, public images obtained from the Internet, or images generated by other systems.

[0018] For example, the text instruction is a control command expressed in natural language, which explicitly indicates what operation to perform on a specific region of the image.

[0019] For example, the text instruction can include "blur the specified target in the image" or "code the license plate".

[0020] For example, the above-mentioned regional content control can include but is not limited to one or more of the following: local modification of a specific region, desensitization reconstruction for privacy protection or compliance, and semantic-based content filtering.

[0021] In one example, the image to be processed and the text instruction can be input synchronously, and the text instruction is a text instruction for the image to be processed.

[0022] In another example, the text instruction can be pre-configured. The system can perform uniform regional content control on a single or batch of images to be processed according to a preset text instruction (such as "code the license plate for all images").

[0023] It should be noted that in the embodiments of the present application, the relevant data related to personal information used in the implementation process of the scheme are all used after being explicitly authorized by relevant personnel and complying with relevant privacy policies and specifications. The relevant processing of the relevant data in the use process strictly complies with the applicable data protection policies and specifications, and the relevant processing behavior has obtained the unambiguous authorization of the data subject for the use of its data for the purposes described in the present application based on the explicit knowledge principle.

[0024] Step S110, inputting the image to be processed and the text instruction into the trained multi-modal diffusion model; wherein the trained multi-modal diffusion model automatically determines the sensitive region in the image to be processed in response to the semantics in the text instruction, and performs regional content control on the sensitive region; the multi-modal diffusion model is trained by injecting a multi-branch dLoRA module, and the multi-branch dLoRA module at least includes a general branch and a task branch, the task branch is used to automatically identify the sensitive region according to the semantics of the text instruction, and perform regional content control on the sensitive region.

[0025] In the embodiments of the present application, the trained multi-modal diffusion model can be used to process the image to be processed and the text instruction as multi-modal input, and based on the semantic understanding of the text instruction, the region-level content control of the image to be processed is performed.

[0026] In the embodiments of the present application, the trained multi-modal diffusion model can be used to process the image to be processed and the text instruction as multi-modal input, and based on the semantic understanding of the text instruction, the region-level content control of the image to be processed is performed.

[0027] For example, the trained multi-modal diffusion model can align and interact the semantic embedding vector of the text instruction and the visual feature of the image in the deep semantic space through the internal multi-modal fusion mechanism (such as the cross-attention layer), so that the automatic focusing and positioning of the specific semantic region in the image can be realized without relying on external segmentation model or manually annotated region mask.

[0028] In order to improve the accuracy and efficiency of the region-level content control on the basis of ensuring the quality of the generated image, during the training process of the multi-modal diffusion model, a multi-branch dLoRA (Dynamic Low-Rank Adaptation) module can be injected into the multi-modal diffusion model, and the multi-modal diffusion model injected with the multi-branch dLoRA module can be trained.

[0029] For example, the multi-branch dLoRA module can include at least two functionally decoupled branches: a general branch and a task branch; wherein: The general branch is used to maintain the general generation ability of the model and ensure the basic quality of the image. For example, this branch can focus on learning general image structure and texture knowledge that is not related to a specific task.

[0030] The task branch is used to learn the semantic control ability for specific safety concepts. For example, this branch is specially designed to capture and respond to feature changes related to specific semantic concepts such as safety, compliance, etc.

[0031] For example, through training, the task branch can internalize the mapping relationship between specific semantic concepts (such as "license plate", "violent instrument", etc.) and image region appearance features, so as to automatically lock the corresponding target region in the image space according to the guidance of the text instruction.

[0032] Based on this, the task branch in the trained multi-modal diffusion model can automatically identify sensitive regions in the image to be processed according to the semantics of the text instruction, and perform region-level content control on the identified sensitive regions.

[0033] Exemplarily, for the obtained inverted image to be processed and the text instruction, image feature extraction (e.g., multi-scale feature extraction of the image to be processed) can be performed on the image to be processed by using a visual encoder, and the text instruction can be converted into a semantic embedding vector by using a text encoder (e.g., a CLIP text encoder).

[0034] The task branch can perform automatic identification of the sensitive region based on the semantics of the current text instruction, and perform specific content control processing on the region in the diffusion process. The general branch works in parallel to ensure the generation quality and naturalness of the whole image.

[0035] After the above process, the model finally outputs a high-quality image, in which the sensitive region related to the text instruction has been accurately controlled in content (e.g., blurred, coded, replaced, etc.), and the non-sensitive region has been maximally maintained in the original state.

[0036] Step S120, receiving the processed image output by the trained multi-modal diffusion model.

[0037] In the embodiment of the present application, in the case of inputting the obtained inverted image to be processed and the text instruction into the trained multi-modal diffusion model, the processed image (which has been subjected to region-level content control on the sensitive region) output by the trained multi-modal diffusion model can be received.

[0038] Exemplarily, for the received processed image, it can be returned to the user (e.g., in the case of image request processing submitted by the user), stored in a specified location, or published to a corresponding platform, etc.

[0039] For example, in the content review scenario, the processed image can be automatically marked as "compliant" and enter the publishing queue; in the privacy protection scenario, the processed image can be used for data exchange or public sharing.

[0040] As can be seen, Figure 1In the method flow shown, by acquiring a to-be-processed image and a text instruction, and inputting the acquired to-be-processed image and the text instruction into the trained multi-modal diffusion model, the trained multi-modal diffusion model is used to automatically identify sensitive regions in the to-be-processed image according to the semantics in the text instruction, and to perform region-level content control on the identified sensitive regions. Then, the processed image output by the trained multi-modal diffusion model is received, realizing end-to-end automatic sensitive region identification and region-level content control, without relying on manual annotation or external segmentation model, significantly improving the efficiency of image security processing and reducing application cost; by injecting a multi-branch dLoRA module for training, the task branch automatically identifies and controls sensitive regions according to semantics, and the general branch maintains the basic quality of the image, realizing the decoupling of general generation ability and specific security tasks, while realizing accurate control of sensitive regions, ensuring the image quality and naturalness of non-sensitive regions, and optimizing the overall visual effect.

[0041] In some embodiments, the training of the above multi-modal diffusion model can include: SVD decomposition is performed on the weight matrix of the target linear transformation layer in the pre-trained multi-modal diffusion model, and the parameters of the general branch and the task branch are initialized according to the decomposition result; wherein the target linear transformation layer is a linear transformation layer to which a multi-branch dLoRA module is injected; The initialized general branch and task branch are trained using different dynamic learning rate strategies.

[0042] For example, in order to realize accurate region-level content control in a parameter-efficient manner while maintaining the generation ability of the pre-trained multi-modal diffusion model, a multi-branch dLoRA module can be injected at a specified linear layer (which can be referred to as a target linear layer) of the pre-trained multi-modal diffusion model.

[0043] For example, a multi-branch dLoRA can be injected in the cross-attention module of the pre-trained multi-modal diffusion model.

[0044] For example, during the training process, in order to ensure that the parameter update direction is consistent with the main component of the pre-trained multi-modal diffusion model, speed up the training convergence and improve the stability, the weight matrix of the target linear layer can be subjected to SVD (Singular Value Decomposition) decomposition, and the parameters of the general branch and the task branch are initialized according to the decomposition result.

[0045] In one example, the initialization parameters of the general branch come from the main component direction after SVD decomposition of the weight matrix of the target linear transformation layer; The initialization parameters of the task branch come from the secondary component direction after SVD decomposition of the weight matrix of the target linear transformation layer.

[0046] For example, assuming the weight matrix of the target linear layer is W, it can be decomposed by SVD as follows:

[0047] where, , and U is the left singular vector, is the transpose of the right singular vector.

[0048] According to the SVD decomposition result, the parameters of the general branch and the task branch can be initialized as follows:

[0049] and are the initialization parameters of the general branch, which come from the principal component direction after SVD decomposition, that is,

[0050]

[0051] where, is the first column of the U matrix, is the product of the first most important singular values and the corresponding part, r is the total rank of dLoRA, is the rank of the general branch.

[0052] The above initialization method ensures that the general branch can focus on preserving the general feature representation of the model.

[0053] and are the initialization parameters of the task branch, which come from the secondary component direction after SVD decomposition, that is,

[0054]

[0055] where, is the column to the r-th column of the U matrix, is the product of the to the r-th singular value and the corresponding part.

[0056] The above initialization method provides the task branch with a "potential space" to learn specific security concepts, avoiding interference with general knowledge.

[0057] In order to optimize the two functionally decoupled branches in the training process, and improve the expression ability of important parameters and the generalization ability of the model, different dynamic learning rate strategies can be used to train the initialized general branch and task branch.

[0058] In one example, an exponential decay learning rate can be used for the general branch, and a periodic cosine restart learning rate can be used for the task branch.

[0059] For example, an exponential decay learning rate is used for the general branch:

[0060] wherein, is the learning rate of the general branch at the training step t, is the initial learning rate of the general branch, λ is the decay rate for controlling the speed of learning rate decay, t is the training step, is the lower limit of the learning rate.

[0061] This strategy makes the learning rate of the general branch high at the beginning of training, quickly learns general knowledge, then gradually reduces it for fine adjustment, and finally stabilizes at a basic value to avoid over-adjustment.

[0062] A periodic cosine restart learning rate is used for the general branch:

[0063] wherein, is the learning rate of the task branch at the training step t, is the maximum value of the learning rate, is the minimum value of the learning rate, T is the length of a period (the number of steps in a period), t mod T is the remainder of t divided by T.

[0064] This strategy periodically reduces and restarts the learning rate of the task branch, which helps the model to jump out of the local optimal solution, maintains the exploration ability in complex and variable "task-specific" learning, and improves the generalization.

[0065] It should be noted that in the embodiments of the present application, the task branch can be flexibly extended according to actual safety requirements, and each task branch is independently designed to learn a specific safety concept or semantic category (for example, "license plate blur branch", "violent content filtering branch", "inappropriate symbol recognition branch", etc. can be set up respectively).

[0066] For example, in the case of multiple task branches, the multiple task branches can share the above-mentioned secondary component direction (such as the above-mentioned and or, the above-mentioned secondary component direction is divided according to a specific rule, for example, assuming that there are K (K≥2) task branches, the above-mentioned secondary component direction can be divided into N segments, and respectively assigned to the N task branches as initialization parameters.

[0067] During the training process, each task branch can independently configure its dynamic learning rate parameter. For example, different cosine restart periods and / or maximum learning rates can be set for task branches of different complexities to achieve more refined optimization control.

[0068] For example, during the inference process, the system dynamically activates one or more task branches related to the semantic of the input text instruction. The activated branches jointly act on the generation process to identify and control different types of sensitive regions in the image, thereby achieving composite security compliance processing.

[0069] For example, assuming that the task branches include a "license plate blurring branch", a "violent content filtering branch", and an "improper symbol identification branch", and the text instruction is "please desensitize the license plate and violent content in the image", then the system can dynamically activate two corresponding task branches during the inference process: the "license plate blurring branch" and the "violent content filtering branch".

[0070] During the inference process, the two activated task branches work in parallel, each searching for a target in the image that matches its semantic: License plate blurring branch: locate all vehicle license plates in the image.

[0071] Violent content filtering branch: analyze the image content to determine whether it contains "violent content" mentioned in the instruction.

[0072] The model finally outputs a processed image. In this image: The license plates are blurred (e.g., covered with a mosaic), and the parts identified as violent content are removed or replaced with harmless patterns.

[0073] The other parts of the image, such as streets, buildings, trees, etc., maintain the original high quality and natural appearance.

[0074] In some embodiments, the training of the above-mentioned multi-modal diffusion model can include: During backpropagation, the general branch updates its parameters according to the gradients of the diffusion reconstruction loss and the region semantic supervision loss; The task branch updates its parameters according to the gradients of the diffusion reconstruction loss and the region semantic supervision loss, or the task branch updates its parameters according to the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss; The diffusion reconstruction loss is used to guarantee the basic quality of the generated image, and the region semantic supervision loss is used to ensure the semantic consistency of the generated content with the text instruction.

[0075] For example, in order to guarantee the semantic consistency of the generated content with the text instruction while guaranteeing the quality of the generated image, the loss function can include a diffusion reconstruction loss and a region semantic supervision loss during training of the multi-modal diffusion model.

[0076] The diffusion reconstruction loss can be used to guarantee the basic quality of the generated image, and the region semantic supervision loss can be used to ensure the semantic consistency of the generated content with the text instruction.

[0077] For example, during backpropagation, the general branch can be updated in parameters according to the gradients of the diffusion reconstruction loss and the region semantic supervision loss, and the task branch can be updated in parameters according to the gradients of the diffusion reconstruction loss and the region semantic supervision loss.

[0078] For example, in order to enable the task branch to more accurately identify sensitive regions in the image and improve the accuracy of region-level content control, the loss function can further include a mask consistency loss during training of the multi-modal diffusion model.

[0079] The mask consistency loss can be used to constrain the accurate control of the model on the specified region.

[0080] During backpropagation, the task branch can also be updated in parameters according to the gradient of the mask consistency loss, that is, the task branch can be updated in parameters according to the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss.

[0081] In one example, the diffusion reconstruction loss can be obtained according to the mean square error between the model predicted noise and the real noise.

[0082] In one example, the region semantic supervision loss is determined according to the cosine similarity of the generated image and the text instruction in the shared semantic space.

[0083] In one example, the mask consistency loss is determined according to the overlap degree between the mask region determined by the model and the real mask region.

[0084] For example, the specific determination method of the above loss can be described in the following specific examples.

[0085] In some embodiments, the total loss of the multi-modal diffusion model training is the weighted sum of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss. The weight coefficients of the losses are dynamically adjusted according to the proportion of the original loss value of each loss in the total original loss value.

[0086] For example, considering that different learning tasks corresponding to different loss functions have different convergence characteristics and difficulty levels during training, in order to prevent over-optimization of a certain task from causing performance degradation of other tasks and ensure balanced improvement of the model in image quality, semantic alignment, and region control, a dynamic weight adjustment strategy based on the proportion of the original loss value (the ratio of the original loss value to the total original loss value) can be used to achieve stable convergence and collaborative optimization of multiple objectives during training.

[0087] In one example, the dynamic adjustment can be based on the deviation of the proportion of the original loss value of each loss in the total original loss value from the average proportion. When the deviation is positive, the weight coefficient of the corresponding loss is increased; when the deviation is negative, the weight coefficient of the corresponding loss is decreased. The sum of the weight coefficients of the losses remains a preset value.

[0088] For example, by monitoring the relative proportions of the original loss values, the system can automatically identify the "bottleneck task" in the current training. If the proportion of the original loss value of a certain loss is too high, it indicates that the corresponding task is currently more difficult and requires more attention.

[0089] For example, tasks that perform poorly (tasks with a high proportion of original loss values) can automatically obtain higher weight coefficients and have greater influence in gradient updates. Thus, through the dynamic adjustment mechanism, intelligent allocation of training resources is achieved.

[0090] In addition, to avoid gradient explosion, a sum constraint can be introduced during the dynamic adjustment of the weight coefficients of the losses, i.e., the sum of the weight coefficients of the losses remains a preset value.

[0091] For example, the total loss function of a multi-modal diffusion model can be as follows:

[0092] The initial weights can be set as λ1=1.0, λ2=0.3, and λ3=0.5.

[0093] The weight coefficients of the losses can be dynamically adjusted as follows:

[0094]

[0095]

[0096] wherein is the original loss value of the kth loss at time step t, is the sum of the original loss values of the three losses at time step t (i.e., the total original loss value), is the weight coefficient of the kth loss at time step t, is the weight coefficient of the kth loss at time step t+1, is the sum of the weight coefficients of the three losses.

[0097] Based on the above dynamic adjustment strategy, for a loss, if the proportion of the original loss value of the loss in the total original loss value is less than the average proportion (1 / 3≈0.33), i.e., the deviation of the proportion of the original loss value in the total original loss value from the average proportion is negative, i.e., the original loss value is relatively small (less than the average of the original loss values), the weight coefficient of the loss is reduced; if the deviation of the proportion of the original loss value in the total original loss value from the average proportion is positive, i.e., the original loss value is relatively large (greater than the average of the original loss values), the weight coefficient of the loss is increased.

[0098] Wherein, in the case of determining the adjusted weight coefficients of the losses according to the above strategy, it is also necessary to ensure that the sum of the weight coefficients of the losses is unchanged.

[0099] For example, by normalizing, the sum of the adjusted weight coefficients of the losses can be kept unchanged.

[0100] For example, assuming that the preset value of the sum of the weight coefficients is 1.8. In the case of calculating the preliminary adjusted weight coefficients according to the deviation of the proportion of the original value of each loss from the target average, the ratio of the sum of these preliminary adjusted weight coefficients to the preset value 1.8 can be calculated as a normalization coefficient, and each preliminary adjusted weight coefficient is multiplied by the normalization coefficient to obtain the final weight coefficient, ensuring that the sum is equal to 1.8.

[0101] In some embodiments, the mask consistency loss is determined according to the overlap between the mask region determined by the model and the true mask region; The true mask region is determined by the following method: For any sample image, the SAM model is used to perform initial candidate region division on the sample image, and the N initial candidate regions with the highest scores are determined as the final candidate regions according to the saliency scores of the initial candidate regions; wherein N≥2; According to the similarity between the image features of the original sample image block corresponding to each final candidate region and the text features of the preset text security label, the final candidate region whose similarity between the image features and the text features of any preset text security label exceeds the preset similarity threshold is determined as a sensitive region; According to the determined sensitive region, a mask region of the sample image is determined.

[0102] For example, in the training process of the multi-modal diffusion model, a region recognition mechanism combining salient region decomposition and semantic-driven screening can be used to automatically identify sensitive regions in the sample image.

[0103] For example, a SAM (Segment Anything Model) model can be used to divide the sample image into salient regions (also referred to as initial candidate regions), and generate initial candidate region masks.

[0104] The SAM model has zero-shot segmentation capability and can automatically extract salient regions in an image without labeled data, so that initial candidate regions can be obtained without manual labeling.

[0105] To improve the efficiency and reliability of subsequent semantic judgment, the initial candidate regions can be ranked according to their saliency intensity (such as region area, boundary gradient change, and / or image contrast), and the top N initial candidate regions with the highest scores can be selected as the final candidate regions.

[0106] For example, for any final candidate region, the image features of the original sample image block corresponding to the candidate region (i.e., the image block at the position of the candidate region in the sample image) can be extracted, and the similarity between the image features of each sample image block and the text features of the preset text safety label can be determined.

[0107] The text features of the preset text safety label can be obtained by extracting the text features of the preset text safety label.

[0108] If the similarity between the image features of the original sample image block and the text features of any preset text safety label exceeds a preset similarity threshold, the final candidate region can be determined as a sensitive region, wherein the mask corresponding to the sensitive region is the sensitive region mask.

[0109] For example, according to the sensitive region determined in the above manner, a mask region of the sample image (referred to as a real mask region) can be determined.

[0110] For example, in the training process of the multi-modal diffusion model, for any sample image, the mask consistency loss can be determined according to the overlap between the mask region of the sample image predicted by the multi-modal diffusion model (i.e., the sensitive region in the sample image predicted by the multi-modal diffusion model) and the real mask region in the sample image.

[0111] In order to make the skilled in the art better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described below in combination with specific examples.

[0112] In this embodiment, a diffusion generation model security control scheme integrating automatic region segmentation, region-level multi-modal semantic alignment, multi-branch low-rank structure injection, and multi-level adaptive compliance determination is provided, and the implementation process can include: 1. Automatic region segmentation: using a SAM model to automatically decompose candidate regions for any input image without manual annotation, obtaining multiple sets of candidate region masks; 2. Region-level multi-modal alignment: for each candidate region (region in the input image corresponding to the candidate region mask), the CLIP model is used to extract the region image features and safety text semantic features, and determine whether the region has sensitive risks through semantic space similarity; 3. Multi-branch dLoRA structure injection: injecting a multi-branch (such as a double-branch) dLoRA structure based on SVD initialization into the cross-attention module of the U-Net backbone, independently optimizing the parameters of the general task and the sensitive region respectively, and ensuring efficient decoupling and dynamic adaptation of the parameters; 4. Adaptive multi-level semantic determination and feedback: in the training stage, introduce a region-level loss function and an adaptive weight adjustment mechanism to realize continuous feedback optimization and safety control for different risk levels; 5. Highly automated whole process: suitable for large-scale task expansion in unsupervised / weakly supervised scenarios.

[0113] The implementation process described above will be described in detail below.

[0114] I. Automatic region segmentation.

[0115] For example, for a sample image, the sample image can be input into a SAM model, and multiple candidate sensitive region masks can be output by the SAM model.

[0116] For example, for multiple candidate region masks (which can be referred to as initial candidate region masks) output by the SAM model, the regions corresponding to each initial candidate region mask can also be scored and sorted according to their saliency intensity, and the initial candidate region mask corresponding to the top N regions is selected as the final candidate region mask to reduce the consumption of low-value regions on computing resources and improve the efficiency and reliability of subsequent semantic judgment.

[0117] II. Region-level multi-modal alignment.

[0118] Exemplarily, for any candidate region mask (such as the final candidate region mask), a corresponding region image block can be cropped from the sample image according to the candidate region mask, and after normalization, it is input into the text-image alignment semantic judgment module, and by comparing with the preset multi-label safety semantic vector set, the semantic relevance vector thereof is obtained.

[0119] Exemplarily, the candidate region mask can be traversed, the corresponding region image in the original sample image is cropped according to the candidate region mask to obtain a region image block, and then the region image block is scaled to the CLIP standard size (224*224) and input into the CLIP image encoder to extract image features to obtain an image feature vector (also referred to as a region image vector) v i .

[0120] In addition, for a set of predefined text safety labels, such as a safety sensitive word set S = (“violent”, “Dangerous articles”, “unsafe for work”), CLIP text encoding can be performed to obtain a text feature vector (also referred to as a text semantic vector) t j .

[0121] The region image vector and the text semantic vector are normalized respectively:

[0122]

[0123] Exemplarily, the semantic similarity (such as using a cosine similarity function) between each region image vector and the text semantic vector can be calculated, and compared with a set threshold θ.

[0124] Exemplarily, the similarity matrix can be as follows:

[0125] wherein d represents the dimension index of the vector, and the image feature vector and the text feature vector output by the CLIP model are usually 512-dimensional, and the summation operation is used to calculate the inner product of each dimension, so as to obtain the semantic cosine similarity between the region image block and the text safety label.

[0126] Exemplarily, for any candidate region mask, if the similarity between the region image block corresponding to the candidate region mask and any safety sensitive word (or referred to as a text safety label) is greater than the threshold θ, the candidate region mask is marked as a sensitive region mask, and the implementation flowchart can be as shown in Figure 2 .

[0127] Exemplarily, the sensitive region masks can be merged to obtain a final sensitive region mask.

[0128] For example, for all sensitive region masks, they can be merged at pixel level in image space to construct the final sensitive region mask M of the image gt .

[0129] M gt As the real sensitive region supervision signal in the training stage, it is used to calculate the mask consistency loss L mask .

[0130] III. Multi-branch dLoRA structure injection.

[0131] For example, inject a multi-branch dLoRA module in the U-Net backbone of a multi-modal diffusion model (such as the Stable Diffusion model).

[0132] For example, take a two-branch dLoRA module as an example, which includes a general branch and a task branch, and can adapt to common structures and local region changes respectively.

[0133] For example, a two-branch dLoRA module can be injected in a cross-attention layer.

[0134] The pre-training weight matrix can be decomposed by SVD, and the general branch and the task branch can be initialized according to the SVD decomposition results respectively.

[0135] The parameters of the general branch and the task branch can be initialized according to the SVD decomposition results in the following way:

[0136] And The initialization parameters of the general branch come from the principal component direction after SVD decomposition, that is:

[0137]

[0138] Where, is the first column of the U matrix, is the product of the first most important singular values and the corresponding part, r is the total rank of dLoRA, is the rank of the general branch.

[0139] The above initialization method ensures that the general branch can focus on maintaining the general feature representation of the model.

[0140] And The initialization parameter for the task branch is the minor component direction after SVD decomposition, that is,

[0141]

[0142] wherein, is the product of the first r columns of the U matrix, is the product of the first r singular values and the corresponding

[0143] The above initialization method provides the task branch with a "potential space" for learning specific safety concepts, avoiding interference with general knowledge.

[0144] In order to optimize the two functionally decoupled branches in the training process and improve the expression ability of important parameters and the generalization ability of the model, different dynamic learning rate strategies can be used to train the initialized general branch and task branch.

[0145] In one example, an exponentially decaying learning rate can be used for the general branch, and a periodic cosine restart learning rate can be used for the task branch.

[0146] For example, an exponentially decaying learning rate is used for the general branch:

[0147] wherein, is the learning rate of the general branch at training step t, is the initial learning rate of the general branch, λ is the decay rate for controlling the speed of learning rate decay, t is the training step, is the lower limit of the learning rate.

[0148] This strategy makes the learning rate of the general branch high at the beginning of training, quickly learns general knowledge, then gradually reduces it for fine adjustment, and finally stabilizes at a basic value to avoid over-adjustment.

[0149] A periodic cosine restart learning rate is used for the general branch:

[0150] wherein, is the learning rate of the task branch at training step t, is the maximum value of the learning rate, is the minimum value of the learning rate, T is the length of a period (the number of steps in a period), t mod T is the remainder of t divided by T.

[0151] ​​​Exemplary, the model training process in the case of injecting dLoRA module can be as shown in Figure 3 .

[0152] This strategy periodically reduces and restarts the learning rate for task branches, helping the model to jump out of local optimal solution, maintain exploration ability in complex and variable "task-specific" learning, and improve generalization.

[0153] Four, adaptive multi-level semantic judgment and feedback.

[0154] Exemplary, in the model training process, the loss function fuses diffusion reconstruction loss (L recon ), regional semantic supervision loss (L clip ) and optional mask consistency loss (L mask ), jointly optimizing image generation quality and safe semantic alignment.

[0155] Exemplary, the region mask and the sample image can be input into the diffusion model respectively (the region mask and the sample image are input into the diffusion model through different input channels respectively, the region mask can be used as a real mask region to realize region mask supervision).

[0156] 1, input M gt and sample image.

[0157] 2, get the predicted noise through UNet, calculate L recon :

[0158] Where x t is the noisy image at time step t, generated by the forward diffusion process.

[0159] is the real Gaussian noise, which obeys N(0, I), that is, the standard normal distribution with mean 0 and variance 1.

[0160] is the noise predicted by UNet, and the model parameters θ include the parameters of the dLoRA module being trained.

[0161] The model learns the denoising mapping by minimizing the L2 norm square error of the two.

[0162] 3, calculate the cosine similarity of image-text pair through CLIP encoder E I and E T :

[0163] Where E Iis the CLIP image encoder, outputting a 512-dimensional normalized vector; E T is the CLIP text encoder, outputting a vector of the same dimension; x gen is the image generated by the model.

[0164] wherein the image in the image-text pair refers to the generated whole-frame image, and the text refers to the text instruction, and the cosine similarity of the image-text pair is used to constrain the overall semantic consistency of the generated result.

[0165] 4. In the presence of region mask supervision, calculate the Dice loss:

[0166] wherein, is the sensitive region mask distribution predicted by the diffusion model, is the real sensitive region mask distribution.

[0167] 5. Total loss function:

[0168] wherein, the initial weights can be set as λ1=1.0, λ2=0.3, λ3=0.5.

[0169] The weight coefficients of each loss can be dynamically adjusted in the following ways:

[0170]

[0171]

[0172] wherein, is the original loss value of the kth loss at time step t, is the sum of the original loss values of the three losses at time step t (i.e., the total original loss value), is the weight coefficient of the kth loss at time step t, is the weight coefficient of the kth loss at time step t+1, is the sum of the weight coefficients of the three losses.

[0173] The final multi-objective joint training mechanism of the system contains three sub-loss functions: L recon (diffusion reconstruction), L clip (semantic alignment), and L mask (mask smoothing), which are weighted and aggregated by the region mask into the total loss L totaland used for back propagation optimization model, finally generating desensitization, compliance, high-quality images, its training implementation flowchart can be as shown in Figure 4

[0174] Exemplary, complete multi-modal region semantic alignment-based diffusion generation security optimization scheme implementation process can be as shown in Figure 5

[0175] As can be seen, based on the above implementation process, the potential semantic regions in the image are automatically determined based on the multi-modal semantic alignment mechanism, and the determined regions are matched and controlled with the corresponding text security concepts end to end, realizing the fine-grained interaction alignment of multi-modal information at the region level, breaking through the single and extensive process of "whole image label-whole image optimization" in the prior art, making the content security control have decision explainability (can explicitly trace the text semantics on which the control of a specific region is based) and precise regional orientation (ensuring that the security operation is strictly limited to the target semantic region, and the non-sensitive content is preserved to the maximum extent), thereby applicable to large-scale non-human intervention compliance image generation tasks with extremely high automation requirements.

[0176] Secondly, compared with the traditional scheme relying on static text matching or single-stage filtering, the embodiment of the present application constructs a multi-stage, multi-modal dynamic compliance control system deeply integrated in the diffusion generation process, changes the security judgment from single inspection before and after generation to continuous semantic verification throughout each denoising step, jointly evaluates the deep semantics of visual features and text instructions in the generation process through dynamic fusion, and can adaptively adjust the generation path according to the verification result, discards the traditional coarse-grained control mode relying on fixed shielding rules, and establishes a dynamic intervention mechanism based on cross-modal deep semantics. Among them, the judgment mechanism based on cross-modal deep semantics effectively resists adversarial attacks such as character replacement, significantly improves the accuracy and robustness of the system; the multi-stage continuous iteration judgment system constitutes a defense in depth, greatly reduces the missed detection rate of illegal content; at the same time, the fine-grained regional control mechanism avoids the damage to the compliance image, ensures the security while significantly improving the usability and controllability of the generation result, and provides a reliable technical foundation for large-scale non-human intervention compliance image generation.

[0177] ​​Further, compared with the full-parameter fine-tuning method of traditional CLIP or base diffusion model, the LoRA method with high parameter efficiency is introduced in the embodiments of the present application, and a multi-branch structure is innovatively adopted: by setting multiple LoRA branches independent of each other for different security concepts or semantic categories, each branch only needs to train a small number of low-rank matrices on the premise that the pre-trained model backbone parameters remain completely unchanged. This technical path breaks through the adjustment mode of traditional single adapter, and realizes efficient learning of multiple security semantics through branch specialization on the basis of strictly preserving the original model knowledge from being forgotten, significantly improving the recognition ability of diversified illegal content; since only a small number of additional parameters need to be trained, the training time and computing resources are greatly reduced, which embodies excellent parameter efficiency; the branched independent design provides flexible and controllable ability for the generation process, supports dynamic enabling or disabling of specific branches according to actual needs, and greatly enhances the safety controllability of generated content.

[0178] Finally, through layer-by-layer fine-grained semantic verification from global to regional, and according to real-time risk dynamic adjustment control strategy, the accuracy and robustness of compliance judgment are systematically improved, and fine control of generated content is realized. The multi-level verification system enables the model to more accurately distinguish between safe and illegal content, effectively reducing the risk of false positives and false negatives.

[0179] The above describes the method provided by the present application. The following describes the device provided by the present application: Please refer to Figure 6 The structure diagram of the diffusion generation safety optimization device based on multi-modal regional semantic alignment provided by the embodiments of the present application is as shown in Figure 6 The diffusion generation safety optimization device based on multi-modal regional semantic alignment can include: An acquisition unit is configured to acquire an image to be processed and a text instruction; An output unit is configured to input the image to be processed and the text instruction into a trained multi-modal diffusion model; wherein the trained multi-modal diffusion model automatically determines a sensitive region in the image to be processed in response to semantics in the text instruction, and performs regional-level content control on the sensitive region; the multi-modal diffusion model is trained by injecting a multi-branch dynamic low-rank adaptation dLoRA module, and the multi-branch dLoRA module includes at least a general branch and a task branch, the task branch is configured to automatically identify the sensitive region according to the semantics of the text instruction, and perform regional-level content control on the sensitive region; A receiving unit is configured to receive an image processed by the trained multi-modal diffusion model.

[0180] For example, the specific implementation process of the diffusion generation security optimization scheme based on multimodal region semantic alignment in each unit of the above-mentioned diffusion generation security optimization device can be found in the relevant description in the above embodiments, and will not be repeated here in the embodiments of this application.

[0181] This application also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; the processor is used to execute the program stored in the memory to implement the diffusion generation security optimization method based on multimodal region semantic alignment described above.

[0182] Please see Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 701 and a memory 702 storing machine-executable instructions. The processor 701 and the memory 702 can communicate via a system bus 703. Furthermore, by reading and executing the machine-executable instructions corresponding to the diffusion generation security optimization logic based on multimodal region semantic alignment in the memory 702, the processor 701 can execute the diffusion generation security optimization method based on multimodal region semantic alignment described above.

[0183] The memory 702 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0184] In some embodiments, a machine-readable storage medium, such as Figure 7 The memory 702 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the diffusion generation security optimization method based on multimodal region semantic alignment described above. For example, the machine-readable storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0185] This application also provides a computer program product storing a computer program, which, when executed by a processor, causes the processor to execute the diffusion generation security optimization method based on multimodal region semantic alignment described above.

Claims

1. A diffusion generation security optimization method based on multimodal region semantic alignment, characterized in that, include: Acquire the image to be processed and the text instructions; The image to be processed and the text command are input into a trained multimodal diffusion model. The trained multimodal diffusion model, responding to the semantics of the text command, automatically identifies sensitive regions in the image to be processed and performs region-level content control on these sensitive regions. The multimodal diffusion model is trained by injecting a multi-branch dynamic low-rank adaptive dLoRA module. The multi-branch dLoRA module includes at least a general branch and a task branch. The task branch is used to automatically identify the sensitive regions based on the semantics of the text command and perform region-level content control on these sensitive regions. Receive the processed image output by the trained multimodal diffusion model.

2. The method according to claim 1, characterized in that, The training of the multimodal diffusion model includes: The weight matrix of the target linear transformation layer in the pre-trained multimodal diffusion model is decomposed by singular value decomposition (SVD), and the parameters of the general branch and the task branch are initialized according to the decomposition results; wherein, the target linear transformation layer is a linear transformation layer injected with the multi-branch dLoRA module; Different dynamic learning rate strategies are used to train the initialized general branch and task branch.

3. The method according to claim 2, characterized in that, The initialization of the parameters of the general branch and the task branch based on the decomposition results includes: The initialization parameters of the general branch are derived from the principal component directions after the SVD decomposition of the weight matrix of the target linear transformation layer; The initialization parameters of the task branch are derived from the minor component directions after the SVD decomposition of the weight matrix of the target linear transformation layer; And / or, The process of training the initialized general branch and task branch using different dynamic learning rate strategies includes: An exponentially decaying learning rate is applied to the general branch; The learning rate is periodically restarted using a cosine function for the task branch.

4. The method according to claim 1, characterized in that, The training of the multimodal diffusion model includes: During backpropagation, the general branch updates its parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss. The task branch updates its parameters based on the gradients of the diffusion reconstruction loss and the region semantic supervision loss, or the task branch updates its parameters based on the gradients of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss. The diffusion reconstruction loss is used to ensure the basic quality of the generated image, the region semantic supervision loss is used to ensure the semantic consistency between the generated content and the text instructions, and the mask consistency loss is used to constrain the model's precise control over the specified region.

5. The method according to claim 4, characterized in that, The diffusion reconstruction loss is obtained based on the mean square error between the model-predicted noise and the actual noise. And / or, The region semantic supervision loss is determined based on the cosine similarity between the generated image and the text instruction in the shared semantic space; And / or, The mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region.

6. The method according to claim 4, characterized in that, The total loss during training of the multimodal diffusion model is a weighted sum of the diffusion reconstruction loss, the region semantic supervision loss, and the mask consistency loss. The weighting coefficient of each loss is dynamically adjusted based on the proportion of the original loss value of each loss in the total original loss value.

7. The method according to claim 6, characterized in that, The dynamic adjustment is based on the deviation between the proportion of each loss's original loss value in the total original loss value and the average proportion. Specifically, when the deviation is positive, the weighting coefficient of the corresponding loss increases; when the deviation is negative, the weighting coefficient of the corresponding loss decreases. The sum of the weighting coefficients for each loss remains at a preset value.

8. The method according to claim 4, characterized in that, The mask consistency loss is determined based on the overlap between the mask region determined by the model and the actual mask region; The actual mask region is determined in the following way: For any sample image, the Segmentation All Model (SAM) is used to divide the sample image into initial candidate regions. Based on the saliency score of each initial candidate region, the N salient regions with the highest scores are determined as the final candidate regions; where N≥2. Based on the similarity between the image features of the original sample image block corresponding to each final candidate region and the text features of the preset text security label, the final candidate region whose image features are more than the same as the text features of any preset text security label are determined as a sensitive region. Based on the sensitive area, the true mask area of ​​the sample image is determined.

9. An electronic device, characterized in that, Including processor and memory, among which, Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product contains a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 8.

Images