A multi-agent style transfer method based on a diffusion model

Abstract

The application provides a multi-agent style transfer method based on a diffusion model, and relates to the fields of generative artificial intelligence and computer vision. The application acquires a content image and a style image and generates a structure condition image; encodes the style image to obtain a style embedding; encodes the content image multiple times to obtain multiple content embeddings and calculates injection weights; in a diffusion denoising process, the style embedding is injected through cross attention, and the style injection intensity is scheduled step by step in sections; in each denoising step, the content injection residual is injected in a weighted manner in a polling mode, and an early stop threshold is set; and a structure residual is injected through a structure guiding network. The application improves structure preservation and multi-agent stability and reduces inference time.

Description

Technical Field

[0001] This invention belongs to the field of generative artificial intelligence and computer vision, and specifically relates to a multi-agent style transfer method based on a diffusion model. Background Technology

[0002] While diffusion models excel in text-to-image generation tasks, they still have shortcomings in style transfer scenarios where "strict content structure preservation" is crucial. For example, when the style injection intensity remains constant throughout the denoising process, high-frequency texture stacking can easily occur in the later stages, disrupting the subject outline and background structure. Furthermore, multi-subject content representations are typically injected using a simple stacking method, which can lead to subject contamination when the content embedding quality is unstable (e.g., random augmentation, occlusion, or noise perturbation). Additionally, there is a conflict between structure guidance and style injection; fixed parameters make it difficult to simultaneously ensure style consistency and structure preservation. Therefore, a multi-subject style transfer method is needed that improves the stability of multi-subject injection and reduces computational redundancy while maintaining structural consistency. Summary of the Invention

[0003] To address the problems of style injection conflict with structure preservation, unstable multi-subject content embedding leading to subject contamination, and computational redundancy in the denoising process in existing technologies, this application proposes a multi-subject style transfer method based on a diffusion model, which improves the stability of multi-subject injection and reduces inference time while maintaining structural consistency.

[0004] This invention provides a multi-agent style transfer method based on a diffusion model, comprising the following steps:

[0005] Acquire content images and style images, generate a structure condition map based on the content image, and generate a structure embedding based on the structure condition map;

[0006] The style image is encoded to obtain the style embedding;

[0007] The content image is randomly augmented and encoded multiple times to obtain multiple augmented images and corresponding content embeddings; injection weights are calculated based on the augmented images;

[0008] Obtain style transfer cue words, encode the style transfer cue words, and generate text embeddings;

[0009] The structure embedding, style embedding, content embedding, and injection weights are input into the diffusion model. Under the semantic conditions provided by the text embedding, the model outputs an image after several denoising steps.

[0010] Specifically, for each denoising step, style injection is performed on the output features of the previous denoising step based on style embedding and the set style injection intensity to obtain the first intermediate state of each denoising step; content injection is performed on the first intermediate state based on injection weight and content embedding to obtain the second intermediate state of each denoising step; and structure injection is performed on the second intermediate state based on structure embedding to obtain the output features of each denoising step.

[0011] Furthermore, the content image is the original image that needs to maintain the main structure and outline; the style image is a reference image that provides texture, color, and artistic style; and the structural condition map is the geometric structural information used to characterize the content image.

[0012] The structural condition map is obtained by performing Canny edge detection on the content image, and the structural embedding is obtained by encoding the structural condition map.

[0013] Furthermore, the specific method for encoding the style image to obtain the style embedding is as follows:

[0014] The style image is encoded using an image encoder, which maps the style image into a high-dimensional feature vector; the high-dimensional feature vector is then converted into a style embedding through a projection layer.

[0015] Furthermore, the specific method for performing multiple random augmentations and encodings on the content image is as follows:

[0016] Perform content image Each random augmentation yields an augmented image, and so on. After random augmentation, we get Zhang Zengguang image; the random augmentation includes horizontal flipping and grayscale perturbation operations;

[0017] To each Zhang Zengguang's image was encoded to obtain Content embedding.

[0018] Furthermore, the specific method for calculating injection weights based on augmented images is as follows:

[0019] Edge operator processing is performed on the augmented image to generate grayscale edge images; the pixel mean of the grayscale edge images is calculated to obtain the reliability score of each augmented image;

[0020] The reliability score of each augmented image is normalized to obtain the injection weights.

[0021] Furthermore, the text embedding participates in the diffusion denoising process through cross-attention to provide semantic condition constraints.

[0022] Furthermore, the style injection intensity is determined based on the denoising stage to which the denoising step belongs, specifically by:

[0023] Set the base injection intensity and the ratio parameters for dividing the denoising stages, divide the denoising steps into different denoising stages, and adjust the style injection intensity of each denoising step according to the denoising stage and the base injection intensity.

[0024] Furthermore, the specific method for style injection of the output features from the previous denoising step based on style embedding and the set style injection intensity is as follows:

[0025] Cross-attention calculation is performed on the style embedding to obtain the style injection residual; style injection is performed on the output features of the previous denoising step according to the style injection residual and the set style injection intensity to obtain the first intermediate state;

[0026] The specific method for injecting content into the first intermediate state based on injection weights and content embedding is as follows:

[0027] right Cross-attention calculation is performed on each content embedding to obtain... The first intermediate state is injected with content injection residuals; the second intermediate state is obtained by injecting content into the first intermediate state according to the content injection residuals and injection weights.

[0028] When the When the norm of the content injection residual is less than the preset early stopping threshold, the content injection of the remaining content embedding in the denoising step is terminated. ;

[0029] The specific method for structural injection into the second intermediate state based on structural embedding is as follows:

[0030] The structure is embedded into the input structure preservation module to generate the structure injection residual; the structure injection is performed on the second intermediate state according to the structure injection residual and the set structure injection intensity to obtain the output features of the denoising step.

[0031] The structure preservation module uses a ControlNet network.

[0032] Furthermore, the basic injection intensity and the structural injection intensity are adjusted through a global adaptive adjustment mechanism based on edge density;

[0033] Global adaptive adjustment mechanisms based on edge density include:

[0034] Calculate the edge density of the structural condition diagram and the conflict metric; adjust the foundation injection intensity and structural injection intensity based on the conflict metric.

[0035] The beneficial effects of adopting the above technical solution are as follows: The multi-agent style transfer method based on a diffusion model provided by this invention reduces the interference of low-quality content embeddings on the generated results and improves multi-agent consistency by weighting the content embeddings obtained from multiple augmentations on reliability. During the diffusion denoising process, style embeddings are injected through cross-attention and the style injection intensity is gradually scheduled in segments. Style expressiveness is maintained in the early stages of denoising, and texture injection is reduced in the later stages, thereby improving structural stability and reducing over-stylization. Within each denoising step, the injected content residuals are weighted in a polling manner, and an early stopping threshold is set to reduce subsequent injection calculations with smaller contributions, thus reducing inference time. Attached Figure Description

[0036] Figure 1 A schematic diagram of the multi-subject style transfer method based on the diffusion model provided in Embodiment 1 of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are partial embodiments, not all embodiments.

[0038] Example 1:

[0039] Unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as commonly understood by one of ordinary skill in the art to which this embodiment belongs. To facilitate understanding of the technical solutions of this embodiment, the main symbols and parameters involved in this embodiment are defined as follows:

[0040]

[0041] A multi-agent style transfer method based on a diffusion model, such as Figure 1 As shown, it includes the following steps:

[0042] Step 1: Obtain the content image and style image, generate a structure condition map based on the content image, and generate a structure embedding based on the structure condition map;

[0043] In this embodiment, the content image refers to the original image that needs to maintain the main structure and outline, such as a photograph containing people or scenery; the style image refers to a reference image that provides texture, color and artistic style; and the structural condition map is used to characterize the geometric structure information of the content image.

[0044] This embodiment preferably uses Canny edge detection to generate the structure condition map, setting the first threshold for edge detection to 50 and the second threshold to 200. By setting specific edge detection thresholds, the contour information of the content image can be extracted more accurately, thereby enhancing the accuracy of structure guidance. It should be understood that in other optional embodiments, the structure condition map can also be generated using methods such as depth map estimation or semantic segmentation map extraction, but these methods require additional experimental verification of their impact on generation quality and are not within the scope of verification in this embodiment. By extracting the structure condition map, a foundation is provided for strictly constraining the image structure in the subsequent generation process.

[0045] Specifically, the Canny edge detection algorithm is a multi-level edge detection algorithm that uses a dual-threshold mechanism to filter strong and weak edges, thereby suppressing noise while preserving true contour information. In this embodiment, the first threshold (low threshold) is set to 50, and the second threshold (high threshold) is set to 200. When the gradient value of a pixel in the content image is higher than the second threshold of 200, the point is identified as a strong edge; when the gradient value is between the first threshold of 50 and the second threshold of 200, the point is identified as a weak edge. Only when a weak edge point is connected to a strong edge point will it be retained in the final edge map; points with gradient values ​​lower than the first threshold of 50 are considered noise and filtered out. This specific threshold setting can typically filter out background texture noise and subtle high-frequency interference in the content image in the application scenario of this embodiment, helping to extract the main contour skeleton of the subject, thereby providing clearer geometric constraints for subsequent structural guidance. It should be understood that although this embodiment preferably uses 50 and 200 as the threshold combination, in other application scenarios, the threshold can be adaptively adjusted according to the noise level and contrast of the image, as long as effective contour extraction can be achieved.

[0046] Step 2: Encode the style image to obtain the style embedding;

[0047] Style images are encoded using an image encoder, which can be a pre-trained visual encoder, such as the CLIP image encoder, to map style images into high-dimensional feature vectors.

[0048] The high-dimensional feature vector is transformed into a style embedding that matches the latent space dimension of the diffusion model through a projection layer. The style embedding contains global style features of the style image and is used to guide the style direction of the generated image during the denoising process. It should be noted that the style embedding `style_prompt_embedding` is a sequence of style cue tokens encoded and projected from the style image, which can also be denoted as `style_prompt_embeds` in the specific implementation.

[0049] Step 3: Process the content image The random augmentation and encoding are then used to encode the following results: Zhang Zengguang's image and corresponding content are embedded, and injection weights are calculated based on the augmented image;

[0050] In this embodiment, the content image is processed. After random augmentation and encoding, we obtain The image is augmented by Zhang Zengguang, and the corresponding content embedding is `content_emb_i`. Random augmentation preferably employs horizontal flipping and grayscale perturbation operations to simulate different content change perspectives, thereby enhancing the model's robustness to the main content. Each augmented image is encoded to obtain the corresponding content embedding. It should be understood that in other alternative implementations, random augmentation can also employ operations such as random cropping, rotation, color dithering, or occlusion; however, these operations require additional experimental verification of their impact on generation quality and are not within the scope of verification in this embodiment.

[0051] To evaluate the quality of each content embedding, this step calculates a reliability score for each augmented image. The reliability score From the first The sub-enhanced image performs edge operators. The mean pixel value of the grayscale edge image is then obtained. Through reliability assessment based on edge strength, low-quality content embeddings can be effectively identified, reducing their interference with the generated results. In this embodiment, the FIND_EDGES operator is preferably used as the edge operator, as it helps extract edge and contour information from the image. It should be understood that in other embodiments, commonly used edge detection operators such as the Sobel operator, Prewitt operator, or Laplacian operator can also be used.

[0052] Taking the FIND_EDGES operator as an example, it highlights edge information by detecting regions in an image where grayscale changes are drastic. For the... After random augmentation, the image is processed using edge operators to obtain a grayscale edge image. This grayscale edge image is first normalized to the [0,1] interval to ensure comparability of reliability scores between different images. Subsequently, the average value of all pixel values ​​in the normalized grayscale edge image is calculated to obtain the reliability score. Reliability score This intuitively reflects the sharpness and edge density of the main structure in the augmented image: if random augmentation operations (such as blurring, occlusion, or color dithering used in other alternative implementations) damage the main structure of the image, the extracted edges may become sparse or weak, resulting in a corresponding reliability score. A lower score indicates poor reliability of the embedded content; conversely, a higher score indicates better reliability if the augmented image retains a clear structure. The reliability score is relatively high. It reflects the clarity or integrity of the main structure in the augmented image. Through this reliability score, this embodiment can quantitatively evaluate the quality of each content embedding, providing a basis for subsequent weighted injection.

[0053] The reliability score calculated based on each augmented image is normalized and transformed into injection weights, which are shown in the following formula:

[0054] ;

[0055] in, This represents the sum of all reliability scores. For example, to preset a very small positive number, Or even smaller. Normalization ensures a reasonable allocation of injection weights, avoiding generation bias caused by weight imbalance. This embodiment uses normalization to convert reliability scores into injection weights, allowing high-quality content embeddings to occupy a larger proportion in subsequent injection processes, thereby reducing the interference of low-quality embeddings on the generation results.

[0056] Specifically, after calculating all After obtaining the reliability score for each augmented image, it needs to be normalized to obtain the final injection weights. Introducing a pre-defined minimum positive number. The main purpose is to enhance the stability of numerical computation and prevent issues from arising in extreme cases (such as when the edge information of all augmented images is extremely weak, causing all reliability scores to drop). (If all values ​​are close to 0, the denominator is zero, which can lead to a division-by-zero error or numerical overflow.) The value of is extremely small, and its impact on the calculation results can be ignored under normal circumstances, but it can ensure the robustness of the system under abnormal circumstances.

[0057] In one specific implementation of this embodiment, if the sum of all reliability scores Less than or equal to the preset minimum threshold (like If the weighted injection strategy is not specified, then an equal-weighted injection strategy is adopted, that is, all injection weights are set to equal weights. The equal-weight injection strategy addresses extreme cases: when the edge information of all augmented images is very weak (e.g., the content image itself has an extremely simple structure or the augmentation operation causes a severe loss of structural information in all images), the reliability score-based weighting mechanism may fail. In this case, equal-weight injection can avoid the degradation of generation quality caused by improper weight allocation. Through this adaptive weight allocation mechanism, this embodiment can maintain the stability of the generation results in different scenarios.

[0058] Through the aforementioned weighting mechanism, this embodiment achieves dynamic adjustment of the content injection strength. Specifically, regarding the reliability score... Higher content embedding corresponds to higher injection weight. A larger value means that the embedding has a greater impact on the generated results during style transfer; while for reliability scores... Lower quality embeddings, their injection weights The image is significantly compressed. This design effectively reduces the interference of low-quality embeddings such as noise, occlusion, or blurring caused by random augmentation on the final generated image, and avoids the problem of subject contamination or structural distortion caused by a single low-quality embedding dominating the generated result, thereby improving the consistency and stability of multi-subject style transfer.

[0059] Step 4: Obtain style transfer cue words, encode the style transfer cue words, and generate text embeddings;

[0060] Text embedding is used to constrain the consistency between the generated image and the input text description during diffusion denoising.

[0061] Step 5: Input the structure embedding, style embedding, content embedding and injection weights into the diffusion model. Under the semantic conditions provided by the text embedding, after a preset number of denoising steps, output the generated image.

[0062] Step 5.1: Set the ratio parameter for dividing the denoising stages, divide the denoising steps into different denoising stages, and adjust the style injection intensity of each denoising step according to the denoising stage.

[0063] The diffusion model includes a cross-attention mechanism, which allows external conditions (such as style cue embeddings) to influence the denoising process. The process of generating an image using the diffusion model involves multiple denoising steps, progressively recovering a sharp image from Gaussian noise. During diffusion denoising, the style cue embedding is injected into the attention module of the diffusion model, preferentially into the cross-attention module, and the style injection intensity is adjusted according to the denoising step index. Perform segmented, step-by-step scheduling;

[0064] This embodiment uses a segmented, step-by-step scheduling strategy to dynamically adjust the style injection intensity. For example, a high injection intensity is maintained in the early stages of denoising to establish the stylistic tone, while the injection intensity is reduced in the later stages of denoising to avoid over-stylization that could damage structural details. This dynamic adjustment mechanism balances stylistic consistency with structural preservation.

[0065] In this embodiment, the ratio parameters for dividing the denoising stages are set as p1 and p2, and the denoising steps are divided into three stages, namely the first stage, the second stage and the third stage.

[0066] When the denoising step index At that time, the first Each denoising step is divided into the first stage and uses a base injection intensity. The style injection intensity of each denoising step in the first stage is set. ,in, Based on injection strength, This represents the total number of noise reduction steps.

[0067] When the denoising step index satisfy At that time, the first The denoising steps are divided into two stages, and the style injection intensity for each denoising step in the second stage is set. To linearly decay from the base injection intensity to the lower limit of the style injection intensity ,in, Inject a lower limit coefficient of intensity into the style; the calculation of linear decay is shown in the following formula:

[0068] ;

[0069] When the denoising step index At that time, the first The denoising steps are divided into three stages, and the style injection intensity of each denoising step in the third stage is set. ;

[0070] Specifically, in this embodiment, during the denoising process of the diffusion model, the denoising step index is used. This indicates the current denoising stage. The total number of denoising steps, for example, can be set to 30 or 50 steps in this embodiment; denoising step index. The value range is 0 to -1, setting , , p1 and p2 are preset proportional parameters for dividing the denoising stages, and floor is the lower limit proportional coefficient of style injection intensity. In this embodiment, the entire denoising process is divided into three stages with different style injection strategies through the above piecewise function.

[0071] In the first stage, i.e., when the denoising step indexes... satisfy Style injection intensity (e.g., the first 30% of the denoising process) Set as base injection strength This means that in the early stages of denoising, the style injection intensity is maintained at a relatively high baseline level. The principle behind this is that the early stages of diffusion denoising are mainly responsible for determining the overall layout, composition, and macroscopic structure of the image. Maintaining a high style injection intensity at this time allows the global features of the style image, such as color tone and brushstroke texture, to quickly permeate into the generated image, establishing the artistic style direction of the image and preventing the final generated image from having a weak or impure style.

[0072] In the second stage, i.e., when the denoising step index... satisfy Style injection intensity at time (e.g., the middle 40% of the denoising process). From base injection strength It begins to decay linearly until it reaches the lower limit of style injection intensity. During this transition phase, as the image content gradually becomes clearer, the subject's outline and structural information begin to stabilize. Gradually reducing the style injection intensity at this point is to prevent excessive style textures from overwriting or disrupting the already formed details of the subject's structure, thus achieving a smooth transition between style and content.

[0073] In the third stage, that is, when the denoising step indexes... satisfy Time (e.g., the last 30% of the denoising process). Locked as the lower limit of style injection intensity (Right now (40% of the total). In the later stages of denoising, the generation process mainly focuses on filling high-frequency details and optimizing the local texture of the image. At this time, keeping the style injection intensity at a low level can effectively avoid the stacking of high-frequency style noise and prevent the blurring of the subject outline, distortion of the background structure, or artifacts caused by excessive style intervention, thereby maximizing the structural consistency and clarity of the content image.

[0074] By employing this segmented, step-by-step scheduling strategy, this embodiment overcomes the limitation of maintaining a constant style injection intensity in existing technologies. Based on the inherent laws governing image generation during denoising, it dynamically adjusts the weights of style and structure control, thereby contributing to both sufficient style transfer and accurate structure preservation, and resolving the conflict between style injection and structure preservation.

[0075] Step 5.2: In each noise reduction step Using the output features of the previous denoising step as input, style injection, content injection, and structure injection are performed sequentially to obtain the first... Step-by-step denoising output features :

[0076] For the The first noise reduction step injects intensity according to the set style. Output features of each denoising step Perform style injection to obtain the first intermediate state. As shown in the formula below:

[0077] ;

[0078] in, Injecting residuals into the style, For noise reduction steps The style injection intensity is determined based on the stage to which the denoising step belongs; the style injection residual It is obtained by cross-attention calculation from style embedding;

[0079] Based on content injection residual pair The first intermediate state of the noise reduction step Content injection is performed to obtain the second intermediate state. As shown in the formula below:

[0080] ;

[0081] in, For the first Step 1 Injecting residual content For the first The content embedding weight is injected into the content; if the currently processed content is the weight of the content embedded. The content injection residual satisfies If this happens, the content injection loop will terminate prematurely. Early stopping threshold; content injection residual It is obtained by cross-attention calculation from content embedding;

[0082] When the current processing of the first When the norm of the content injection residual is less than the preset early stopping threshold, the content injection process for embedding the remaining content in the current denoising step is terminated.

[0083] When the norm of any content injection residual is less than a preset early stopping threshold, the remaining content injection in the current denoising step is terminated. This early stopping mechanism is used to reduce invalid computation. When the norm of a content injection residual (characterizing its correction magnitude to the feature map) is less than the preset threshold, it indicates that the subsequent content embedding contributes negligibly to the current denoising step. Terminating the injection of the remaining embeddings at this time helps to reduce inference time while maintaining generation quality.

[0084] In this embodiment, the early stop threshold is set to... Furthermore, the norm is the L2 norm of the content injection residual. The early stopping mechanism in this embodiment aims to optimize computational resources and avoid invalid feature updates at specific stages of the denoising process.

[0085] Specifically, within each denoising step, each content embedding is processed sequentially using a polling method. When processing the first... When embedding content, calculate the content embedding for the first intermediate state. The correction amount, i.e., the content injection residual. To quantify content, residuals are injected. For the first intermediate state To measure the degree of influence, this embodiment uses the L2 norm (i.e., the Euclidean norm) as a metric. The formula for calculating the L2 norm is: The L2 norm reflects the magnitude of the residual vector in the feature space.

[0086] After calculating the L2 norm, Compared with the preset early stop threshold Compare. If This means that the current content embedding contributes very little to the feature map. From a microscopic perspective, this typically occurs in the later stages of the denoising process, when image features have converged and stabilized, making it difficult for subsequent content embedding to provide significant structural or texture correction. Continuing to perform attention calculations and injection operations for the remaining content embeddings not only fails to improve the generation quality but also consumes a large amount of computational resources.

[0087] Therefore, once the above early stopping condition is met, the injection process for embedding the remaining content in the current denoising step will be immediately terminated, skipping subsequent calculation steps and proceeding to the next stage of processing (such as structure-guided injection or the next denoising step). This mechanism reduces redundant floating-point operations and memory access overhead without sacrificing the quality of the generated image by identifying "invalid calculation points".

[0088] Under the above parameter settings, the average time of this embodiment (including early stop and related control strategies) is reduced to 25.7 seconds, which improves the real-time performance of multi-subject style transfer to a certain extent.

[0089] It should be understood that, although this embodiment preferably sets the early stopping threshold to However, in other application scenarios, this threshold can be adaptively adjusted based on model size, number of denoising steps, or different preferences for inference speed and generation quality. For example, if extremely high generation quality is required, the threshold can be appropriately lowered to retain more details; if strict requirements are placed on inference speed, the threshold can be appropriately increased to more aggressively trigger early stopping.

[0090] In this embodiment, content attention injection is performed sequentially using content embedding_emb_i as a condition in a polling manner to obtain the content injection residual. and according to the injection weight The weighted summation is applied to the current feature. The polling approach means that within each denoising step, all content embeddings are traversed sequentially, and the correction amount (i.e., content injection residual) of each content embedding to the current feature is calculated through a content attention mechanism. Weighted summation through injection weights ensures more stable fusion of multi-subject content and avoids the problem of a single low-quality embedding dominating the generated result.

[0091] Based on structural injection residuals, the first The second intermediate state of the denoising step Perform structure injection to obtain the first Output features of each denoising step As shown in the formula below:

[0092] ;

[0093] in, Injecting residuals into the structure, The structural injection strength. The structural injection residual. The structural condition map is obtained by inputting it into the structure preservation module and then injected into the second intermediate state to obtain the output features of the denoising step, thus maintaining the structural consistency between the generated image and the content image.

[0094] The structure preservation module in this embodiment includes a ControlNet network, specifically designed to learn the mapping relationship between structure condition maps and image features, generate structure injection residuals (also known as structure residuals), and use the structure injection strength as the basis for the structure preservation module. Injected into the second intermediate state This ensures that the generated image follows the outline and layout of the content image. This structural injection works in conjunction with style injection to ensure that the generated result possesses the target style while strictly preserving the original content structure.

[0095] Output features Dimensions are usually ,in For batch size, For the number of channels, and These represent the height and width of the feature map, respectively; in the attention module, this feature can be expanded into... The form is adapted to attention calculation. Indicates and The residual tensor of the same dimension is used to correct intermediate features; The style hint token is obtained by encoding the style image using an encoder. Indicates the first The augmented content embedding is obtained by encoding the augmented content image using an encoder.

[0096] This embodiment introduces a global adaptive adjustment mechanism based on edge density to address potential conflicts between style injection and structural guidance. Specifically, a conflict metric is obtained based on the edge density of the structural condition map, and the base injection intensity is adaptively set accordingly. With structural injection strength This reduces the base injection strength when the conflict metric increases. And improve the structural injection strength .

[0097] Specifically, in multi-subject style transfer tasks, when the content image contains complex textures or high-frequency details, the edge density of the structural condition map (such as the Canny edge map) is often high. This high-density edge information means that the image contains a large number of structural constraints. If a high style injection intensity is maintained at this time, the style texture can easily cover or destroy these fine structural details, resulting in structural distortion or style artifacts in the generated image.

[0098] To quantify this potential conflict, this embodiment first calculates the edge density of the structural condition graph. Edge Density It is calculated by statistically analyzing the proportion of non-zero pixels in the edge image, as shown in the following formula:

[0099] ;

[0100] in Indicates the total number of pixels in the image. This is an indicator function.

[0101] Based on the calculated edge density Further calculate the conflict metric To map edge density to a reasonable conflict metric range, this embodiment employs a pruning function, such as a conflict metric. This means that when the edge density is low ( ), Conflict measurement With edge density Linear growth; when edge density is high ( ), Conflict measurement It is limited to a maximum value of 1 to prevent over-adjustment.

[0102] In obtaining conflict measurement Then, dynamically adjust the basic injection intensity. (As the initial strength for segmented scheduling) and structural injection strength During the denoising process, the style injection intensity is calculated step by step based on the base style intensity. The adjustment strategy follows the principle of "suppressing style and strengthening structure when there is significant conflict." The specific adjustment formula can be expressed as:

[0103] ;

[0104] ;

[0105] in, and These are the preset basic style strength and basic structural strength, respectively.

[0106] Through the above adaptive mechanism, when dense edges of the structural condition graph are detected (i.e., conflict metric), When the injection intensity is relatively high, the base injection strength will be automatically reduced. (Its lower limit is 0.4) This reduces the interference of style textures on the structure and simultaneously increases the structural injection strength. (Increase by up to 0.3) to enhance ControlNet's constraint on structural details. Conversely, when edges are sparse, style intensity is appropriately restored and structural constraints are reduced to avoid generating overly rigid images. This dynamic balancing strategy effectively solves the problem of balancing style consistency and structural preservation with fixed parameters, significantly improving the generation quality in complex scenes.

[0107] In this embodiment, all residuals , and All are obtained by guiding network computation through corresponding attention mechanisms or structures, and their dimensions are... To maintain consistency and ensure direct tensor addition operations, this embodiment achieves coordinated optimization of style transfer and structure preservation through the closed-loop update of the three-stage injection process. This ensures that the generated image possesses both the artistic features of the target style and strictly adheres to the geometric structure of the content image. The structure guidance network is ControlNet, the diffusion model in this embodiment is StableDiffusionXL, and cross-attention injection is implemented through the IP-Adapter attention processor. Specifically, StableDiffusionXL (SDXL), as an advanced latent diffusion model, serves as the basic generation backbone in this embodiment, responsible for performing the denoising process from Gaussian noise to a sharp image in the latent space. ControlNet, as a neural network architecture, adds additional side connection layers and performs zero-initialized convolutional connections with the U-Net backbone of SDXL, thereby transforming edge information in the structure condition map into structural residuals and injecting them into each layer of SDXL's features, forcing the generated image to conform to the geometric layout of the content image. The IP-Adapter (ImagePromptAdapter) is a plug-and-play adapter module that uses a pre-trained image encoder to extract feature vectors from style images and injects these features into the cross-attention layer of SDXL through a cross-attention mechanism. This enables precise control over the style of the generated image, and outputs the generated image after completing a preset number of denoising steps. The above scheme maintains structural consistency by introducing a structure-guided network, implements style injection using style cue embedding and segmented stepwise scheduling, and solves the problems of multi-subject contamination and computational redundancy through weighted round-robin and early stopping mechanisms for multi-subject content embedding, achieving high-quality style transfer.

[0108] Example 2:

[0109] This embodiment verifies and illustrates the practical effect of the diffusion model-based multi-subject style transfer method by combining specific application scenarios and experimental data. In this embodiment, the content image used is snow.jpeg containing multiple subjects (such as people and scenery), and the style images used are style1.jpg, style2.jpg, and style3.png for testing. To verify the effectiveness of the diffusion model-based multi-subject style transfer method, a baseline scheme is set for comparison. The baseline scheme uses fixed style injection parameters (style_scale=1.0) and fixed structure guidance strength (controlnet_conditioning_scale=0.8), and does not enable segmented step-by-step scheduling, the reliability weighting mechanism of content embedding, or the early stopping mechanism.

[0110] Specifically, when testing the multi-agent style transfer method based on the diffusion model, the segmented step-by-step scheduling strategy, content embedding weighting mechanism, and early stopping mechanism described in the aforementioned embodiments were enabled. Experimental data show that, with a total number of denoising steps N=30, the adaptive global style intensity style_scale_used calculated by the multi-agent style transfer method based on the diffusion model is 0.9486033823529412, and the structure guidance intensity control_scale_used is 0.8256983088235295. For the content image snow.jpeg, after random augmentation and edge operator calculation, the obtained content embedding injection weights are [0.501853931289289, 0.498146068710711], indicating that the quality of the two augmented views is relatively close, and the content injection is evenly distributed accordingly. Regarding the segmented scheduling of style injection intensity, according to the settings of p1=0.3 and p2=0.7, the style injection intensity of the key denoising step is... They are respectively: When =0, it is 0.9486. =9 remains at 0.9486. At =21, it decays to 0.3794. =29, maintaining at 0.3794. This "strong at the beginning and weak at the end" scheduling strategy allows the generated image to establish a distinct stylistic tone in the early stages, while effectively avoiding the destruction of the subject's outline by excessive stylization in the later stages.

[0111] Regarding inference efficiency, since the multi-subject style transfer method based on the diffusion model introduces an early stopping mechanism in each denoising step, when the L2 norm of the content injection residual is less than a preset threshold... When the time is reached, the injection of the remaining content embedding is automatically terminated. Under the same hardware environment, the average inference time of the baseline scheme is 27.9 seconds, while the average inference time of the multi-subject style transfer method based on the diffusion model (including early stopping and related control strategies) is reduced to 25.7 seconds. This shows that the multi-subject style transfer method based on the diffusion model can improve the real-time performance of multi-subject style transfer to a certain extent while maintaining the generation quality. In terms of generation quality, by comparing the images output by the multi-subject style transfer method based on the diffusion model (image1_ours, image2_ours, image3_ours) with the images output by the baseline scheme (image1_baseline, image2_baseline, image3_baseline), differences can be observed in the test samples of this embodiment. Due to the constant and excessively high style injection intensity, the baseline scheme is more prone to high-frequency texture stacking in the comparative samples, and some subject outlines may be covered by style textures, and the background structure may also be distorted. The multi-subject style transfer method based on the diffusion model can effectively integrate the artistic texture of the style image while maintaining the geometric structure of the subject and the background layout in the content image. No obvious subject contamination or structural distortion issues were observed in the test samples of this embodiment. This verifies the effectiveness of the synergistic effect of segmented stepwise scheduling, reliability weighting, and structure-guided injection. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

[0112] Example 3:

[0113] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the multi-agent style transfer method based on the diffusion model.

[0114] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the multi-agent style transfer method based on the diffusion model as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0115] The processor is used to execute all or part of the steps in the diffusion-based multi-subject style transfer method described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in an electronic device, as well as application-related data.

[0116] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the diffusion-based multi-agent style transfer method described in the above embodiments.

[0117] Example 4:

[0118] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0119] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the diffusion-based multi-subject style transfer method described in the various embodiments of this application.

[0120] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) application stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the diffusion-based multi-agent style transfer method described above.

[0121] Example 5:

[0122] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the multi-agent style transfer method based on a diffusion model.

[0123] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0124] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0125] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A multi-agent style transfer method based on a diffusion model, characterized in that, Includes the following steps: Acquire content images and style images, generate a structure condition map based on the content image, and generate a structure embedding based on the structure condition map; The style image is encoded to obtain the style embedding; The content image is randomly augmented and encoded multiple times to obtain multiple augmented images and corresponding content embeddings; Calculate injection weights based on augmented images; Obtain style transfer cue words, encode the style transfer cue words, and generate text embeddings; The structure embedding, style embedding, content embedding, and injection weights are input into the diffusion model. Under the semantic conditions provided by the text embedding, the model outputs an image after several denoising steps. Specifically, for each denoising step, style injection is performed on the output features of the previous denoising step based on style embedding and the set style injection intensity to obtain the first intermediate state of each denoising step; content injection is performed on the first intermediate state based on injection weight and content embedding to obtain the second intermediate state of each denoising step; and structure injection is performed on the second intermediate state based on structure embedding to obtain the output features of each denoising step.

2. The multi-agent style transfer method based on a diffusion model according to claim 1, characterized in that, Content images are the original images that need to retain the main structure and outline; style images are reference images that provide texture, color, and artistic style. The structural condition map is used to represent the geometric structure information of the content image; The structural condition map is obtained by performing Canny edge detection on the content image, and the structural embedding is obtained by encoding the structural condition map.

3. The multi-agent style transfer method based on a diffusion model according to claim 1, characterized in that, The specific method for encoding style images to obtain style embeddings is as follows: The style image is encoded using an image encoder, which maps the style image into a high-dimensional feature vector; the high-dimensional feature vector is then converted into a style embedding through a projection layer.

4. The multi-agent style transfer method based on a diffusion model according to claim 1, characterized in that, The specific method for performing multiple random augmentations and encodings on the content image is as follows: The content image is randomly augmented n times, and an augmented image is obtained after each random augmentation. After n random augmentations, n augmented images are obtained. The random augmentation includes horizontal flipping and grayscale perturbation operations. Encode each of the n augmented images to obtain n content embeddings.

5. The multi-agent style transfer method based on a diffusion model according to claim 4, characterized in that, The specific method for calculating injection weights based on augmented images is as follows: Edge operator processing is performed on the augmented image to generate grayscale edge images; the pixel mean of the grayscale edge images is calculated to obtain the reliability score of each augmented image; The reliability score of each augmented image is normalized to obtain the injection weights.

6. The multi-agent style transfer method based on a diffusion model according to claim 1, characterized in that, The text embedding participates in the diffusion denoising process through cross-attention to provide semantic condition constraints.

7. The multi-agent style transfer method based on a diffusion model according to claim 1, characterized in that, The style injection intensity is determined based on the denoising stage to which the denoising step belongs, specifically by the following method: Set the base injection intensity and the ratio parameters for dividing the denoising stages, divide the denoising steps into different denoising stages, and adjust the style injection intensity of each denoising step according to the denoising stage and the base injection intensity.

8. The multi-agent style transfer method based on a diffusion model according to claim 7, characterized in that, The specific method for performing style injection on the output features of the previous denoising step based on style embedding and the set style injection strength is as follows: Cross-attention calculation is performed on the style embedding to obtain the style injection residual; style injection is performed on the output features of the previous denoising step according to the style injection residual and the set style injection intensity to obtain the first intermediate state; The specific method for injecting content into the first intermediate state based on injection weights and content embedding is as follows: Cross-attention calculation is performed on n content embeddings to obtain n content injection residuals; content injection is performed on the first intermediate state according to the content injection residuals and injection weights to obtain the second intermediate state; When the norm of the residual of the i-th content injection is less than the preset early stopping threshold, the content injection of the remaining content embedded in the denoising step is terminated, where, ; The specific method for structural injection into the second intermediate state based on structural embedding is as follows: The structure is embedded into the input structure preservation module to generate the structure injection residual; the structure injection is performed on the second intermediate state according to the structure injection residual and the set structure injection intensity to obtain the output features of the denoising step. The structure preservation module uses a ControlNet network.

9. A multi-agent style transfer method based on a diffusion model according to claim 8, characterized in that, The basic injection intensity and the structural injection intensity are adjusted through a global adaptive adjustment mechanism based on edge density; Global adaptive adjustment mechanisms based on edge density include: Calculate the edge density of the structural condition graph and the conflict metric; Adjust the foundation injection intensity and structural injection intensity based on the conflict metric.

Images