Cross-attention modulated digital printing pattern color semantic consistency generation method

Abstract

This invention discloses a method for generating color semantic consistency of digital printing patterns using cross-attention modulation, specifically including the following steps: Step 1, constructing a digital printing pattern dataset; Step 2, training a stable diffusion model as the backbone network on the dataset from Step 1 to obtain model training weights; Step 3, using the weights trained in Step 2 for sampling inference, including two noise predictions in each denoising step of the inference process, extracting color cross-attention weights and constructing a cross-attention color modulation module in the first noise prediction; Step 4, calculating the color loss for the color cross-attention weights extracted in Step 3; Step 5, updating the noisy image through progressive color loss minimization and gradient descent. z t for z t The second step involves noise prediction and denoising; step 6 involves generating a clear printed pattern through multi-step iterative denoising. This invention solves the problem of poor semantic consistency of pattern color in existing pattern generation methods.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology and relates to a method for generating color semantic consistency of digital printing patterns using cross-attention modulation. Background Technology

[0002] Digital printing, as an important development direction in the printing and dyeing industry, directly relates to the industry's digital upgrade and the achievement of green manufacturing goals through its intelligent pattern generation level. Compared with traditional printing processes, digital printing technology has significant advantages in pattern accuracy, color performance, and environmental friendliness, and has promising application prospects. However, at present, the design process of digital printing patterns still mainly relies on manual methods. Designers need to repeatedly draw and adjust using professional drawing software, resulting in long design cycles, high labor costs, and design quality that is greatly affected by personal experience, making it difficult to adapt to rapidly changing and personalized market demands. In recent years, with the development of Artificial Intelligence Generative Content (AIGC) technology, deep learning methods based on stable diffusion models have been introduced into the field of digital printing pattern generation, achieving automatic generation and preliminary control of patterns through text-driven methods. However, due to the limitations of text encoding models in fine-grained semantic understanding and the modal differences between text description and image expression, existing methods are prone to color semantic deviations during pattern generation. Meanwhile, due to the uneven quality and insufficient annotation precision of existing printed pattern datasets, the generated results often suffer from problems such as blurred details and color distortion, resulting in inconsistencies between the colors of the generated printed patterns and the color keywords in the text description, which fails to accurately meet the design needs of customers. Summary of the Invention

[0003] The purpose of this invention is to provide a method for generating digital printing pattern color semantic consistency using cross-attention modulation, which solves the problem of poor pattern color semantic consistency in existing pattern generation methods.

[0004] The technical solution adopted in this invention is a method for generating color semantic consistency of digital printing patterns using cross-attention modulation, which specifically includes the following process: Step 1: Construct a digital printing pattern dataset; Step 2: Using the stable diffusion model as the backbone network, train the model on the dataset from Step 1 to obtain the model training weights. Step 3: Use the weights trained in Step 2 to perform sampling inference. Each denoising step in the inference process includes two noise predictions. In the first noise prediction, extract the color cross-attention weights and construct the cross-attention color modulation module. Step 4: Calculate the color loss based on the color cross-attention weights extracted in Step 3; Step 5: Update the noisy image by progressively minimizing color loss and gradient descent.z t for z t ', and perform noise prediction and denoising a second time; Step 6: Generate a clear printed pattern through multi-step iterative noise reduction.

[0005] The invention is further characterized by: The specific process of step 2 is as follows: Step 2.1: The Stable Diffusion network is used as the backbone network, and U-Net is used as the noise prediction network. Step 2.2: Fine-tune the network model using Low-Rank Adaptive (LoRA). The LoRA method is used to freeze the original pre-trained weights of SD during fine-tuning training, and training is performed by inserting a low-rank matrix into the network model.

[0006] The specific process of step 2.2 is as follows: The noise loss function is trained using text descriptions as control conditions. L Represented as: (1) In the formula, z t Represents an image with potential spatial noise. t For time steps, c For text prompts, This represents the noise in the prediction. This indicates the actual noise added.

[0007] The specific process of step 3 is as follows: Step 3.1: Segment the input text description into a sequence of words. w The text is then fed into the CLIP text encoder for encoding to obtain text embeddings. These text embeddings are then cross-attentiond with the noisy image. In the cross-attention layer of the noise prediction network, the noisy image is mapped to the query matrix. Q Embed text onto key K Time step t Time and n Cross-attention feature map corresponding to each text word A Represented as: (2) in, d This indicates that the softmax value has been normalized. A Depend on n Cross-attention feature maps { }composition, ∈[0,1] h×w The first in the text description iCross-attention weights corresponding to each word element; Step 3.2: Extract cross-attention weights of two sizes, 16×16 and 8×8, from the cross-attention weights. A cross Then, from A cross The text description retrieved keywords related to the foreground object color, foreground object, background color, and background. background "Corresponding cross-attention weights; Step 3.3: Calculate a weighted average of the extracted cross-attention weights, and then calculate the average cross-attention weight of the object color by weighting the foreground color cross-attention and the foreground object cross-attention. The average cross-attention weight of background color is obtained by weighting the cross-attention of background color and the corresponding cross-attention of the background keyword "background". , i This indicates the index of the foreground object in text segmentation. j Indicates the word segmentation of text. background The index where the word segmentation is located.

[0008] Step 4 involves smoothing the surface using a Gaussian function. To ensure a natural transition between foreground objects, since the foreground objects in the pattern are distributed in a point-like manner, the maximum value among the weights is taken, as shown in formula (3), to calculate the average cross-attention loss of object colors. L o : (3) Where max represents taking the maximum value, and Gauss represents Gaussian smoothing; Since the background area of ​​the pattern is distributed in a planar shape, the average cross-attention weight of the background color is calculated. The mean value is shown in Formula (4), which is used to calculate the average cross-attention loss of the background color. L B : (4) Here, mean represents the average value of the cross-attention weights for the background color.

[0009] The specific process of step 5 is as follows: The summation formulas (3) and (4) yield the total loss of object color and background color. L sum : L sum = L o + LB (5) By minimizing L sum The cross-attention weights corresponding to the color keywords are adjusted and gradually increased, and the loss function is calculated. L sum For potential noise Z t The gradient, and using Z t Subtracting the gradient yields potential noise in gradient updates. Z t ’ The gradient update is shown in equation (6): (6) in, This represents the scale factor for gradient updates in the cross-attention loss. The gradient calculation for the loss is indicated by ←, which represents the gradient update operation. The gradient update operation in formula (6) must be executed continuously. m Second-rate, m With the current iteration round number Related, when When ≤0.3*T, m =3; 0.3*T< When ≤0.7*T, m =2; 0.7*T< hour, m =1.

[0010] The specific process of step 6 is as follows: the noisy image is updated by gradient update to obtain the updated image. z t ', z t The image re-enters the noise prediction network, undergoes noise prediction and denoising, and then yields the noise image for the next time step. z t-1 After multiple noise reduction steps, a clearer, higher-quality printed pattern is finally obtained. z 0.

[0011] In step 6, the noise image for the next time step is obtained using the following formula (7). z t-1 : (7) in, It is a time step t The corresponding noise retention factor, It is predicted noise. Indicates the first t The standard deviation of the noise added during the reverse sampling step. It is a standard Gaussian random noise.

[0012] The beneficial effects of this invention are that the proposed cross-attention modulation method for generating digital printed patterns with semantic consistency of color improves upon text-to-image generation methods based on stable diffusion models. It utilizes LoRA for efficient model fine-tuning training, constructing a dedicated generation model for text-to-textile digital printed patterns. Simultaneously, addressing the issue of inaccurate pattern color generation, a cross-attention color modulation module is constructed during model inference, introducing cross-attention color modulation loss for foreground and background colors. Attention weights are gradually increased through loss minimization and gradient updates, thereby improving the accuracy of pattern color generation. This invention effectively enhances the semantic consistency of pattern color, showing a significant improvement compared to existing advanced pattern generation methods. Attached Figure Description

[0013] Figure 1 This is a network framework diagram of the digital printing pattern color semantic consistency generation method based on cross-attention modulation of the present invention; Figure 2 This is a visualization of the generation result of the digital printing pattern color semantic consistency generation method of cross-attention modulation of the present invention; Figure 3 This is a visual comparison of the digital printing pattern color semantic consistency generation method of the present invention with the advanced Attend-and-Excite method; Figure 4 This is a visual comparison of the digital printing pattern color semantic consistency generation method of cross-attention modulation of the present invention without using color modulation (a) and using only foreground color modulation (b); Figure 5 This invention provides a visual comparison of the digital printing pattern color semantic consistency generation method using cross-attention modulation with only foreground color modulation (a) versus using both foreground and background color modulation (b). Detailed Implementation

[0014] The following detailed description is provided in conjunction with specific implementation methods.

[0015] Example 1 This invention presents a method for generating digital printed patterns with color semantic consistency based on cross-attention modulation. It utilizes the robust and stable diffusion model SD as the backbone network, and fine-tunes a dedicated text-to-textile digital printed pattern generation model using LoRA. To address the issue of color semantic consistency in generated printed patterns, a cross-attention color modulation module is constructed during the model inference stage. By minimizing the loss, the model's attention to foreground and background colors is enhanced. This invention effectively improves the color semantic consistency between the generated printed pattern and the input text description.

[0016] The design concept of this invention is as follows: First, a dedicated textile printing pattern dataset is established and detailed text annotations are performed. This dataset is used for fine-tuning the model, during which the model learns to generate printing patterns that basically conform to semantics based on given text. Then, a cross-attention color modulation module is constructed during the inference stage. This module collects cross-attention features from the U-Net noise prediction network and extracts cross-attention feature maps corresponding to the foreground and background colors in the text description. Through loss calculation and gradient update, the cross-attention of colors is adjusted and enhanced, thereby improving the accuracy of color generation.

[0017] Example 2 The present invention provides a method for generating color semantic consistency in digital printing patterns using cross-attention modulation, which specifically includes the following steps: Step 1: Build a new, high-quality digital printing pattern dataset with detailed text annotations as a benchmark dataset for training the textile digital printing pattern generation model.

[0018] Step 2: Using the stable diffusion model (SD) as the backbone network, fine-tune the training on the self-built dataset to obtain the model training weights.

[0019] Step 3: Perform sampling inference using the weights trained in Step 2. Each denoising step in the inference process includes two noise predictions. In the first noise prediction, a cross-attention color modulation module is constructed, such as... Figure 1 As shown, cross-attention weights are extracted from the cross-attention layer of the U-Net noise prediction network in SD. A cross The predicted noise is not processed.

[0020] Step 4: Calculate the loss for the cross-attention weights extracted in Step 3.

[0021] Step 5: Update the noisy image using stepwise loss minimization and gradient descent. z t for z t '.

[0022] Step 6, z t 'Re-feeding into the noise prediction network to predict the noise' And denoise the image to obtain the noisy image at the next time step. z t-1 。 Finally, a clear printed pattern is generated through multi-step iterative noise reduction.

[0023] Example 3 Step 1 involves the following process: A dedicated dataset for digital printing pattern generation was constructed, consisting of 3090 text description-printed pattern sample pairs. The image data primarily comes from various digital printing pattern resource platforms, encompassing 12 styles including watercolor, cartoon, oil painting, abstract, and Chinese style. Pattern layouts include diagonal arrangement, staggered arrangement, and square array arrangement. The text descriptions for the TileVL dataset are in English and formatted as follows: "seamless (omit if the pattern is not a four-way continuous pattern) + style + foreground color_1 + foreground object_1 + ... + foreground color_n + foreground object_n + layout + pattern background" to help the model learn the style, color, foreground object, layout arrangement, and background features of the pattern.

[0024] Example 4 Step 2 involves designing a backbone network using Stable Diffusion as the main network and U-Net as the noise prediction network. Then, the model is fine-tuned using Low-Rank Adaptation (LoRA). LoRA freezes the original pre-trained weights of the SD (Digital Printing Loss) during fine-tuning and significantly reduces the number of trainable parameters by inserting a low-rank matrix into the model, thus improving training speed. The input condition for training is a text description, and the output is the corresponding digital print pattern. The purpose of fine-tuning is to allow the model to learn more features about the style, color, foreground object category, and layout of the digital print pattern based on the general pre-trained SD weights, constructing a dedicated generative model for text-to-digital print pattern generation. The noise loss function trained with the text description as the control condition is used. L The number is represented as: (1) In the formula z t Represents an image with potential spatial noise. t For time steps, c For text prompts, This represents the noise in the prediction. This represents the actual noise added. The model fine-tuning training epochs are set to 55 epochs. The model parameters are continuously updated and iterated until the set number of training epochs is reached, resulting in the optimal model.

[0025] Example 5 The specific process of step 3 is as follows: First, the input text description statement is segmented into a word sequence. w Then w The text is then fed into the CLIP text encoder for text encoding to obtain the embedded text.

[0026] The obtained text embedding is then cross-attentiond with the noisy image. Specifically, in the cross-attention layer of the noise prediction network, the noisy image is mapped to the query matrix. Q Embed text onto key K Time step t Time and n Cross-attention feature map corresponding to each text word A Represented as: (2) in, d This indicates that the softmax value has been normalized. A Depend on n Cross-attention feature maps { }composition, ∈[0,1] h×w The first in the text description i each word element w i The corresponding cross-attention weights.

[0027] To extract cross-attention weights corresponding to text keywords and reduce computational cost while extracting deep features, this invention extracts 16×16 and 8×8 cross-attention weights from the calculated cross-attention weights. A cross Then, from A cross The text description retrieved keywords related to "foreground color", "foreground object", "background color", and "background". background "The corresponding cross-attention weights, such as Figure 1 shown A 4 , A 5 , A 9 , A 10 .

[0028] The extracted cross-attention weights are weighted and averaged. A 4 and A 5 The weighted average yields the foreground average cross-attention weight. ,Will A 9 and A 10 Weighted average to obtain the background color average cross attention weight . i This indicates the index of the foreground object in text segmentation. j Indicates the word segmentation of text. backgroundThe index where the word segmentation is located.

[0029] Example 6 Step 4 involves smoothing the surface using a Gaussian function. To ensure a natural transition between foreground objects, since the foreground objects in the pattern are distributed in a point-like manner, the maximum value among the weights is taken, as shown in formula (3), to calculate the average cross-attention loss of the foreground color. L o : (3) Where max represents taking the maximum value, and Gauss represents Gaussian smoothing.

[0030] Meanwhile, since the background area of ​​the pattern is distributed in a planar shape, the average cross-attention weight of the background color is calculated. The mean value is shown in Formula (4), which is used to calculate the average cross-attention loss of the background color. L B : (4) Here, mean represents the average value of the cross-attention weights for the background color.

[0031] Example 7 The specific process of step 5 is as follows: summation formulas (3) and (4) yields the total loss of object color and background color. L sum : L sum = L o + L B (5) By minimizing L sum Adjust the cross-attention weights corresponding to the color keywords, gradually increasing them. Calculate the loss function. L sum For potential noise Z t The gradient, and using Z t Subtracting the gradient yields potential noise in gradient updates. Z t ’ The gradient update is shown in equation (6): (6) in, This represents the scale factor for the gradient update of the cross-attention loss, and is greater than or equal to 1. ← represents the gradient calculation of the loss, and ← represents the gradient update operation. The gradient update operation in formula (6) must be executed continuously. m Second-rate, m With the current iteration round number Related, when When ≤0.3*T (T represents the total number of rounds), m =3; 0.3*T< When ≤0.7*T, m =2; 0.7*T< hour, m =1. Since the initial stage of denoising is a critical period for pattern color formation, the gradient update force is greater in the early stages. m The value is greater.

[0032] Example 8 The specific process of step 6 is as follows: the noisy image is updated by gradient update to obtain the updated image. z t '.like Figure 1 As shown, z t 'After re-entering the noise prediction network and undergoing noise prediction and denoising, a clearer noisy image is obtained at the next time step.' z t-1 After multiple noise reduction steps, a clearer, higher-quality printed pattern is finally obtained. z 0. In step 6, the noise image for the next time step is obtained using the following formula (7). z t-1 : (7) in, It is a time step t The corresponding noise retention factor is used to control the noise retention factor in the first... t The amount of effective information retained and the amount of noise added during the step. It is predicted noise. Indicates the first t The standard deviation of noise added during reverse sampling. It is a standard Gaussian random noise.

[0033] Example 9 The generated result is as follows Figure 2 As shown, the first line contains the input text description, and the second line contains the corresponding generated pattern. It can be seen that the generated digital printed pattern achieves good visual effects in terms of clarity, artistic design, and texture detail. The foreground and background colors of the pattern are consistent with the color cues in the text description, indicating that the digital printed pattern generated by this invention has high color accuracy. Furthermore, Figure 2 Chinese Figure (a) and Figure 2The texture of the middle image (b) is clear, the layout is balanced, and the style is consistent with the text description, indicating that the overall quality of the pattern generated by the invention is significantly high.

[0034] Figure 3 A visual comparison of the results generated by this invention with the advanced Attend-and-Excite method is presented. Figure 3 Figure (a) shows the result generated by the Attend-and-Excite method, with a blue striped background, which does not match the white background in the text description. The result generated by this invention... Figure 3 The background color and pattern color of Figure (b) are consistent with the colors of the text description, which verifies the effectiveness of the present invention.

[0035] Figure 4 Demonstrates the use of color modulation Figure 4 The middle image (a) and the image using only foreground color modulation. Figure 4 A visual comparison of Figure (b) shows that... Figure 4 In the middle image (a), the flowers in the pattern are not clear and the details are blurry. In contrast, after adding foreground color modulation, the yellow rose flowers are clearer, more detailed, and the colors match the text description. This verifies the effectiveness of foreground color cross-attention modulation.

[0036] Figure 5 Demonstrates the use of only foreground color modulation ( Figure 5 The middle image (a) uses two modulation methods: foreground object color and background color. Figure 5 Visual comparison of the middle image (b). Clearly, the background color in (a) is incorrectly blue because the background color cross-attention feature is not modulated. Figure 5 Based on the image in (a), after adding background color modulation, Figure 5 In Figure (b), the background color of the pattern was corrected to a semantically appropriate white. This verifies the effectiveness of background color cross-attention modulation.

[0037] In summary, this method can generate high-quality digital printing patterns that conform to semantics and color based on a given text description, while also achieving significant results in generating overall pattern quality.

[0038] Table 1 below compares the evaluation metrics of the existing models: Table 1

[0039] Note: Bold and underlined text indicate the best and second-best results, respectively.

[0040] 1. Comparison of similarity scores between circular texts (T2T-Sim): T2T-Sim refers to using the visual language model BLIP to map the generated printed pattern to a text description, and then calculating the similarity between this text description and the original input text description to evaluate the semantic consistency of the generated pattern. As shown in Table 1, the T2T-Sim calculated by this invention achieves the best results compared to other methods, improving upon the second-best method, StructuredDiffusion, by 0.96%, indicating that this invention can generate printed patterns that are more consistent with the semantics of the text.

[0041] 2. Color Text Accuracy Comparison (C2T-Acc): C2T-Acc represents the ratio of the number of consistent color words in the test set to the total number of color keywords in the text. Table 1 shows that the C2T-Acc index of this invention is the best compared to other methods, and it improves by 4.35% compared to the second-best Attend-and-Excite method. This indicates that the proposed cross-attention color modulation module is highly effective and verifies that the colors of the printed patterns generated by this invention are more consistent with the semantics of text color.

Claims

1. A method for generating color semantic consistency in digital printing patterns using cross-attention modulation, characterized in that: Specifically, the process includes the following steps: Step 1: Construct a digital printing pattern dataset; Step 2: Using the stable diffusion model as the backbone network, train the model on the dataset from Step 1 to obtain the model training weights. Step 3: Use the weights trained in Step 2 for sampling inference. Each denoising step in the inference process includes two noise predictions. In the first noise prediction, a cross-attention color modulation module is constructed. The specific process of Step 3 is as follows: Step 3.1: Segment the input text description into a sequence of words. w Then w The text is fed into the CLIP text encoder for text encoding to obtain text embeddings. The obtained text embeddings are then cross-attentiond with the noisy image. In the cross-attention layer of the noise prediction network, the noisy image is mapped to the query matrix. Q Embed text onto key K Time step t Time and n Cross-attention feature map corresponding to each text word A Represented as: (2) in, d This indicates that the softmax value has been normalized. A Depend on n Cross-attention feature maps { }composition, ∈[0,1] h×w The first in the text description i each word element w i The corresponding cross-attention weights; Step 3.2: Extract the cross-attention weights for each layer, and extract cross-attention weights of two sizes: 16×16 and 8×8. A cross , and then, from A cross The text description retrieved keywords related to "foreground color of the pattern", "foreground object", "background color", and "background". background "Corresponding cross-attention weights; Step 3.3: Calculate a weighted average of the extracted cross-attention weights, and obtain the average cross-attention weight of the object color by weighting the foreground object color cross-attention and the foreground object cross-attention. The average cross-attention weight of background color is obtained by weighting the cross-attention of background color and the corresponding cross-attention of the background keyword "background". , i This indicates the index of the foreground object in text segmentation. j Indicates the word segmentation in text. background The index where the word segmentation is located; Step 4: Calculate the loss for the cross-attention weights extracted in Step 3; the specific process of Step 4 is as follows: Smoothing with Gaussian function To ensure a natural transition between foreground objects, since the foreground objects in the pattern are distributed in a point-like manner, the maximum value among the weights is taken, as shown in formula (3), to calculate the average cross-attention loss of object colors. L o : （3） Where max represents taking the maximum value, and Gauss represents Gaussian smoothing; Since the background area of ​​the pattern is distributed in a planar shape, the average cross-attention weight of the background color is calculated. The mean value is shown in Formula (4), which is used to calculate the average cross-attention loss of the background color. L B : （4） Where, mean represents the mean of the average cross-attention weights for the background color; Step 5: Update the noisy image using stepwise loss minimization and gradient descent. z t for z t '; Step 6: Generate a clear printed pattern through multi-step iterative noise reduction.

2. The method for generating color semantic consistency of digital printing patterns using cross-attention modulation according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 2.1: The Stable Diffusio (SD) network is used as the backbone network, and U-Net is used as the noise prediction network. Step 2.2: Fine-tune the network model using Low-Rank Adaptive (LoRA). The LoRA method is used to freeze the original pre-trained weights of SD during fine-tuning training, and training is performed by inserting a low-rank matrix into the network model.

3. The method for generating color semantic consistency of digital printing patterns using cross-attention modulation according to claim 2, characterized in that: The specific process of step 2.2 is as follows: a noise loss function trained using text description as a control condition. L Represented as: （1） In the formula, z t Represents an image with potential spatial noise. t For time steps, c For text prompts, This represents the noise in the prediction. This indicates the actual noise added.

4. The method for generating color semantic consistency of digital printing patterns using cross-attention modulation according to claim 3, characterized in that: The specific process of step 5 is as follows: The total loss of object color and background color is obtained from formulas (3) and (4). L sum : L sum = L o + L B （5） By minimizing L sum The cross-attention weights corresponding to the color keywords are adjusted and gradually increased, and the loss function is calculated. L sum For potential noise Z t The gradient, and using Z t Subtracting the gradient yields potential noise in gradient updates. Z t ’ Gradient updates are shown in equation (6): (6) in, This represents the scale factor for gradient updates in the cross-attention loss. The gradient calculation for the loss is indicated by ←, which represents the gradient update operation. The gradient update operation in formula (6) must be executed continuously. m Second-rate, m With the current iteration round number Related, when When ≤0.3*T, T represents the total number of rounds. m =3; 0.3*T< When ≤0.7*T, m =2; 0.7*T< hour, m =1; Since the initial stage of denoising is a critical period for pattern color formation, the gradient update intensity is greater in the early stage. m The value is greater.

5. The method for generating color semantic consistency of digital printing patterns using cross-attention modulation according to claim 4, characterized in that: The specific process of step 6 is as follows: the noisy image is updated by gradient update to obtain the updated image. z t ', z t The image re-enters the noise prediction network, undergoes noise prediction and denoising, and then yields the noise image for the next time step. z t-1 After multiple noise reduction steps, a clearer, higher-quality printed pattern is finally obtained. z 0.

6. The method for generating color semantic consistency of digital printing patterns using cross-attention modulation according to claim 5, characterized in that: In step 6, the noise image for the next time step is obtained using the following formula (7). z t-1 : （7） in, It is a time step t The corresponding noise retention factor, It is predicted noise. Indicates the first t The standard deviation of the noise added during the reverse sampling step. It is a standard Gaussian random noise.

Images