Image style migration method based on cross-modal attention and adaptive regulation

Abstract

The invention belongs to the field of computer vision, and particularly relates to an image style migration method based on cross-modal attention and adaptive regulation. According to the migration method, a cross-modal feature alignment mechanism based on a CLIP model is introduced, text semantic guidance and image region features are effectively combined, and fine-grained style control is achieved; meanwhile, in combination with a self-adaptive style adjustment strategy of region perception, the migration intensity is dynamically adjusted according to different region characteristics of the image, so that the detail fidelity, the visual consistency and the semantic expression capability are remarkably improved. According to the method, the multi-stage progressive optimization process is adopted, the problems of style overfitting and structure distortion which are prone to occurring in single-stage migration are effectively relieved, the image generation quality is improved, meanwhile, the calculation overhead is reduced, the good real-time application potential is achieved, and a new solution thought is provided for high-quality style migration in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to an image style transfer method based on cross-modal attention and adaptive regulation. Background Art

[0002] In recent years, image style transfer technology has demonstrated broad application prospects in fields such as image editing, video processing, and artistic creation. Since Gatys et al. proposed a style transfer method based on a combination of deep neural networks (DNNs) and VGG networks, they have effectively separated image content and style features by optimizing a combination of content loss and style loss, resulting in a qualitative leap in the visual quality of the transferred images. Since then, generative models such as generative adversarial networks (GANs) have also been gradually introduced into style transfer tasks, further improving the realism and detail expression of generated images and broadening the application scope of style transfer.

[0003] like Figure 2 As shown in Figure 2, traditional style transfer methods typically rely on convolutional neural networks (CNNs) to extract and fuse multi-level image features. By capturing multi-scale features from low-level textures to high-level semantics, they achieve a harmonious unity between overall style and local details. While existing methods have continuously improved their generation performance, practical applications still face some significant challenges.

[0004] Most existing methods are still centered on a single modality (i.e., image-to-image), and lack the effective use of multimodal information, especially text descriptions. This limitation makes the generated images obviously insufficient in terms of flexibility and consistency in semantic expression. Although the emergence of cross-modal pre-training models such as CLIP has made it possible to jointly model images and texts, when directly applied to style transfer tasks, there are still problems with coarse granularity and difficulty in achieving detailed regional control. The current mainstream style transfer methods generally adopt a unified global style control strategy, which cannot dynamically adjust the style transfer strength of each region (such as the subject, background, or details) according to the differentiated needs of the internal structure of the image. The AdaIN method proposed by Huang and Belongie [3] , by adjusting the mean and variance of the features, fast arbitrary style transfer is achieved, which improves real-time performance, but the ability to perceive and regulate local feature differences is limited; the WCT method of Li et al. uses whitening and coloring transformations to achieve style feature alignment in the feature space. Although it enhances the retention of content structure, it brings a large computational overhead, which limits its application in high-resolution image processing scenarios.

[0005] As the Transformer architecture has become widely used in image processing tasks due to its superior global modeling capabilities, methods such as StyTr2 have introduced a multi-level attention mechanism to achieve a fine-grained fusion of content and style features, further improving the performance of generated images in terms of detail expression and natural style transitions. However, these methods still have significant shortcomings in terms of flexible control of style intensity in different regions and dynamic adaptive adjustment.

[0006] In summary, the problems with existing technologies are as follows: (1) Existing methods cannot effectively balance generation quality and computational efficiency when processing high-resolution images, resulting in slow inference speed and inability to meet the needs of real-time applications. (2) Existing methods often face the problem of image detail loss or inconsistent style expression when processing complex texture expressions, and are unable to fully capture the details and artistic style of the image. In addition, as application requirements develop towards higher resolution and more complex texture expressions, existing methods face greater challenges in inference speed and computational cost, making it difficult to balance generation quality and real-time performance. During the training process, due to the complexity of feature integration, gradient vanishing or training instability are prone to occur, which further affects the stability and consistency of the style transfer system. Summary of the Invention

[0007] The purpose of the present invention is to address the problems of insufficient semantic guidance, inflexible regional style adjustment, limited detail fidelity and low generation efficiency in existing image style transfer methods. A multi-stage image style transfer method that integrates a cross-modal attention mechanism and an adaptive style adjustment strategy is proposed. By combining cross-modal feature alignment of images and texts, regional fine-grained style transfer is guided. At the same time, the transfer strength is dynamically adjusted according to the local structure of the image, thereby improving the detail expression, style naturalness and semantic consistency of the generated image. Through a staged optimization process, efficient and stable style transfer is achieved, taking into account both generation quality and computational efficiency, thereby effectively overcoming the above-mentioned shortcomings in the prior art.

[0008] The technical solution adopted by the method of the present invention is as follows: an image style transfer method based on cross-modal attention and adaptive regulation, comprising the following steps:

[0009] S1: Scale the content image to a uniform size (e.g., 512×512 pixels) and crop it to 256×256 pixels to ensure consistency of the input data. The style text description is preprocessed and standardized to achieve a uniform data format.

[0010] S2: A pre-trained convolutional neural network (VGG-19) is used to perform multi-level feature extraction on the content image to capture low-level texture and high-level semantic information. At the same time, the CLIP model is used to encode the style text and extract text features containing style semantics, providing semantic guidance for subsequent cross-modal fusion.

[0011] S3: Using the extracted content features as queries and the style text features as keys and values, we perform feature fusion via a cross-modal attention mechanism. This step accurately guides style transfer across different regions of the image based on text semantics, enabling text-driven fine-grained style expression.

[0012] S4: Based on the image's structural characteristics, masks are extracted for the subject, background, and details within the image, and dynamic style transfer strengths are assigned to different regions. This adaptive regional control effectively avoids excessive background stylization or loss of subject details, improving the overall naturalness and detail fidelity of the image.

[0013] S5: Based on the style-guiding features and content features, the feature information is further integrated and the local style strength is adjusted according to the guidance of the adaptive control module to ensure that the style and content of each region are balanced, generating an intermediate feature map with high consistency and rich details.

[0014] S6: The fused features are input into the decoder, and through convolution and upsampling operations, they are restored layer by layer to the final stylized image, ensuring that the output image achieves an ideal balance between style expression and content structure preservation.

[0015] S7: We use content loss and style loss to jointly optimize network parameters, dynamically adjusting the weights of the two losses to balance style expression and content preservation. We use the Adam optimizer during training to accelerate convergence and improve the detail and naturalness of the generated images.

[0016] Preferably, in step S1, in order to achieve the standardization of the content image and the style text description, the following steps are specifically included:

[0017] S101: Content image I c Scaling to 512×512 pixels ensures a uniform input size and improves the accuracy of subsequent feature extraction.

[0018] S102: Perform center cropping on the scaled image to obtain a 256×256 pixel image I' c , retaining the main area of the image to the greatest extent.

[0019] S103: Normalize the image pixel values to the interval [0,1] to unify the data distribution and accelerate network convergence; text description T s It also undergoes word segmentation and standardized encoding to facilitate subsequent feature extraction.

[0020] The image normalization formula is:

[0021] I′ c =Crop 256×256(Reize 512×512 (I c ))#(1)

[0022] Among them, I c is the input content image.

[0023] Preferably, in step S2, in order to fully extract the visual structure information of the content image and the style semantic information of the text description, a multimodal feature extraction strategy is adopted. Specifically, the following steps are included:

[0024] S201: Use pre-trained convolutional neural network (VGG-19) to train content image I' c Perform feature extraction layer by layer to obtain feature maps of each layer

[0025]

[0026] Among them, l is the convolution layer number, and different levels of texture, structure and semantic features are extracted through different convolution layers. For image encoder.

[0027] S202: Use CLIP model to style text T s Encode and extract text feature vectors:

[0028] F s =E text (T s )#(3)

[0029] Among them, T s For style text, E text A text encoder.

[0030] S203: Calculate the mean and standard deviation of content features and style features respectively, and unify the numerical scale:

[0031]

[0032] Represents the content image features of layer l extracted from the image encoder. N represents the total number of elements in the feature map of this layer (i.e., the dimension of the feature map). Representing content image features The i-th element in (i.e., the eigenvalue at the i-th position in the feature map). Representing content image features The mean of , calculated as the average of all elements. Representing content image features The standard deviation of is calculated as the standard deviation of all elements in the feature map.

[0033] Preferably, in step S3, in order to achieve accurate style guidance based on text semantics, the present invention designs a cross-modal attention fusion module. Specifically, it includes the following steps:

[0034] S301: Image features and style text features F s The input is sent to the respective encoders (image encoder and text encoder) for preprocessing and normalization so that the image features and text features are consistent in their numerical range:

[0035]

[0036] in, Represents the content image features from the image encoder (features from different layers). s Features representing the style text description, extracted by the text encoder (CLIP).

[0037] S302: The image features and style text features are dimensionally adjusted through 1×1 convolution so that their number of channels is consistent and subsequent attention calculation can be performed.

[0038]

[0039] K=Conv2(F s )#(9)

[0040] V=Conv3(f s )#(10)

[0041] S303: Feature Mapping is used as query vector Q, style feature F s is mapped to key K and value V. The attention weight is calculated by QK T Calculate and apply to V to generate a cross-modal style guidance representation f attn .

[0042] Attn=softmax(QK T ),f attn =Attn×V#(11)

[0043] The style guidance signal and the original image features are weightedly fused through 1x1 convolution to generate a stylized feature map Used for subsequent diffusion modeling.

[0044]

[0045] Among them, W cs is the learned weight matrix used to adjust the influence of style features.

[0046] Preferably, in step S4, the present invention extracts masks of the subject, background, and details in the image based on the image structure characteristics, and dynamically assigns style transfer strengths to different regions. Through regional adaptive control, excessive background stylization or loss of subject details is effectively avoided, thereby improving the overall naturalness and detail fidelity of the image. Specifically, the steps include:

[0047] S401: Extract different area masks M in the image based on the image segmentation information or feature response r , divided into multiple regions such as subject, background, and details. Each region mask M r Indicates the location of the region in the image:

[0048]

[0049] in, For the style-guided image features, segmentation operations are performed to obtain multiple region masks.

[0050] S402: Assigning a migration weight λ to each region r , the fusion method is:

[0051]

[0052] Among them, ⊙ is the element-by-element multiplication to ensure the style adjustment at the regional level. r The style transfer strength of the control area, F a is the output feature of the adaptive control module, and finally the adjusted feature F is obtained r .

[0053] Preferably, in step S5, based on the style guidance features and content features, feature information is further integrated, and the local style strength is adjusted according to the guidance of the adaptive control module to generate an intermediate feature map with high consistency and rich details. The specific steps are as follows:

[0054] S501: Feature fusion is adjusted as follows:

[0055]

[0056] Among them, γ is the fusion strength adjustment coefficient.

[0057] Preferably, in step S6, the fused features are input into the decoder, and through convolution and upsampling operations, the final stylized image is restored layer by layer to ensure that the output image achieves an ideal balance between style expression and content structure preservation. The specific steps are as follows:

[0058] The decoding generation process is as follows:

[0059] I cs=D(F final )#(16)

[0060] Where D() represents the decoder operation.

[0061] Preferably, in step S7, the network parameters are optimized jointly using content loss and style loss. By dynamically adjusting the weight coefficients of the two types of losses, the style expression and content preservation are balanced. The Adam optimizer is used during the training process to accelerate convergence and improve the detail expression and style naturalness of the generated image. The specific steps are as follows:

[0062] S701: Content loss is used to ensure that the content features of the generated image are consistent with the original content image. The original image is extracted by the image encoder. and generate images feature representations, calculate the difference between them, and minimize the change of image content.

[0063] Content loss function L c Expressed as:

[0064]

[0065] Where N is the dimension of image features, and ||·||2 represents the Euclidean distance, which reflects the difference in image content features.

[0066] S702: Style loss is used to ensure that the style of the generated image is consistent with the target style description. In this framework, the style text is extracted through the text encoder to extract the feature F s , and calculate the style difference between the generated image and the target style image. The style difference is quantified by comparing their Gram matrices.

[0067] Style loss function L s Expressed as:

[0068]

[0069] Among them, G c is the Gram matrix of the content image, G gen is the Gram matrix of the generated image, and N is the dimension of the Gram matrix. By minimizing the style loss, the present invention can ensure that the generated image not only maintains the original content but also accurately reflects the semantic information of the target style.

[0070] S703: Adversarial loss introduces the Generative Adversarial Network (GANs) framework to improve the realism and detail quality of generated images through adversarial training between the generator and the discriminator. The goal of the generator is to minimize the adversarial loss so that the discriminator believes that the generated image is real. The adversarial loss function L a Expressed as:

[0071]

[0072] Among them, D(I gen ) is the adversarial judgment value output by the discriminator, indicating the generated image I gen Is it a real image? The goal of the generator is to make the discriminator think that the generated image is "real" and thus minimize the adversarial loss.

[0073] S704: Comprehensive loss function L t It is the weighted sum of content loss, style loss and adversarial loss, which is calculated as follows:

[0074] L t =λ c L c +λ s L s +λ a L a #(20)

[0075] Among them, λ c ,λ s ,λ a They are the weighted coefficients of content loss, style loss and adversarial loss, which control the influence of each loss in the total loss.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] (1) Improved semantic consistency and guidance flexibility of style transfer: By introducing a cross-modal attention mechanism based on the CLIP model, the present invention can guide the content image according to the style text description, so that the local area can be more closely aligned with the semantic features of the text during the image style transfer process. Although the overall style control accuracy is still limited by the expressive power of the basic encoder, compared with the method that only uses image features for guidance, the image and text matching degree has been significantly improved in subjective evaluation. In particular, when describing specific styles (such as "soft" and "retro"), the semantic consistency of the generated image is higher.

[0078] (2) Improved style transfer effects in different regions, and enhanced detail preservation capabilities: The adaptive regional style adjustment module proposed in this invention can dynamically adjust the transfer strength according to the image structure to a certain extent. By moderately strengthening style fusion in the main area and weakening style intervention in the background area, this method effectively reduces the problem of over-stylization of background textures, and retains more natural detail contours. Quantitative experiments show that in terms of the structural similarity (SSIM) indicator, the method of this invention has a reduced detail loss phenomenon compared to the unified style transfer method.

[0079] (3) The overall visual naturalness of the generated images has been improved: By leveraging cross-modal guidance and regional adaptive fusion, the color transitions and style rendering of the generated images are more natural. Compared with the traditional single-transfer strength method, the visual quality indicators of the present invention have declined. Although the problem of less smooth fusion in local areas still exists, the overall visual coherence and smoothness of style transitions have been substantially improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0081] Figure 1 Schematic diagram of the method flow of the present invention;

[0082] Figure 2 This is a framework diagram of the existing style transfer method;

[0083] Figure 3 This is a framework diagram of the style transfer method of the present invention;

[0084] Figure 4 This is a graph showing the application results of the image style transfer of the present invention in artistic creation. DETAILED DESCRIPTION

[0085] The present invention will be further explained below in detail with reference to the accompanying drawings so that those skilled in the art can have a deeper understanding of the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0086] like Figure 1 As shown in FIG, an image style transfer method based on cross-modal attention and adaptive regulation includes the following steps:

[0087] S1: Scale the content image to a uniform size (e.g., 512×512 pixels) and crop it to 256×256 pixels to ensure consistency of the input data. The style text description is preprocessed and standardized to achieve a uniform data format.

[0088] S2: A pre-trained convolutional neural network (VGG-19) is used to perform multi-level feature extraction on the content image to capture low-level texture and high-level semantic information. At the same time, the CLIP model is used to encode the style text and extract text features containing style semantics, providing semantic guidance for subsequent cross-modal fusion.

[0089] S3: Using the extracted content features as queries and the style text features as keys and values, we perform feature fusion via a cross-modal attention mechanism. This step accurately guides style transfer across different regions of the image based on text semantics, enabling text-driven fine-grained style expression.

[0090] S4: Based on the image's structural characteristics, masks are extracted for the subject, background, and details within the image, and dynamic style transfer strengths are assigned to different regions. This adaptive regional control effectively avoids excessive background stylization or loss of subject details, improving the overall naturalness and detail fidelity of the image.

[0091] S5: Based on the style-guiding features and content features, the feature information is further integrated and the local style strength is adjusted according to the guidance of the adaptive control module to ensure that the style and content of each region are balanced, generating an intermediate feature map with high consistency and rich details.

[0092] S6: The fused features are input into the decoder, and through convolution and upsampling operations, they are restored layer by layer to the final stylized image, ensuring that the output image achieves an ideal balance between style expression and content structure preservation.

[0093] S7: We use content loss and style loss to jointly optimize network parameters, dynamically adjusting the weights of the two losses to balance style expression and content preservation. We use the Adam optimizer during training to accelerate convergence and improve the detail and naturalness of the generated images.

[0094] In step S1, in order to achieve the standardization of the content image and the style text description, the following steps are specifically included:

[0095] S101: Content image I c Scaling to 512×512 pixels ensures a uniform input size and improves the accuracy of subsequent feature extraction.

[0096] S102: Perform center cropping on the scaled image to obtain a 256×256 pixel image I' c , retaining the main area of the image to the greatest extent.

[0097] S103: Normalize the image pixel values to the interval [0,1] to unify the data distribution and accelerate network convergence; text description T s It also undergoes word segmentation and standardized encoding to facilitate subsequent feature extraction.

[0098] The image normalization formula is:

[0099] I′ c =Crop 256×256 (Resize 512×512 (I c))#(1)

[0100] Among them, I c is the input content image.

[0101] In step S2, in order to fully extract the visual structure information of the content image and the style semantic information of the text description, a multimodal feature extraction strategy is adopted. Specifically, the following steps are included:

[0102] S201: Use pre-trained convolutional neural network (VGG-19) to train content image I' c Perform feature extraction layer by layer to obtain feature maps of each layer

[0103]

[0104] Among them, l is the convolution layer number, and different levels of texture, structure and semantic features are extracted through different convolution layers. For image encoder.

[0105] S202: Use CLIP model to style text T s Encode and extract text feature vectors:

[0106] F s =E text (T s )#(3)

[0107] Among them, T s For style text, E text A text encoder.

[0108] S203: Calculate the mean and standard deviation of content features and style features respectively, and unify the numerical scale:

[0109]

[0110] Represents the content image features of layer l extracted from the image encoder. N represents the total number of elements in the feature map of this layer (i.e., the dimension of the feature map). Representing content image features The i-th element in (i.e., the eigenvalue at the i-th position in the feature map). Representing content image features The mean of , calculated as the average of all elements. Representing content image features The standard deviation of is calculated as the standard deviation of all elements in the feature map.

[0111] In step S3, in order to achieve accurate style guidance based on text semantics, the present invention designs a cross-modal attention fusion module. Specifically, it includes the following steps:

[0112] S301: Image features and style text features F s The input is sent to the respective encoders (image encoder and text encoder) for preprocessing and normalization so that the image features and text features are consistent in their numerical range:

[0113]

[0114] in, Represents the content image features from the image encoder (features from different layers). s Features representing the style text description, extracted by the text encoder (CLIP).

[0115] S302: The image features and style text features are dimensionally adjusted through 1×1 convolution so that their number of channels is consistent and subsequent attention calculation can be performed.

[0116]

[0117] K=Conv2(F s )#(9)

[0118] V=Conv3(f s )#(10)

[0119] S303: Feature Mapping is used as query vector Q, style feature F s is mapped to key K and value V. The attention weight is calculated by QK T Calculate and apply to V to generate a cross-modal style guidance representation f attn .

[0120] Attn=softmax(QK T ),f attn =Attn×V#(11)

[0121] The style guidance signal and the original image features are weightedly fused through 1x1 convolution to generate a stylized feature map Used for subsequent diffusion modeling.

[0122]

[0123] Among them, W cs is the learned weight matrix used to adjust the influence of style features.

[0124] In step S4, the present invention extracts masks of the subject, background, and details in the image based on the image's structural characteristics and dynamically assigns style transfer strengths to different regions. This adaptive regional control effectively avoids excessive background stylization or loss of subject details, improving the overall naturalness and detail fidelity of the image. Specifically, the process includes the following steps:

[0125] S401: Extract different area masks M in the image based on the image segmentation information or feature response r , divided into multiple regions such as subject, background, and details. Each region mask M r Indicates the location of the region in the image:

[0126]

[0127] in, For the style-guided image features, segmentation operations are performed to obtain multiple region masks.

[0128] S402: Assigning a migration weight λ to each region r , the fusion method is:

[0129]

[0130] Among them, ⊙ is the element-by-element multiplication to ensure the style adjustment at the regional level. r The style transfer strength of the control area, F a is the output feature of the adaptive control module, and finally the adjusted feature F is obtained r .

[0131] like Figure 3 As shown, in step S5, based on the style guidance features and content features, feature information is further integrated, and the local style strength is adjusted according to the guidance of the adaptive control module to generate an intermediate feature map with high consistency and rich details. The specific steps are as follows:

[0132] S501: Feature fusion is adjusted as follows:

[0133]

[0134] Among them, γ is the fusion strength adjustment coefficient.

[0135] In step S6, the fused features are input into the decoder, and through convolution and upsampling operations, they are restored layer by layer to the final stylized image, ensuring that the output image achieves an ideal balance between style expression and content structure preservation. The specific steps are as follows:

[0136] The decoding generation process is as follows:

[0137] I cs =D(Ffinal )#(16)

[0138] Where D() represents the decoder operation.

[0139] In step S7, the network parameters are optimized jointly using content loss and style loss. By dynamically adjusting the weight coefficients of the two types of losses, a balance is achieved between style expression and content preservation. The Adam optimizer is used during training to accelerate convergence and improve the detail expression and naturalness of the generated images. The specific steps are as follows:

[0140] S701: Content loss is used to ensure that the content features of the generated image are consistent with the original content image. The original image is extracted by the image encoder. and generate images feature representations, calculate the difference between them, and minimize the change of image content.

[0141] Content loss function L c Expressed as:

[0142]

[0143] Where N is the dimension of image features, and ||·||2 represents the Euclidean distance, which reflects the difference in image content features.

[0144] S702: Style loss is used to ensure that the style of the generated image is consistent with the target style description. In this framework, the style text is extracted through the text encoder to extract the feature F s , and calculate the style difference between the generated image and the target style image. The style difference is quantified by comparing their Gram matrices.

[0145] Style loss function L s Expressed as:

[0146]

[0147] Among them, G c is the Gram matrix of the content image, G gen is the Gram matrix of the generated image, and N is the dimension of the Gram matrix. By minimizing the style loss, the present invention can ensure that the generated image not only maintains the original content but also accurately reflects the semantic information of the target style.

[0148] S703: Adversarial loss introduces the Generative Adversarial Network (GANs) framework to improve the realism and detail quality of generated images through adversarial training between the generator and the discriminator. The goal of the generator is to minimize the adversarial loss so that the discriminator believes that the generated image is real. The adversarial loss function L a Expressed as:

[0149]

[0150] Among them, D(I gen ) is the adversarial judgment value output by the discriminator, indicating the generated image I gen Is it a real image? The goal of the generator is to make the discriminator think that the generated image is "real" and thus minimize the adversarial loss.

[0151] S704: Comprehensive loss function L t It is the weighted sum of content loss, style loss and adversarial loss, which is calculated as follows:

[0152] L t =λ c L c +λ s L s +λ a L a #(20)

[0153] Among them, λ c ,λ s ,λ a They are the weighted coefficients of content loss, style loss and adversarial loss, which control the influence of each loss in the total loss.

[0154] In specific implementation, Example 1: Cross-modal style transfer on public datasets

[0155] This example compares the accuracy of our cross-modal attention and adaptive control-based image style transfer method with traditional style transfer methods. Large-scale testing on multiple style transfer datasets demonstrates the advantages of our method in improving the representation of style details and preserving content.

[0156] We randomly selected and combined 8,000 images from multiple standard style transfer datasets, including the VGG19 style image set, WikiArt, and COCO, as the test set. These images cover a variety of styles (oil painting, impressionism, abstract art, etc.) and include different resolutions and scenes to ensure the diversity and representativeness of the dataset.

[0157] This experiment uses PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and Style Detail Retention (SDR) as evaluation metrics to calculate the difference between the generated image and the style image. To ensure fairness, the proposed method is compared with a traditional VGG-based style transfer method, using the same evaluation criteria and parameter settings.

[0158] Each test image is input into the method of the present invention and divided into content and style feature streams after data preprocessing. The feature extraction module includes a cross-modal attention mechanism and an adaptive control module to optimize the style expression, detail retention and content consistency of the generated image, ensuring the optimal balance between style and detail.

[0159] Experimental results show that the proposed method achieves a PSNR of 34.2dB on the test set, a 3.4dB improvement over the traditional method (30.8dB). The SSIM value is 0.91, compared to 0.83 for the traditional method, demonstrating a significant improvement in the structural consistency of the generated images. The style detail retention (SDR) is also improved by 15%, particularly in complex style images, where the proposed method is able to better capture details and textures.

[0160] Example 2: Low-resolution image style transfer

[0161] This example demonstrates the effectiveness of our cross-modal attention and adaptive control-based image style transfer method for low-resolution images, particularly in preserving style details. By comparing this method to traditional style transfer methods, we further demonstrate its advantages in low-resolution image processing, particularly in terms of detail preservation and style consistency.

[0162] To simulate the style transfer problem of low-resolution images, this experiment scaled the content image and style image to 128×128 pixels and 256×256 pixels, respectively. The experiment used a variety of images, including portraits, natural scenery, and cityscapes, to ensure the diversity and representativeness of the test dataset.

[0163] This experiment uses PSNR, SSIM, and Style Detail Retention (SDR) as evaluation metrics to calculate the difference between the generated image and the style image. To ensure fairness, the proposed method is compared with a traditional VGG-based style transfer method, using the same evaluation criteria and parameter settings.

[0164] Each test image is subjected to feature extraction using the image encoder and text encoder of the proposed method. Subsequently, style transfer is performed on the content image and style image using a cross-modal attention mechanism and adaptive control module. Specifically, during low-resolution image processing, the proposed method dynamically adjusts regional style strengths, effectively avoiding the detail loss and blurring issues commonly encountered in style transfer from low-resolution images, ensuring detailed expression and consistent style in the generated image.

[0165] Experimental results show that the proposed method achieves a PSNR of 30.5dB on low-resolution images, a 3.8dB improvement over the traditional method (26.7dB). The SSIM value is 0.85, compared to 0.78 for the traditional method, demonstrating that the proposed method can maintain good image structure in low-resolution images. The proposed method also improves style detail preservation by 12%, with significant advantages particularly in color and texture representation, resulting in stylized images.

[0166] This example verifies the significant advantages of the method of the present invention in low-resolution image style transfer, especially in terms of detail preservation and style consistency. Compared with traditional methods, the method of the present invention can generate stylized images that are clearer, more natural, and more artistic.

[0167] like Figure 4 As shown, Example 3: Application of image style transfer in artistic creation

[0168] This example demonstrates the application of the image style transfer method based on cross-modal attention and adaptive regulation in artistic creation, especially in artistic style imitation and creative practice. Artists often need to deepen their understanding of different artistic styles and improve their creative efficiency by converting ordinary images into works of specific artistic styles. By applying the method of the present invention, artistic creation can receive significant technical support, making image style transfer more accurate and natural while maintaining high-quality artistic expression.

[0169] In the experiment, we selected a target style image set from art libraries such as WikiArt, encompassing styles such as Impressionism, Post-Impressionism, Cubism, and Abstract Art. We also selected 5,000 natural landscape and portrait photos as content images. To ensure data diversity and representativeness, the images included not only classic landscapes and portraits, but also cityscapes and abstract artworks.

[0170] Each image is first processed through an image encoder (VGG network) to extract content features, while the style text is processed through the CLIP model to extract style semantic features. Then, using the cross-modal attention mechanism and adaptive control module in the proposed method, style transfer is performed between the content image and the style image. Specifically, during the style transfer process, the style transfer strength is dynamically adjusted based on the feature differences between different regions of the image (such as the subject and background), ensuring that details are not lost and that the style transition is natural and harmonious.

[0171] Experimental results demonstrate that the proposed method can generate high-quality artistic style images, particularly with outstanding detail preservation. Compared to traditional style transfer methods, the stylized images generated by the proposed method exhibit more vivid colors, textures, and details. This is particularly true for images in complex artistic styles, such as Impressionism and abstract art, where the proposed method is able to better preserve the artistic characteristics and creative details of the original image.

[0172] Furthermore, the method's adaptive control module intelligently adjusts the strength of style transfer based on the content and stylistic characteristics of different regions, avoiding the loss of detail and over-stylization common in traditional methods and ensuring a good balance between artistic style and content in the final image. This method allows artists to easily transform their own images into works of art with diverse styles, providing an efficient and flexible creative tool for both artistic creation and style learning.

[0173] This embodiment demonstrates the wide application potential of the method of the present invention in artistic creation. It can not only help artists understand the characteristics of different artistic styles, but also accelerate the creation process of artistic styles and broaden the technical boundaries of artistic creation.

[0174] Figure 1 Experimental results diagram

[0175] Example 4: Application of style transfer in social media image processing

[0176] This example demonstrates the application of the image style transfer method based on cross-modal attention and adaptive regulation in social media image style processing. With the popularity of social media, users often want to transform personal photos or other images into works with a specific artistic style so that they can be shared on the platform. In order to meet users' needs for personalized expression and artistic effects, style transfer technology plays an important role in social media image editing. This example verifies the application of the method of the present invention in this field, especially its advantages in processing image details, style consistency and generation efficiency.

[0177] In the experiment, the present invention collected a large number of images from social media platforms, including landscapes, portraits, and cityscapes. A variety of artistic styles, such as Impressionism, abstract art, and oil painting, were selected as the target style image set. The goal of the experiment was to verify the efficiency and effectiveness of the present method in social media image editing, ensuring that user-uploaded images could be quickly and accurately converted into the desired artistic style while maintaining visual quality.

[0178] This experiment uses PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), and style detail retention (SDR) as evaluation indicators to calculate the difference between the generated image and the original content image to comprehensively evaluate the style transfer effect of the method of the present invention.

[0179] Experimental results show that the proposed method can effectively enhance the artistic style expression of social media images. The generated stylized images are significantly superior to traditional style transfer methods in terms of detail preservation and style consistency. Specifically, the PSNR of the images generated by the proposed method is 31.5dB, which is 3.3dB higher than the traditional method (28.2dB); the SSIM value is 0.89, while the traditional method is 0.81, indicating that the generated images have a significant improvement in structural consistency. In terms of style detail retention (SDR), the proposed method has improved by 10%, especially in the expression of complex styles, and can effectively preserve the texture and details of the style.

[0180] Furthermore, the proposed method demonstrates excellent computational efficiency, enabling high-quality image style transfer in a short timeframe, meeting the real-time and user experience requirements of social media platforms. Compared to traditional methods, the stylized images generated by the proposed method exhibit greater artistry and creativity, while more precisely matching user needs and enhancing the image editing capabilities of social media platforms.

[0181] This example demonstrates the potential of the method of the present invention for image style processing on social media, and can provide social media users with a more personalized and high-quality image editing experience, thus having broad market application prospects.

[0182] The specific implementation scheme described above further illustrates in detail the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in this field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. An image style transfer method based on cross-modal attention and adaptive control, characterized by: The steps include: Step S1: Scale the content image to a uniform size and crop it to 256×256 pixels; standardize and encode the style text description to a unified data format; Step S2: Use the pre-trained convolutional neural network VGG-19 to perform multi-level feature extraction on the content image to capture low-level texture and high-level semantic information; use the CLIP model to encode the style text and extract text features containing style semantics; Step S3: The extracted content features are used as queries, and the style text features are used as keys and values, and feature fusion is performed through the cross-modal attention mechanism; Step S4: extracting masks of the main body, background, details and other areas in the image based on the image structure characteristics, and assigning dynamic style transfer strength to different areas; Step S5: Fuse feature information and adjust local style strength according to the guidance of the adaptive control module to ensure that the style and content of each region are taken into account, and generate an intermediate feature map with high consistency and rich details; Step S6: The fused features are input into the decoder, and restored layer by layer to the final stylized image through convolution and upsampling operations; Step S7: Use content loss and style loss to jointly optimize network parameters, and balance style expression and content preservation by dynamically adjusting the loss weight coefficient.

2. The image style transfer method based on cross-modal attention and adaptive control according to claim 1 is characterized in that In step S1, in order to achieve the standardization of the content image and the style text description, the following steps are specifically included: S101: Content image I c Scaling to 512×512 pixels ensures a uniform input size and improves the accuracy of subsequent feature extraction; S102: Perform center cropping on the scaled image to obtain a 256×256 pixel image I' c , retain the main area of the image to the greatest extent; S103: Normalize the image pixel values to the interval [0,1] to unify the data distribution and accelerate network convergence; text description T s It also undergoes word segmentation and standardized coding to facilitate subsequent feature extraction; The image normalization formula is: I' c =Crop 256×256 (Resize 512×512 (I c )) (1) Among them, I c is the input content image.

3. The image style transfer method based on cross-modal attention and adaptive control according to claim 2, characterized in that: In step S2, in order to fully extract the visual structure information of the content image and the style semantic information of the text description, a multimodal feature extraction strategy is adopted, which specifically includes the following steps: S201: Use the pre-trained convolutional neural network VGG-19 to train the content image I' c Perform layer-by-layer feature extraction to obtain feature maps of each layer Among them, l is the convolution layer number, and different levels of texture, structure and semantic features are extracted through different convolution layers respectively; is an image encoder; S202: Use CLIP model to style text T s Encode and extract text feature vectors: F s =E text (T s ) (3) Among them, T s For style text, E text For text encoder; S203: Calculate the mean and standard deviation of content features and style features respectively, and unify the numerical scale: in, represents the content image features of the lth layer extracted from the image encoder; N represents the total number of elements in the feature map of this layer; Representing content image features The i-th element in ; Representing content image features The mean of , calculated as the average of all elements; Representing content image features The standard deviation of is calculated as the standard deviation of all elements in the feature map.

4. The image style transfer method based on cross-modal attention and adaptive control according to claim 3 is characterized in that The step S3 specifically includes the following steps: S301: Image features and style text features F s The input is sent to the respective encoders, namely the image encoder and the text encoder, for preprocessing and normalization so that the image features and text features are consistent in their numerical ranges: in, Represents the content image features from the image encoder, features from different layers; F s Features representing the style text description, extracted by the text encoder CLIP; S302: The image features and the style text features are dimensionally adjusted through 1×1 convolution so that their number of channels is consistent for subsequent attention calculation; K=Conv2(F s ) (9) V=Conv3(f s ) (10) S303: Feature Mapping is used as query vector Q, style feature F s Mapped to key K and value V; attention weight is passed through QK T Calculate and apply to V to generate a cross-modal style guidance representation f attn ; Attn=softmax(QK T ),f attn =Attn×V (11) The style guidance signal and the original image features are weightedly fused through 1x1 convolution to generate a stylized feature map For subsequent diffusion modeling; Among them, W cs is the learned weight matrix used to adjust the influence of style features.

5. The image style transfer method based on cross-modal attention and adaptive control according to claim 4, characterized in that: In step S4, masks of the subject, background, and details in the image are extracted based on the image structure characteristics, and dynamic style transfer strengths are assigned to different regions. Through regional adaptive control, excessive background stylization or loss of subject details is effectively avoided, thereby improving the overall naturalness and detail fidelity of the image. The specific steps include: S401: Extract different area masks M in the image based on the image segmentation information or feature response r , divided into multiple regions of subject, background, and details, each region mask M r Indicates the location of the region in the image: in, Perform segmentation operations on the style-guided image features to obtain multiple region masks; S402: Assigning a migration weight λ to each region r , the fusion method is: Among them, ⊙ is the element-by-element multiplication to ensure the style adjustment at the regional level; λ r The style transfer strength of the control area, F a is the output feature of the adaptive control module, and finally the adjusted feature F is obtained r .

6. The image style transfer method based on cross-modal attention and adaptive control according to claim 5, characterized in that: In step S5, based on the style guidance features and content features, feature information is further integrated, and the local style strength is adjusted according to the guidance of the adaptive control module to generate an intermediate feature map with high consistency and rich details. The specific steps are as follows: S501: Feature fusion is adjusted as follows: Among them, γ is the fusion strength adjustment coefficient.

7. The image style transfer method based on cross-modal attention and adaptive control according to claim 6, characterized in that In step S6, the fused features are input into the decoder, and the final stylized image is restored layer by layer through convolution and upsampling operations to ensure that the output image achieves an ideal balance between style expression and content structure preservation. The specific steps are as follows: The decoding generation process is as follows: I cs =D(F final ) (16) Where D() represents the decoder operation.

8. The image style transfer method based on cross-modal attention and adaptive control according to claim 7, characterized in that: In step S7, the network parameters are optimized jointly using content loss and style loss. By dynamically adjusting the weight coefficients of the two types of losses, the style expression and content preservation are balanced. The Adam optimizer is used during the training process to accelerate convergence and improve the detail expression and style naturalness of the generated image. The specific steps are as follows: S701: Content loss is used to ensure that the content features of the generated image are consistent with the original content image, and the original image is extracted through the image encoder. and generate images feature representations, calculate the differences between them, and minimize the changes in image content; Content loss function L c Expressed as: Where N is the dimension of image features, ||·||2 represents the Euclidean distance, which reflects the difference in image content features; S702: Style loss is used to ensure that the style of the generated image is consistent with the target style description. The style text is extracted through the text encoder to extract the feature F s , and calculate the style difference between the generated image and the target style image. The style difference is quantified by comparing their Gram matrices; Style loss function L s Expressed as: Among them, G c is the Gram matrix of the content image, G gen is the Gram matrix that generates the image, and N is the dimension of the Gram matrix; S703: Adversarial loss introduces the Generative Adversarial Network (GANs) framework. Through adversarial training between the generator and the discriminator, the realism and detail quality of the generated image are improved. The goal of the generator is to minimize the adversarial loss so that the discriminator believes that the generated image is real. The adversarial loss function L a Expressed as: Among them, D(I gen ) is the adversarial judgment value output by the discriminator, indicating the generated image I gen Is it a real image? The goal of the generator is to make the discriminator think that the generated image is real, thereby minimizing the adversarial loss. S704: Comprehensive loss function L t It is the weighted sum of content loss, style loss and adversarial loss, which is calculated as follows: L t =λ c L c +λ s L s +λ a L a (20) Among them, λ c ,λ s ,λ a They are the weighted coefficients of content loss, style loss and adversarial loss, which control the influence of each loss in the total loss.

Images