Image style migration method based on ViT pre-training model and multi-control network, electronic equipment and storage medium

Abstract

The invention discloses an image style migration method based on a ViT pre-training model and a multi-control network, electronic equipment and a storage medium, and the core idea of the method is to construct a generation framework which takes the ViT pre-training model (such as a VAE encoder of Stable Diffusion and a Transform block in UNet) as a bottom layer basis and innovatively integrates two major control modules of IP-Adapter and ControlNet. According to the framework, the style semantics of a reference image is injected through the IP-Adapter, and the structural constraint of a content image is kept through the ControlNet, so that accurate, controllable and high-quality image style transfer drawing is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically refers to an image style transfer method, electronic device, and storage medium based on a ViT pre-trained model and a multi-control network. Background Technology

[0002] Neural style transfer in the field of artificial intelligence is a technique that uses deep learning to transfer the visual style of one image to the content of another. It is generally achieved by analyzing features extracted by a convolutional neural network (CNN) to generate a new image that combines the characteristics of both images.

[0003] Existing image style transfer methods have the following limitations: (1) Weak style control: Traditional methods (such as Gram matrix) and single model methods have difficulty accurately transferring the complex semantics, texture and color style of the style image used as a reference to the content image, especially when the style image and the content image do not match semantically, the effect is poor.

[0004] (2) Insufficient preservation of content structure: When implementing strong stylization, the key structure, outline and details of the content image are often easily lost or distorted, resulting in a completely different generated result.

[0005] (3) Limited diversity of generated results: Existing methods usually output a single, deterministic result and lack the ability to flexibly control the degree of stylization and details without changing the core content. Summary of the Invention

[0006] The main objective of this invention is to provide an image style transfer method, electronic device, and storage medium based on a ViT pre-trained model and a multi-control network. This invention aims to address the shortcomings of existing technologies, enabling precise style control through a single reference image, effectively maintaining the spatial structure of the content image, and simultaneously outputting high-quality and diverse effects.

[0007] To achieve the above objectives, one solution of the present invention is: An image style transfer method based on a ViT pre-trained model and a multi-control network includes: Step 1: Process the three inputs in parallel, namely: (1) Content encoding: The VAE encoder is used to encode the content image. Encode into a low-dimensional latent space to obtain a latent representation. .

[0008] (2) Style condition extraction: The style image used as a reference The input is a dedicated image encoder for the IP-Adapter, which extracts multi-level image features as style conditions. ,in This refers to the image encoder of the IP-Adapter.

[0009] (3) Spatial control condition extraction: extracting content images Input a pre-trained ControlNet encoder, extract its spatial structure information, and obtain the control conditions. ,in This refers to the ControlNet encoder.

[0010] Step 2: Using a UNet network built on ViT Transformer blocks, the following three types of conditional information are simultaneously fused during the denoising process of the diffusion model: (1) Time step embedding: embedding the time steps of the diffusion process The vector is converted into a mathematical vector through mathematical transformation and then input into UNet.

[0011] (2) Style condition injection: The IP-Adapter module will extract the style conditions. As the key and value, they are used in conjunction with the query from the cross-attention layer in UNet for computation, using the following formula: ; in, This indicates cross-attention calculation; This indicates a query. Indicates Key; Represents Value; Represents a stable function; Index of the style reference image; A visual resource library representing style reference images; and All are determined by style conditions Mapped to; superscript Indicates the transpose operation; Indicates the dimension of the Key vector.

[0012] (3) Spatial control condition injection: The ControlNet module takes the encoder features of UNet as input and injects them with the extracted control conditions. Combined, through its zero convolutional layer ( Output a feature offset and add it to the backbone features of UNet. The calculation formula is as follows: ; in, Indicates the noise in the prediction; Indicates the first The noise potential representation of the step.

[0013] Step 3: After After iterative denoising steps, the denoised latent representation is obtained. The VAE decoder is used to denoise the latent representation. Decode back into pixel space to obtain the final stylized output image. ,in This indicates the VAE decoder.

[0014] Furthermore, the entire training process of this method is divided into two parts: (1) IP-Adapter training: Freeze the pre-trained diffusion model and the weights of ControlNet, and train only the image encoder of IP-Adapter and its corresponding embedding projection layer; use image-text pair data to learn to align image features to the text cross attention space.

[0015] (2) ControlNet training: For preset control conditions, train the zero convolutional layer and encoder of ControlNet to learn how to map the input control conditions into the correct modulation signal of UNet features.

[0016] The second solution of the present invention is: An electronic device includes a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the image style transfer method.

[0017] The third solution of the present invention is: A computer-readable storage medium storing a computer program; when executed in the computer, the computer program causes the computer to perform the image style transfer method.

[0018] By adopting the above technical solution, this invention constructs a deeply collaborative multi-condition control architecture. This architecture achieves precise, decoupled, and flexible control over the style and structure of the generated results. Users can generate diverse, high-quality results by changing different style reference images and control condition types, fundamentally solving the problems of weak control capabilities and monotonous output in traditional image style transfer methods. Existing technologies are merely single ViT-based style transfer methods or diffusion models using only text prompts. Furthermore, this invention has the following advantages: (I) High precision in style control and strong semantic correspondence; (II) The content structure has extremely high fidelity; (III) Excellent flexibility and controllability; (IV) High quality of generated images. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall process of a specific embodiment of the present invention. Detailed Implementation

[0020] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0021] The core idea of ​​this invention is to construct a generative framework based on ViT pre-trained models (such as the VAE encoder in Stable Diffusion and the Transformer block in UNet) and innovatively integrating two major control modules: IP-Adapter and ControlNet. This framework injects the style semantics of the reference image through the IP-Adapter and maintains the structural constraints of the content image through ControlNet, thereby achieving accurate, controllable, and high-quality image style transfer.

[0022] refer to Figure 1 As shown, this invention discloses an image style transfer method based on a ViT pre-trained model and a multi-control network, comprising: Step 1. Image Encoding and Conditional Extraction Parallel processing of three inputs, namely: (1) Content encoding: The content image is encoded using a VAE (Variational Autoencoder). Encode into a low-dimensional latent space to obtain a latent representation. .

[0023] (2) Style condition extraction: The style image used as a reference The input is a dedicated image encoder for the IP-Adapter (typically a ViT-based CLIP image encoder), which extracts multi-level image features as style conditions. ,in This represents the image encoder of the IP-Adapter; the IP-Adapter injects style conditions into the UNet through a decoupled cross-attention mechanism to guide the generation process to approximate the style image.

[0024] (3) Spatial control condition extraction: extracting content images Input a pre-trained ControlNet encoder (such as an edge detector), extract its spatial structure information (such as an edge map), and obtain the control conditions. (Taking the edge as an example), where This indicates the ControlNet encoder; this condition will be used to precisely constrain the structure of the generated image to be consistent with the content image.

[0025] Step 2. Conditional Injection and Diffusion Generation This is the core step of the invention, specifically using a UNet network built based on ViT Transformer blocks to simultaneously fuse the following three types of conditional information during the denoising process of diffusion models (such as Latent Diffusion Models): (1) Time step embedding: embedding the time steps of the diffusion process The vector is transformed into an information-rich, high-dimensional mathematical vector through mathematical transformation, and then input into UNet to guide UNet in making the correct denoising decision at the current step.

[0026] (2) Style condition injection: The IP-Adapter module will extract the style conditions. As the key and value, it is computed with the query in the cross-attention layer of UNet, dynamically and adaptively injecting style conditions into each step of the denoising process; the cross-attention calculation formula is as follows: ; in, This indicates cross-attention calculation; This indicates a query. Indicates Key; Represents Value; Represents a stable function; An index (or "catalog") representing style reference images, which encodes the characteristics of various visual elements in the style images, such as brushstrokes, color blocks, texture patterns, etc. This represents a specific "visual resource library" of style reference images, which contains the specific representations of the visual elements in the style images; and All are determined by style conditions Mapped to; superscript Indicates the transpose operation; The dimension of the key vector is indicated, and its most important role in the formula is as a scaling factor. (3) Spatial control condition injection: The ControlNet module takes the encoder features of UNet as input and injects them with the extracted control conditions. Combined, through its zero convolutional layer ( The output is a feature offset, which is added to the backbone features of UNet; this process ensures that the generation process is strongly constrained by the content image structure at each step of denoising; the comprehensive denoising process of UNet can be represented as: ; in, Indicates the noise in the prediction; Indicates the first The noise potential representation of the step.

[0027] Step 3. Image Decoding and Output go through After iterative denoising steps, the denoised latent representation is obtained. The VAE decoder is used to denoise the latent representation. Decode back into pixel space to obtain the final stylized output image. ,in This refers to the VAE decoder, which is a pre-trained neural network whose core function is to "decode" compressed, abstract data back into a complete image.

[0028] Furthermore, the training process of this invention is divided into two parts: (1) IP-Adapter training: Freeze the weights of the pre-trained diffusion model (UNet, VAE) and ControlNet, and train only the image encoder of IP-Adapter and its corresponding embedding projection layer; use image-text pair data to learn to align image features to the text cross attention space so that image features can be used as conditions to effectively guide generation.

[0029] (2) ControlNet training: For specific control conditions (such as edge maps), train the zero convolutional layer and encoder of ControlNet to learn how to map the input control conditions into the correct modulated signal of UNet features.

[0030] Through the above scheme, this invention constructs a deeply collaborative multi-condition control architecture. This architecture achieves precise, decoupled, and flexible control over the style and structure of the generated results. Users can generate diverse, high-quality results by changing different style reference images and control condition types, fundamentally solving the problems of weak control capabilities and monotonous output in traditional image style transfer methods. Existing technologies are merely single ViT-based style transfer methods or diffusion models using only text prompts. Specifically, compared with these technologies, the advantages of this invention are: (I) High style control precision and strong semantic correspondence: Through the IP-Adapter module, the present invention can directly understand and transfer the overall visual style and detailed texture of the reference style image without complex text description; compared with the Gram matrix-based method, the IP-Adapter is based on ViT features, which can capture higher-level semantic information, and the style transfer effect is more accurate and natural.

[0031] (II) Extremely high fidelity of content structure: By introducing ControlNet, this invention can strictly maintain the key spatial structure and contour (such as edges and pose) of the content image. Even when applying a very abstract or strong artistic style, the content structure of the generated image will not be distorted, solving the core pain point of insufficient content preservation in traditional methods.

[0032] (III) Excellent Flexibility and Controllability: This invention provides a flexible framework. Users can obtain an infinite number of styles by changing different style reference maps; they can control the structural fidelity by adjusting the control strength of ControlNet (such as the threshold of the edge map); and they can even combine text prompts for more granular control. This multi-condition control capability is unmatched by a single model.

[0033] (IV) High quality of generated images: The generative framework based on the latent diffusion model (LDM) can generate high-resolution, high-visual-fidelity images. Combined with the global modeling capabilities of ViT, the generated stylized images are superior to many traditional methods in terms of overall consistency and local detail.

[0034] The present invention also discloses an electronic device, including a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the methods described above.

[0035] The present invention also discloses a computer-readable storage medium storing a computer program; when the computer program is executed in a computer, it causes the computer to perform the above-described method. The storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0036] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. An image style transfer method based on a ViT pre-trained model and a multi-control network, characterized in that, include: Step 1: Process the three inputs in parallel, namely: (1) Content encoding: The VAE encoder is used to encode the content image. Encode into a low-dimensional latent space to obtain a latent representation. ; (2) Style condition extraction: The style image used as a reference The input is a dedicated image encoder for the IP-Adapter, which extracts multi-level image features as style conditions. ,in This refers to the image encoder of the IP-Adapter; (3) Spatial control condition extraction: extracting content images Input a pre-trained ControlNet encoder, extract its spatial structure information, and obtain the control conditions. ,in Indicates ControlNet encoder; Step 2: Using a UNet network built on ViT Transformer blocks, the following three types of conditional information are simultaneously fused during the denoising process of the diffusion model: (1) Time step embedding: embedding the time steps of the diffusion process The vector is converted into a mathematical vector through mathematical transformation and then input into UNet; (2) Style condition injection: The IP-Adapter module will extract the style conditions. As the key and value, they are used in conjunction with the query from the cross-attention layer in UNet for computation, using the following formula: ； in, This indicates cross-attention calculation; Indicates Query; Indicates Key; Represents Value; Represents a stable function; Index of the style reference image; A visual resource library representing style reference images; and All are determined by style conditions Mapped to; superscript Indicates the transpose operation; Indicates the dimension of the Key vector; (3) Spatial control condition injection: The ControlNet module takes the encoder features of UNet as input and injects them with the extracted control conditions. In combination, a feature offset is output from its zero convolutional layer and added to the backbone features of UNet. The calculation formula is as follows: ； in, Indicates the noise in the prediction; Indicates the first The potential noise representation of the step; Step 3: After After iterative denoising steps, the denoised latent representation is obtained. The VAE decoder is used to denoise the latent representation. Decode back into pixel space to obtain the final stylized output image. ,in This indicates the VAE decoder.

2. The image style transfer method based on the ViT pre-trained model and multi-control network as described in claim 1, characterized in that... The training process of the method consists of two parts: (1) IP-Adapter training: Freeze the pre-trained diffusion model and the weights of ControlNet, and train only the image encoder of IP-Adapter and its corresponding embedding projection layer; use image-text pair data to learn to align image features to the text cross attention space; (2) ControlNet training: For preset control conditions, train the zero convolutional layer and encoder of ControlNet to learn how to map the input control conditions into the correct modulation signal of UNet features.

3. An electronic device, characterized in that, It includes a processor, memory, and an application program; the application program is stored in the memory and configured to be executed by the processor as described in claim 1 or 2, using an image style transfer method based on a ViT pre-trained model and a multi-control network.

4. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed in the computer, it causes the computer to perform the image style transfer method based on the ViT pre-trained model and multi-control network as described in claim 1 or 2.

Images