Multi-modal large model-based portrait style migration method, storage medium and equipment

Abstract

The invention discloses a portrait style migration method based on a multi-modal large model, a storage medium and equipment, and the method comprises the steps: obtaining an original portrait image, extracting spatial structure features in the original portrait image, and providing a target portrait style; encoding the target portrait style into a semantic vector through a multi-modal large model; and constructing a diffusion model based on a ControlNet architecture, gradually adding Gaussian noise to the original human image until the original human image becomes a pure noise image, and guiding the pure noise image to gradually generate an image of a target portrait style by using semantic vectors and spatial structure features. According to the method, generation of the image of the target portrait style is guided by utilizing the semantic vector and the spatial structure characteristics, so that the portrait style migration process is more flexible, a finer portrait style control option is provided, the portrait style migration image with high resolution and clear details can be generated, and the requirement of a user on the image quality is met.

Description

Technical Field

[0001] This invention relates to the field of portrait generation technology, specifically to a portrait style transfer method, storage medium, and device based on a multimodal large model. Background Technology

[0002] Early portrait style transfer techniques relied on image filtering and graphics methods. While these could achieve some basic stylization effects, they lacked a deep understanding of image semantics and couldn't handle complex artistic styles. Subsequently, neural network-based style transfer methods utilized convolutional neural networks to extract content and style features, generating images that both preserved content and matched the portrait style through iterative optimization, gaining widespread research attention. However, these methods are computationally expensive, especially slow on high-resolution images, and the balance between portrait style and content is difficult to control flexibly, often resulting in unstable details in the generated results, affecting practical applications. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides a method, storage medium, and device for portrait style transfer based on a multimodal large model, which improves the quality and resolution of portrait style transfer and reduces the computational resource requirements.

[0004] To achieve the above technical objectives, the present invention adopts the following technical solution: A portrait style transfer method based on a multimodal large model includes the following steps: Step S1: Obtain the original portrait image and extract the spatial structure features from the original portrait image; Step S2: Provide the target portrait style and encode the target portrait style into a semantic vector using a multimodal large model; Step S3: Construct a diffusion model based on the ControlNet architecture, gradually add Gaussian noise to the original portrait image until it becomes a pure noise image, and use semantic vectors and spatial structure features to guide the pure noise image to gradually generate an image in the style of the target portrait.

[0005] Furthermore, the target portrait style provided in step S1 is one or more of the following: a text description of the target portrait style and a reference image of the target portrait style.

[0006] Furthermore, if a text description of the target portrait style is provided, the text description is encoded into a semantic vector using the CLIP architecture text-image alignment model; if a reference image of the target portrait style is provided, a visual style representation is extracted using an image encoder and encoded into a semantic vector.

[0007] Furthermore, the denoising network in the diffusion model based on the ControlNet architecture adopts U-Net, which includes: a downsampling module, a cross-attention module, an upsampling module, and a self-attention module; At each time step, the downsampling module is used to extract multi-scale feature maps from the current portrait style image, the cross-attention module generates a portrait style enhancement feature map based on the multi-scale feature map and semantic vector, the upsampling module generates a portrait style image based on the portrait style enhancement feature map and the current portrait style image, and the self-attention module dynamically corrects the portrait style image using spatial structure features. Iterate through all time steps to obtain an image in the style of the target portrait. In the first time step, the downsampling module inputs a pure noise image.

[0008] Furthermore, the portrait style enhancement feature map The generation process is as follows:

[0009] in, This indicates the number of feature maps at different scales. express index, Indicates the first i The query matrix is ​​obtained by linearly transforming the feature maps at each scale. The key matrix represents the semantic vector obtained through linear transformation. This represents the value matrix obtained by linearly transforming a semantic vector. Key matrix Dimensions Indicates the first i Feature maps at various scales express Activation function This indicates transpose.

[0010] Furthermore, the process of generating the portrait-style image is as follows:

[0011] in, This represents a portrait style enhancement feature map. express The weight value, Indicates the current time step The following portrait-style images, This indicates element-wise multiplication.

[0012] Furthermore, the self-attention module dynamically adjusts the weight values ​​of the portrait style enhancement feature map based on the difference map between the portrait style image and the spatial structure features:

[0013] in, Disparity diagram The query matrix obtained through linear transformation Disparity diagram The bond matrix obtained through linear transformation Disparity diagram The value matrix obtained through linear transformation express Dimensions express Activation function This indicates transpose.

[0014] Furthermore, the diffusion model is adjusted using LoRA technology: the weight matrix of the diffusion model is decomposed into a low-rank matrix using LoRA technology, and the low-rank matrix is ​​trained and optimized.

[0015] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that causes a computer to execute the described portrait style transfer method based on a multimodal large model.

[0016] Furthermore, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the aforementioned portrait style transfer method based on a multimodal large model.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention constructs a diffusion model based on the ControlNet architecture based on the portrait style transfer method of multimodal large model, and uses semantic vectors and spatial structure features to guide the generation of images of target portrait style, making the portrait style transfer process more flexible, while providing more refined portrait style control options, and can generate high-resolution, detailed portrait style transfer images to meet users' requirements for image quality. (2) The portrait style transfer method based on multimodal large model of the present invention uses LoRA technology to efficiently fine-tune the diffusion model, which reduces the computation and storage requirements when fine-tuning the diffusion model, enabling the present invention to realize customized portrait style transfer in a resource-limited environment and improve the feasibility of practical application.

[0018] In summary, this invention provides an efficient and precise method for portrait style transfer, which promotes the widespread application of portrait style transfer in fields such as artistic creation, digital media, and film and television production. Attached Figure Description

[0019] Figure 1 This is a flowchart of the portrait style transfer method based on a multimodal large model according to the present invention; Figure 2 This is a schematic diagram of the structure of U-Net in this invention. Detailed Implementation

[0020] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0021] like Figure 1 This is a flowchart of the portrait style transfer method based on a multimodal large model according to the present invention. The portrait style transfer method includes the following steps: Step S1: Obtain the original portrait image. To ensure that the generated target portrait style image maintains the geometric structure and key facial features of the original portrait during the stylization process, it is necessary to perform structural analysis on the original portrait image and extract the spatial structural features in the original portrait image, including: edge contours, depth distribution, or pose key points.

[0022] Step S2: Provide the target portrait style, specifically one or more of the following: a text description of the target portrait style and a reference image of the target portrait style. Encode the target portrait style into a semantic vector using a multimodal large model. Specifically, if a text description of the target portrait style is provided, such as Impressionist style, soft brushstroke style, warm color style, etc., the text description is encoded into a semantic vector using the CLIP architecture's text-image alignment model. This semantic vector carries the abstract semantic features of the style. If a reference image of the target portrait style is provided, visual style representations are extracted using an image encoder and encoded into a semantic vector. Both types of semantic vectors can serve as semantic guidance signals in the subsequent image generation process of the target portrait style.

[0023] Step S3: Construct a diffusion model based on the ControlNet architecture. Gaussian noise is gradually added to the original portrait image until it becomes a pure noise image. Semantic vectors and spatial structure features are used to guide the generation of the target portrait-style image from the pure noise image. Without changing the main structure of the diffusion model, ControlNet introduces a parallel branch network to fuse the spatial structure features and semantic vectors in the original portrait image across layers. This constrains the spatial layout of the generated portrait-style image in each denoising step, effectively preventing common problems such as facial feature distortion and disproportion.

[0024] This invention constructs a diffusion model based on the ControlNet architecture for portrait style transfer based on a multimodal large model. It uses semantic vectors and spatial structure features to guide the generation of images with the target portrait style, making the portrait style transfer process more flexible and providing more refined portrait style control options. It can generate high-resolution, detailed portrait style transfer images to meet users' requirements for image quality.

[0025] like Figure 2 In one technical solution of the present invention, the denoising network in the diffusion model based on the ControlNet architecture adopts U-Net, which includes: a downsampling module, a cross-attention module, an upsampling module, and a self-attention module.

[0026] At each time step, the downsampling module extracts multi-scale feature maps from the current portrait style image, the cross-attention module generates a portrait style enhancement feature map based on the multi-scale feature map and semantic vector, and the upsampling module generates a portrait style image based on the portrait style enhancement feature map and the current portrait style image. To enhance the structural fidelity of key facial regions such as eyes, nose, and lips during the portrait style transfer process, the self-attention module dynamically corrects the portrait style image using spatial structural features, strengthens the preservation and restoration of key region features, and suppresses structural distortion or semantic drift caused by over-stylization.

[0027] Iterate through all time steps to obtain an image in the style of the target portrait. In the first time step, the downsampling module inputs a pure noise image.

[0028] U-Net not only needs to predict the noise components in the current portrait-style image, but also dynamically adjusts the feature response based on the semantic vector to ensure that the generated content is consistent with the user's intent at the semantic level; at the same time, it guides the accurate reconstruction of the image structure by using spatial structural features in a spatial constraint manner.

[0029] Portrait style enhancement feature map in this invention The generation process is as follows:

[0030] in, This indicates the number of feature maps at different scales. express index, Indicates the first i The query matrix is ​​obtained by linearly transforming the feature maps at each scale. The key matrix represents the semantic vector obtained through linear transformation. This represents the value matrix obtained by linearly transforming a semantic vector. Key matrix Dimensions Indicates the first i Feature maps at various scales express Activation function This indicates transpose.

[0031] The process of generating portrait-style images in this invention is as follows:

[0032] in, This represents a portrait style enhancement feature map. express The weight value, Indicates the current time step The following portrait-style images, This indicates element-wise multiplication.

[0033] In one technical solution of the present invention, the self-attention module dynamically adjusts the weight value of the portrait style enhancement feature map based on the difference map between the portrait style image and the spatial structure features, and implicitly guides attention focus through structural consistency feedback, without the need for additional annotation or supervision.

[0034] The process of adjusting the weight values ​​of the portrait style enhancement feature map in this invention is as follows:

[0035] in, Disparity diagram The query matrix obtained through linear transformation Disparity diagram The bond matrix obtained through linear transformation Disparity diagram The value matrix obtained through linear transformation express Dimensions express Activation function This indicates transpose.

[0036] In one technical solution of this invention, to reduce the computational overhead of training the diffusion model and improve the customization capability of style transfer, LoRA technology is used to adjust the diffusion model: the weight matrix of the diffusion model is decomposed into a low-rank matrix using LoRA technology, and the low-rank matrix is ​​trained and optimized to achieve customized style transfer without requiring a large amount of computational resources. During the training phase, a multimodal dataset containing various art styles such as oil painting, watercolor, animation, and cyberpunk paired with portraits is constructed. This dataset is used to fine-tune the diffusion model, enabling it to have cross-style generalization ability and structure preservation ability. The fine-tuned diffusion model can quickly adapt to new styles during the inference phase without retraining the entire diffusion model.

[0037] In one technical solution of this invention, users can adjust the semantic weight of keywords in text descriptions or upload multiple style reference images for style blending, achieving fine-grained and personalized style control. This portrait style transfer method provides a graphical user interface and, combined with a GPU-accelerated inference engine, can generate high-quality 1024×1024 resolution portrait style transfer images within 30 seconds. It also supports real-time preview of intermediate results and dynamic parameter adjustment, significantly improving creative efficiency and user experience.

[0038] The portrait style transfer method based on multimodal large models in this invention significantly outperforms traditional style transfer methods based on StyleGAN and single diffusion model schemes without structural control in objective evaluation indicators such as FID and LPIPS. In particular, it performs well in terms of facial structure fidelity, style semantic consistency and detail restoration ability. The generated target portrait style transfer images achieve a good balance between visual quality and artistic expression.

[0039] In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the portrait style transfer method based on a multimodal large model of the present invention.

[0040] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the portrait style transfer method based on a multimodal large model of the present invention.

[0041] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0043] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A portrait style transfer method based on a multi-modal large model, characterized in that, The method comprises the following steps: Step S1: obtaining an original portrait image and extracting spatial structure features in the original portrait image; Step S2: providing a target portrait style, and encoding the target portrait style into a semantic vector through a multi-modal large model; Step S3: constructing a diffusion model based on a ControlNet architecture, gradually adding Gaussian noise to the original portrait image until it becomes a pure noise image, and using the semantic vector and the spatial structure features to guide the pure noise image to gradually generate an image of the target portrait style.

2. The portrait style transfer method based on a multi-modal large model according to claim 1, characterized in that, The target portrait style provided in step S1 is one or more of a text description of the target portrait style and a reference image of the target portrait style.

3. The portrait style transfer method based on a multi-modal large model according to claim 2, characterized in that, If a text description of the target portrait style is provided, the text description is encoded into a semantic vector through a text-image alignment model of a CLIP architecture; if a reference image of the target portrait style is provided, a visual style representation is extracted through an image encoder and encoded into a semantic vector.

4. The portrait style transfer method based on a multi-modal large model according to claim 1, characterized in that, The denoising network in the diffusion model based on the ControlNet architecture adopts a U-Net, which comprises a down-sampling module, a cross-attention module, an up-sampling module and a self-attention module; At each time step, the down-sampling module is used to extract multi-scale feature maps from the current portrait style image, the cross-attention module is used to generate a portrait style enhanced feature map according to the multi-scale feature maps and the semantic vector, the up-sampling module is used to generate a portrait style image according to the portrait style enhanced feature map and the current portrait style image, and the self-attention module is used to dynamically correct the portrait style image by using the spatial structure features; All time steps are traversed to obtain an image of the target portrait style; At the first time step, the down-sampling module inputs a pure noise image.

5. The portrait style transfer method based on a multi-modal large model according to claim 4, characterized in that, The portrait style enhancement feature map The generation process is: wherein, denotes the number of feature maps of different scales, denotes an index of, denotes the i query matrix of the feature maps at the key matrix of the semantic vectors by linear transformation, value matrix of the semantic vectors by linear transformation, denotes the dimension of the key matrix, denotes the feature map at the i scale, denotes an activation function, denotes a transpose.

6. The portrait style transfer method based on a multi-modal large model according to claim 4, characterized in that, The generation process of the portrait style image is as follows: wherein, represents a portrait style enhancement feature map, represents a weight value of the portrait style enhancement feature map, represents a portrait style image at a current time step under the portrait style enhancement feature map, represents an element-wise multiplication.

7. The portrait style transfer method based on a multi-modal large model according to claim 6, characterized in that, The self-attention module dynamically adjusts the weight value of the portrait style enhanced feature map according to a difference map between the portrait style image and the spatial structure features: wherein, representation difference map query matrix obtained by linear transformation, representation difference map key matrix obtained by linear transformation, representation difference map value matrix obtained by linear transformation, representation dimension of representation activation function, representation transpose.

8. The portrait style transfer method based on a multi-modal large model according to claim 1, characterized in that, The diffusion model is adjusted by using LoRA technology: the weight matrix of the diffusion model is decomposed into a low-rank matrix by using LoRA technology, and the low-rank matrix is trained and optimized.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program enables a computer to execute the portrait style transfer method based on the multi-modal large model according to any one of claims 1-8.

10. An electronic device, comprising: It comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the portrait style transfer method based on the multi-modal large model according to any one of claims 1-8.

Images