Efficient face style transfer method, device and equipment based on diffusion model

Abstract

This invention provides an efficient face style transfer method, apparatus, and device based on a diffusion model. The method includes: acquiring a face image to be stylized and a target style image; inputting the face image to be stylized and the target style image into a pre-trained style model for face stylization processing to obtain a final face style image; the pre-trained style model is constructed using a cross-attention mechanism based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompting model; the first basic style transfer model is used to acquire style feature information of the target style image; the second basic style transfer model is used to acquire facial fusion information of the face image to be stylized and the target style image. By using a pre-trained model, processing efficiency is improved; and by combining the style transfer model, diffusion model, and text prompting model for processing, the realism and naturalness of the final face style image are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an efficient face style transfer method, apparatus, and device based on a diffusion model. Background Technology

[0002] With the popularization of digital media and the advancement of technology, people's demand for image processing is growing. Among them, facial style transfer, as a key technology, aims to preserve the identity features of the original face while giving it a new artistic style. This technology has broad application prospects in many fields such as entertainment and advertising. However, existing technologies still face many challenges in achieving this goal, especially how to efficiently and accurately complete style transfer without losing facial features.

[0003] Currently, the most relevant face style transfer techniques are primarily based on Generative Adversarial Networks (GANs) and diffusion models. GANs, through a game-like process between a generator and a discriminator, can generate synthetic images that are difficult to distinguish from real images; while diffusion models recover images by progressively removing noise. Both have shown powerful capabilities in image synthesis tasks. In particular, diffusion models, such as Stable Diffusion and its improved version SDXL, have achieved significant results in terms of image generation quality and resolution. Furthermore, text inversion techniques are also used to enhance the model's understanding of specific styles by converting a set of images of an object into corresponding text representations, thereby optimizing the text embeddings to better match the generated images.

[0004] While the aforementioned techniques have shown promise in the field of face style transfer, they still suffer from several significant drawbacks. First, GAN-based methods typically require substantial amounts of labeled data for training and fine-tuning, increasing model costs and limiting their flexibility and generalization capabilities. Second, both GANs and diffusion models struggle to effectively preserve facial identity information during style transfer, resulting in generated faces that may differ significantly from the original images and fail to meet practical application requirements. Finally, existing techniques often focus only on surface-level style transfer such as color and brushstrokes, neglecting to utilize deeper semantic information within the style image, thus impacting the realism and naturalness of the final result. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention provides an efficient face style transfer method, apparatus, and device based on a diffusion model.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention provides an efficient face style transfer method based on a diffusion model, comprising:

[0008] Obtain the face image to be stylized and the target style image;

[0009] The face image to be stylized and the target style image are input into a pre-trained style model for face stylization processing to obtain the final face style image. The pre-trained style model is constructed using a cross-attention mechanism based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompt model. The first basic style transfer model is used to obtain the style feature information of the target style image. The second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0010] Optionally, the face image to be stylized and the target style image are input into a pre-trained style model for face stylization processing to obtain the final face style image, including:

[0011] The target style image is input into the first basic style transfer model to extract style information and obtain style feature information.

[0012] Noise is added to the stylized face image to obtain a noisy image;

[0013] The face image to be stylized and the target style image are input into the second basic style transfer model for facial style fusion processing to obtain facial fusion information.

[0014] Text encoding information is obtained based on the face image to be stylized, the target style image, and the text prompt model;

[0015] Style feature information and facial fusion information are fused using a cross-attention mechanism to obtain network fusion information;

[0016] The noisy image, text encoding information, and network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

[0017] Optionally, the text prompting model includes: the PaliGemma model and the CLIP text encoder.

[0018] Optionally, text encoding information is obtained based on the face image to be stylized, the target style image, and the text prompting model, including:

[0019] Based on the PaliGemma model, the text information of the face image to be stylized and the target style image are obtained respectively, and the face text information and style text information are obtained accordingly.

[0020] The CLIP text encoder is used to encode the face text information and style text information respectively, resulting in face encoding information and style encoding information.

[0021] Face encoding information and style encoding information are used together as text encoding information.

[0022] Optionally, the style feature information and facial fusion information are fused using a cross-attention mechanism to obtain network fusion information, including:

[0023] Based on the corresponding layers of the first and second basic style transfer models, a cross-attention mechanism is used to fuse style feature information and facial fusion information according to a preset fusion ratio to obtain network fusion information.

[0024] Optionally, the preset blending ratio is 2:8.

[0025] Optionally, both the first and second basic style transfer models adopt the ControlNet network architecture. The first basic style transfer model includes a CLIP image encoder, while the second basic style transfer model includes an Insight Face encoder.

[0026] Optionally, the diffusion model uses the Stable Diffusion XL model.

[0027] Secondly, the present invention provides an efficient face style transfer device based on a diffusion model, the efficient face style transfer device based on a diffusion model includes: an acquisition unit and a processing unit;

[0028] The acquisition unit is used to: acquire the face image to be stylized and the target style image;

[0029] The processing unit is used to: input the face image to be stylized and the target style image into the pre-trained style model, perform face stylization processing, and obtain the final face style image; the pre-trained style model is constructed based on the first basic style transfer model, the second basic style transfer model, the diffusion model, and the text prompt model, using a cross-attention mechanism; the first basic style transfer model is used to obtain the style feature information of the target style image; the second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0030] Thirdly, the present invention provides an efficient face style transfer device based on a diffusion model, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the efficient face style transfer device based on a diffusion model is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the efficient face style transfer method based on a diffusion model as described in the first aspect above.

[0031] This invention provides an efficient face style transfer method, apparatus, and device based on a diffusion model. The efficient face style transfer method based on a diffusion model includes: acquiring a face image to be stylized and a target style image; inputting the face image to be stylized and the target style image into a pre-trained style model for face stylization processing to obtain a final face style image; the pre-trained style model is constructed using a cross-attention mechanism based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompting model; the first basic style transfer model is used to acquire style feature information of the target style image; the second basic style transfer model is used to acquire facial fusion information of the face image to be stylized and the target style image. In this invention, by using a pre-trained model, the annotation and training processes of sample data are avoided, improving processing efficiency; by combining the style transfer model, diffusion model, and text prompting model for processing, style information can be extracted while avoiding the loss of facial identity information, resulting in a more natural face stylization effect and improving the realism and naturalness of the final face style image.

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0033] Figure 1 A flowchart illustrating an efficient face style transfer method based on a diffusion model, provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the structure of an efficient face style transfer device based on a diffusion model provided in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of an efficient face style transfer device based on a diffusion model, provided in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0037] To improve the realism and naturalness of the final face style image, this invention provides an efficient face style transfer method based on a diffusion model. Figure 1 This is a flowchart illustrating an efficient face style transfer method based on a diffusion model, provided as an embodiment of the present invention. Figure 1 As shown, it includes:

[0038] S101. Obtain the face image to be stylized and the target style image.

[0039] It should be noted that, in this embodiment of the invention, the face image to be stylized is the original face image, i.e., the real image. The target style image represents the style image that the user wants to overlay on the real image; examples of target style images include: cute style, ancient style, and European and American style.

[0040] S102. Input the face image to be stylized and the target style image into the pre-trained style model, perform face stylization processing, and obtain the final face style image.

[0041] The pre-trained style model is constructed using a cross-attention mechanism based on the first basic style transfer model, the second basic style transfer model, the diffusion model, and the text prompt model. The first basic style transfer model is used to obtain the style feature information of the target style image. The second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0042] This invention provides an efficient face style transfer method based on a diffusion model. By employing a pre-trained model, the annotation and training processes of sample data are avoided, thus improving processing efficiency. By combining the style transfer model, diffusion model, and text prompting model, style information can be extracted while avoiding the loss of facial identity information, resulting in a more natural face stylization effect and improving the realism and naturalness of the final face style image.

[0043] Optionally, S102 may specifically include:

[0044] The target style image is input into the first basic style transfer model to extract style information and obtain style feature information.

[0045] Noise is added to the stylized face image to obtain a noisy image;

[0046] The face image to be stylized and the target style image are input into the second basic style transfer model for facial style fusion processing to obtain facial fusion information.

[0047] Text encoding information is obtained based on the face image to be stylized, the target style image, and the text prompt model;

[0048] Style feature information and facial fusion information are fused using a cross-attention mechanism to obtain network fusion information;

[0049] The noisy image, text encoding information, and network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

[0050] In this embodiment of the invention, the process of obtaining facial fusion information may specifically involve inputting the face image to be stylized and the target style image into the second basic style transfer model, and performing feature concatenation processing on the output features of the two to finally obtain facial fusion information.

[0051] It should be noted that, in the embodiments of the present invention, the noise reduction processing of the face image to be stylized can specifically be to add Gaussian noise to the face image to obtain the final face style image.

[0052] Optionally, the text prompting model includes: the PaliGemma model and the CLIP text encoder.

[0053] Optionally, text encoding information is obtained based on the face image to be stylized, the target style image, and the text prompting model, including:

[0054] Based on the PaliGemma model, the text information of the face image to be stylized and the target style image are obtained respectively, and the face text information and style text information are obtained accordingly.

[0055] The CLIP text encoder is used to encode the face text information and style text information respectively, resulting in face encoding information and style encoding information.

[0056] Face encoding information and style encoding information are used together as text encoding information.

[0057] Optionally, the style feature information and facial fusion information are fused using a cross-attention mechanism to obtain network fusion information, including:

[0058] Based on the corresponding layers of the first and second basic style transfer models, a cross-attention mechanism is used to fuse style feature information and facial fusion information according to a preset fusion ratio to obtain network fusion information.

[0059] Optionally, the preset blending ratio is 2:8.

[0060] In this embodiment of the invention, the preset fusion ratio was obtained through extensive experimental verification. Specifically, fusion ratios of 1:9, 3:7, 4:6, 5:5, and 2:8 were used successively in the experimental phase. After 500 experimental verifications, it was finally found that the fusion ratio of 2:8 could obtain high-level features of style information while maintaining facial information, achieving ideal experimental results. Therefore, the fusion ratio of 2:8 was finally adopted.

[0061] Optionally, both the first and second basic style transfer models adopt the ControlNet network architecture. The first basic style transfer model includes a CLIP image encoder, while the second basic style transfer model includes an Insight Face encoder.

[0062] It should be noted that an image consists of content and low-level visual elements. If feature transfer only remains at the content level, although it may produce some effects, the final result will differ from the style image in terms of brushstrokes and materials. Therefore, it is still necessary to transfer low-level visual elements. Since low-level visual elements do not distort the facial identity information that needs to be preserved, the low-level visual elements of the style image can be completely transferred. Therefore, in this embodiment of the invention, the more general CLIP image encoder is used to encode the style image to obtain the low-level visual features of the style for the facial stylization task, making the final style feature information more complete. The Insight Face encoder can completely acquire facial feature information, making the acquired facial features complete and accurate.

[0063] Optionally, the diffusion model uses the Stable Diffusion XL model.

[0064] This invention provides an efficient face style transfer method based on a diffusion model. This method requires only one style image and one face image to complete the style transfer, eliminating the dependence on a large amount of labeled data and significantly reducing cost and time. Furthermore, by setting a second basic style transfer model, style can be extracted from the face image to be stylized and the target style image, and facial identity information can be fused, ensuring that the generated stylized face is visually consistent with the face image to be stylized, avoiding the loss of identity information. By designing a first basic style transfer model to extract coarse-grained style information from the target style image and combining it with the final facial fusion information, style and facial content can be comprehensively fused, improving the detail and quality of the generated image. Further, by combining a text prompting model, a more comprehensive feature description is provided for the face image to be stylized and the target style image, further improving the realism and style accuracy of the final face style image.

[0065] The method provided in this embodiment of the invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc., and this embodiment of the invention does not limit the application to such devices.

[0066] Based on the same inventive concept, embodiments of the present invention also provide an efficient face style transfer device based on a diffusion model. Figure 2 This is a schematic diagram of the structure of an efficient face style transfer device based on a diffusion model, provided as an embodiment of the present invention. Figure 2 As shown, it includes: an acquisition unit 201 and a processing unit 202;

[0067] The acquisition unit 201 is used to: acquire the face image to be stylized and the target style image.

[0068] The processing unit 202 is used to: input the face image to be stylized and the target style image into a pre-trained style model, perform face stylization processing, and obtain the final face style image; the pre-trained style model is constructed based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompt model, using a cross-attention mechanism; the first basic style transfer model is used to obtain the style feature information of the target style image; the second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0069] Figure 3 A schematic diagram of a high-efficiency face style transfer device based on a diffusion model provided in an embodiment of the present invention includes: a processor 310, a storage medium 320, and a bus 330. The storage medium 320 stores machine-readable instructions executable by the processor 310. When the high-efficiency face style transfer device based on the diffusion model is running, the processor 310 and the storage medium 320 communicate via the bus 330. The processor 310 executes the machine-readable instructions to perform the steps of the above-described method embodiment. Specific implementation methods and technical effects are similar and will not be repeated here.

[0070] The storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0071] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0072] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0073] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0074] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0075] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the inventive concept, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An efficient face style transfer method based on a diffusion model, characterized in that, include: Obtain the face image to be stylized and the target style image; The face image to be stylized and the target style image are input into a pre-trained style model for face stylization processing to obtain the final face style image. The pre-trained style model is constructed using a cross-attention mechanism based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompting model. The first basic style transfer model is used to obtain style feature information of the target style image; The second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image; both the first basic style transfer model and the second basic style transfer model adopt the ControlNet network architecture, the first basic style transfer model is equipped with a CLIP image encoder; the second basic style transfer model is equipped with an Insight Face encoder; The step of inputting the face image to be stylized and the target style image into a pre-trained style model for face stylization processing to obtain the final face style image includes: The target style image is input into the first basic style transfer model to extract style information and obtain the style feature information; The face image to be stylized is subjected to noise reduction processing to obtain a noisy image; The face image to be stylized and the target style image are input into the second basic style transfer model for facial style fusion processing to obtain the facial fusion information. Based on the text prompting model, the text information of the face image to be stylized and the target style image are obtained respectively, and the face text information and style text information are obtained accordingly; the face text information and style text information are encoded to obtain the face encoding information and style encoding information respectively; the face encoding information and style encoding information are used together as the text encoding information. According to the corresponding layers of the first basic style transfer model and the second basic style transfer model, a cross-attention mechanism is used to fuse the style feature information and the facial fusion information according to a preset fusion ratio to obtain network fusion information; The noisy image, the text encoding information, and the network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

2. The efficient face style transfer method based on a diffusion model according to claim 1, characterized in that, The text prompting model includes: the PaliGemma model and the CLIP text encoder; The step of obtaining text encoding information based on the face image to be stylized, the target style image, and the text prompting model includes: Based on the PaliGemma model, the text information of the face image to be stylized and the target style image are obtained respectively, and the face text information and style text information are obtained accordingly. The CLIP text encoder is used to encode the face text information and the style text information respectively, so as to obtain face encoding information and style encoding information. The face encoding information and the style encoding information are used together as the text encoding information.

3. The efficient face style transfer method based on the diffusion model according to claim 1, characterized in that, The preset fusion ratio is 2:

8.

4. The efficient face style transfer method based on the diffusion model according to claim 1, characterized in that, The diffusion model used is the Stable Diffusion XL model.

5. A highly efficient face style transfer device based on a diffusion model, characterized in that, The efficient face style transfer device based on the diffusion model includes: an acquisition unit and a processing unit; The acquisition unit is used to: acquire the face image to be stylized and the target style image; The processing unit is used to: input the face image to be stylized and the target style image into a pre-trained style model, perform face stylization processing, and obtain the final face style image; the pre-trained style model is constructed based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompting model, using a cross-attention mechanism; the first basic style transfer model is used to obtain the style feature information of the target style image; the second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image; both the first basic style transfer model and the second basic style transfer model adopt the ControlNet network architecture, the first basic style transfer model is equipped with a CLIP image encoder; the second basic style transfer model is equipped with an Insight Face encoder; The processing unit is specifically used for: The target style image is input into the first basic style transfer model to extract style information and obtain the style feature information; The face image to be stylized is subjected to noise reduction processing to obtain a noisy image; The face image to be stylized and the target style image are input into the second basic style transfer model for facial style fusion processing to obtain the facial fusion information. Based on the text prompting model, the text information of the face image to be stylized and the target style image are obtained respectively, and the face text information and style text information are obtained accordingly; the face text information and style text information are encoded to obtain the face encoding information and style encoding information respectively; the face encoding information and style encoding information are used together as the text encoding information. According to the corresponding layers of the first basic style transfer model and the second basic style transfer model, a cross-attention mechanism is used to fuse the style feature information and the facial fusion information according to a preset fusion ratio to obtain network fusion information; The noisy image, the text encoding information, and the network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

6. A high-efficiency face style transfer device based on a diffusion model, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the efficient face style transfer device based on the diffusion model is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the efficient face style transfer method based on the diffusion model as described in any one of claims 1-4.

Images