Efficient face style migration method, device and equipment based on diffusion model

Abstract

The invention provides an efficient face style migration method, device and equipment based on a diffusion model. The method comprises the following steps: acquiring a to-be-stylized face image and a target style image; inputting the to-be-stylized face image and the target style image into a pre-training style model, and performing face stylization processing to obtain a final face style image; the pre-training style model is constructed by adopting a cross attention mechanism based on a first basic style transfer model, a second basic style transfer model, a diffusion model and a text prompt model; the first basic style transfer model is used for acquiring style feature information of the target style image; the second basic style transfer model is used for acquiring face fusion information of the to-be-stylized face image and the target style image. By adopting the pre-training model, the processing efficiency is improved; the style transfer model, the diffusion model and the text prompt model are combined for processing, so that the sense of reality and the naturalness of the final face style image are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an efficient face style migration method, device and equipment based on a diffusion model. Background Art

[0002] With the popularization of digital media and the advancement of technology, people's demand for image processing is growing. Among them, face style transfer is a key technology that aims to preserve the identity characteristics of the original face while giving it a new artistic style. This technology has broad application prospects in many fields such as entertainment and advertising. However, existing technologies still face many challenges in achieving this goal, especially how to complete style transfer efficiently and accurately without losing facial features.

[0003] At present, the closest face style transfer technology is mainly based on generative adversarial networks (GANs) and diffusion models. Generative adversarial networks can generate synthetic images that are difficult to distinguish from real images through the game process between the generator and the discriminator; while diffusion models restore images by gradually removing noise, both of which have shown strong capabilities in image synthesis tasks. In particular, diffusion models, such as Stable Diffusion and its improved version SDXL, have achieved remarkable results in the quality and resolution of image generation. In addition, text inversion technology is also used to enhance the model's understanding of a specific style by converting a set of images of the object into corresponding text representations, and then optimizing the text embedding to make it more closely match the generated image.

[0004] Although the above technologies have shown certain potential in the field of face style transfer, they still have some significant problems. First, GANs-based methods usually require a large amount of annotated data for training and fine-tuning, which not only increases the cost of the model but also limits its flexibility and generalization ability. Secondly, whether it is GANs or diffusion models, it is difficult to effectively maintain the identity information of the face when performing style transfer, resulting in the generated face may be quite different from the original image, which does not meet the requirements of practical applications. Finally, existing technologies often only focus on style transfer at the surface level such as color and strokes, and lack the use of deeper semantic information in the style image, which affects the realism and naturalness of the final result. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an efficient face style transfer method, device and equipment based on a diffusion model.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides an efficient face style transfer method based on a diffusion model, comprising:

[0008] Obtain the face image to be stylized and the target style image;

[0009] The face image to be stylized and the target style image are input into the pre-trained style model, and the face stylization processing is performed to obtain the final face style image; the pre-trained style model is based on the first basic style transfer model, the second basic style transfer model, the diffusion model and the text prompt model, and is constructed using the cross-attention mechanism; the first basic style transfer model is used to obtain the style feature information of the target style image; the second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0010] Optionally, the face image to be stylized and the target style image are input into a pre-trained style model to perform face stylization processing to obtain a final face style image, including:

[0011] Input the target style image into the first basic style transfer model to extract style information and obtain style feature information;

[0012] Performing noise processing on the stylized face image to obtain a noise image;

[0013] Inputting the face image to be stylized and the target style image into the second basic style transfer model for facial style fusion processing to obtain facial fusion information;

[0014] Acquire text encoding information based on the face image to be stylized, the target style image, and the text prompt model;

[0015] The style feature information and the facial fusion information are fused using a cross-attention mechanism to obtain network fusion information;

[0016] The noisy image, text encoding information and network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

[0017] Optionally, the text prompt model includes: a PaliGemma model and a CLIP text encoder.

[0018] Optionally, obtaining text encoding information based on the face image to be stylized, the target style image, and the text prompt model includes:

[0019] Based on the PaliGemma model, the text information of the face image to be stylized and the target style image are respectively obtained, and the face text information and the style text information are correspondingly obtained;

[0020] The CLIP text encoder is used to encode the face text information and the style text information respectively, and the face encoding information and the style encoding information are obtained correspondingly;

[0021] The face encoding information and the style encoding information are used together as the text encoding information.

[0022] Optionally, the style feature information and the facial fusion information are fused using a cross attention mechanism to obtain network fusion information, including:

[0023] According to the corresponding layers of the first basic style transfer model and the second basic style transfer model, a cross attention mechanism is adopted to fuse the style feature information and the facial fusion information according to a preset fusion ratio to obtain network fusion information.

[0024] Optionally, the preset fusion ratio is 2:8.

[0025] Optionally, both the first basic style transfer model and the second basic style transfer model adopt a ControlNet network architecture, and a CLIP image encoder is provided in the first basic style transfer model; and an Insight Face encoder is provided in the second basic style transfer model.

[0026] Optionally, the diffusion model adopts a Stable Diffusion XL model.

[0027] In a second aspect, the present invention provides an efficient face style migration device based on a diffusion model, and the efficient face style migration device based on a diffusion model includes: an acquisition unit and a processing unit;

[0028] The acquisition unit is used to: acquire the face image to be stylized and the target style image;

[0029] The processing unit is used to: input the face image to be stylized and the target style image into the pre-trained style model, perform face stylization processing, and obtain the final face style image; the pre-trained style model is based on the first basic style transfer model, the second basic style transfer model, the diffusion model and the text prompt model, and is constructed using a cross-attention mechanism; the first basic style transfer model is used to obtain style feature information of the target style image; the second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image.

[0030] In a third aspect, the present invention provides an efficient facial style transfer device based on a diffusion model, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the efficient facial style transfer device based on the diffusion model is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the efficient facial style transfer method based on the diffusion model in the first aspect mentioned above.

[0031] The present invention provides an efficient face style transfer method, device and equipment based on a diffusion model. Among them, an efficient face style transfer method based on a diffusion model includes: obtaining a face image to be stylized and a target style image; inputting the face image to be stylized and the target style image into a pre-trained style model, performing face stylization processing, and obtaining a final face style image; the pre-trained style model is based on a first basic style transfer model, a second basic style transfer model, a diffusion model and a text prompt model, and is constructed using a cross attention mechanism; the first basic style transfer model is used to obtain style feature information of the target style image; the second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image. In the present invention, by adopting a pre-trained model, the labeling processing and training process of sample data are avoided, and the processing efficiency is improved; because the style transfer model, the diffusion model and the text prompt model are combined for processing, the loss of face identity information can be avoided while extracting style information, so that the generated face stylization effect is more natural, and the realism and naturalness of the final face style image are improved.

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a flow chart of an efficient face style transfer method based on a diffusion model provided by an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the structure of an efficient face style transfer device based on a diffusion model provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of the structure of an efficient face style transfer device based on a diffusion model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0037] In order to improve the realism and naturalness of the final facial style image, an embodiment of the present invention provides an efficient facial style transfer method based on a diffusion model. Figure 1 The present invention provides a flow chart of an efficient method for face style transfer based on a diffusion model. Figure 1 As shown, including:

[0038] S101, obtaining a face image to be stylized and a target style image.

[0039] It should be noted that in the embodiment of the present invention, the face image to be stylized is the original face picture, that is, the real image. The target style image represents the style image that the user wants to superimpose on the real image, such as cute style, ancient style, European and American style, etc.

[0040] S102: Input the face image to be stylized and the target style image into a pre-trained style model, perform face stylization processing, and obtain a final face style image.

[0041] The pre-trained style model is based on the first basic style transfer model, the second basic style transfer model, the diffusion model and the text prompt model, and is constructed using the cross-attention mechanism; the first basic style transfer model is used to obtain the style feature information of the target style image; the second basic style transfer model is used to obtain the facial fusion information of the face image to be stylized and the target style image.

[0042] The embodiment of the present invention provides an efficient facial style transfer method based on a diffusion model. By adopting a pre-trained model, the labeling processing and training process of sample data are avoided, thereby improving the processing efficiency. Since the style transfer model, the diffusion model and the text prompt model are combined for processing, the loss of facial identity information can be avoided while extracting style information, making the generated facial stylization effect more natural, thereby improving the realism and naturalness of the final facial style image.

[0043] Optionally, S102 may specifically include:

[0044] Input the target style image into the first basic style transfer model to extract style information and obtain style feature information;

[0045] Performing noise processing on the stylized face image to obtain a noise image;

[0046] Inputting the face image to be stylized and the target style image into the second basic style transfer model for facial style fusion processing to obtain facial fusion information;

[0047] Acquire text encoding information based on the face image to be stylized, the target style image, and the text prompt model;

[0048] The style feature information and the facial fusion information are fused using a cross-attention mechanism to obtain network fusion information;

[0049] The noisy image, text encoding information and network fusion information are input into the diffusion model, and fusion denoising is performed based on the cross-attention mechanism to obtain the final face style image.

[0050] In an embodiment of the present invention, the facial fusion information acquisition process may specifically be: inputting the face image to be stylized and the target style image into the second basic style transfer model respectively, and performing feature splicing processing on the output features of the two, and finally obtaining the facial fusion information.

[0051] It should be noted that, in the embodiment of the present invention, the noise processing is performed on the stylized face image, and specifically, Gaussian noise is added to the stylized face image to obtain the final face style image.

[0052] Optionally, the text prompt model includes: a PaliGemma model and a CLIP text encoder.

[0053] Optionally, obtaining text encoding information based on the face image to be stylized, the target style image, and the text prompt model includes:

[0054] Based on the PaliGemma model, the text information of the face image to be stylized and the target style image are respectively obtained, and the face text information and the style text information are correspondingly obtained;

[0055] The CLIP text encoder is used to encode the face text information and the style text information respectively, and the face encoding information and the style encoding information are obtained correspondingly;

[0056] The face encoding information and the style encoding information are used together as the text encoding information.

[0057] Optionally, the style feature information and the facial fusion information are fused using a cross attention mechanism to obtain network fusion information, including:

[0058] According to the corresponding layers of the first basic style transfer model and the second basic style transfer model, a cross attention mechanism is adopted to fuse the style feature information and the facial fusion information according to a preset fusion ratio to obtain network fusion information.

[0059] Optionally, the preset fusion ratio is 2:8.

[0060] In the embodiment of the present invention, the preset fusion ratio is obtained through a large number of experimental verifications. Specifically, in the experimental phase, fusion ratios of 1:9, 3:7, 4:6, 5:5 and 2:8 are used successively. After 500 experimental verifications, it is finally found that the fusion ratio of 2:8 can obtain high-level features of style information while maintaining facial information, and achieve ideal experimental results. Therefore, the fusion ratio of 2:8 is finally adopted.

[0061] Optionally, both the first basic style transfer model and the second basic style transfer model adopt a ControlNet network architecture, and a CLIP image encoder is provided in the first basic style transfer model; and an Insight Face encoder is provided in the second basic style transfer model.

[0062] It should be noted that an image is composed of content and low-level visual elements. If the feature transfer is only kept at the content level, although some effects may be produced, the final result will be different from the style image in terms of brushstrokes and materials. Therefore, it is still necessary to transfer low-level visual elements. Since low-level visual elements will not distort the facial identity information content that needs to be maintained, the low-level visual elements of the style image can be completely transferred. Therefore, in an embodiment of the present invention, a more general CLIP image encoder is used to encode the style image to obtain low-level visual features of the style for facial stylization tasks, so that the final style feature information is more complete. The Insight Face encoder can fully obtain the feature information of the face, so that the acquired facial features are complete and accurate.

[0063] Optionally, the diffusion model adopts a Stable Diffusion XL model.

[0064] The embodiment of the present invention provides an efficient facial style transfer method based on a diffusion model. The method only requires one style image and one facial image to complete the style transfer, eliminating the reliance on a large amount of annotated data and significantly reducing the cost and time. In addition, by setting a second basic style transfer model, it is possible to extract the style and fuse the facial identity information from the facial image to be stylized and the target style image, ensuring that the generated stylized face is visually consistent with the facial image to be stylized, thereby avoiding the loss of identity information. By designing a first basic style transfer model to extract coarse-grained style information from the target style image and combining it with the final facial fusion information, the style and facial content can be fully integrated to improve the details and quality of the generated image. Furthermore, in combination with a text prompt model, a more comprehensive feature description is performed on the stylized facial image and the target style image, further improving the authenticity and style accuracy of the final facial style image.

[0065] The method provided in the embodiment of the present invention can be applied to an electronic device. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc., which is not limited in the embodiment of the present invention.

[0066] Based on the same inventive concept, an embodiment of the present invention also provides an efficient facial style transfer device based on a diffusion model. Figure 2 The following is a schematic diagram of the structure of an efficient face style transfer device based on a diffusion model provided by an embodiment of the present invention. Figure 2 As shown, it includes: an acquisition unit 201 and a processing unit 202;

[0067] The acquisition unit 201 is used to acquire a face image to be stylized and a target style image.

[0068] The processing unit 202 is used to: input the face image to be stylized and the target style image into the pre-trained style model, perform face stylization processing, and obtain the final face style image; the pre-trained style model is based on the first basic style transfer model, the second basic style transfer model, the diffusion model and the text prompt model, and is constructed using a cross-attention mechanism; the first basic style transfer model is used to obtain style feature information of the target style image; the second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image.

[0069] Figure 3 A structural diagram of an efficient face style transfer device based on a diffusion model provided by an embodiment of the present invention includes: a processor 310, a storage medium 320 and a bus 330, wherein the storage medium 320 stores machine-readable instructions executable by the processor 310, and when the efficient face style transfer device based on the diffusion model is running, the processor 310 communicates with the storage medium 320 through the bus 330, and the processor 310 executes the machine-readable instructions to execute the steps of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0070] The storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the storage medium may also be at least one storage device located away from the aforementioned processor.

[0071] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0072] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0073] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0074] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes of the above disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0075] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. An efficient face style transfer method based on diffusion model, characterized in that: include: Obtain the face image to be stylized and the target style image; Inputting the face image to be stylized and the target style image into a pre-trained style model, performing face stylization processing, and obtaining a final face style image; the pre-trained style model is constructed based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompt model, and adopts a cross attention mechanism; The first basic style transfer model is used to obtain style feature information of the target style image; The second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image.

2. The efficient face style transfer method based on diffusion model according to claim 1, characterized in that: The step of inputting the face image to be stylized and the target style image into a pre-trained style model to perform face stylization processing to obtain a final face style image includes: Inputting the target style image into the first basic style transfer model to extract style information to obtain the style feature information; Performing noise processing on the face image to be stylized to obtain a noise image; Inputting the face image to be stylized and the target style image into the second basic style transfer model to perform facial style fusion processing to obtain the facial fusion information; Acquire text encoding information based on the face image to be stylized, the target style image, and the text prompt model; The style feature information and the facial fusion information are fused using a cross attention mechanism to obtain network fusion information; The noise image, the text encoding information and the network fusion information are input into the diffusion model, and fusion denoising processing is performed based on the cross attention mechanism to obtain the final face style image.

3. The efficient face style transfer method based on diffusion model according to claim 2, characterized in that: The text prompt model includes: a PaliGemma model and a CLIP text encoder.

4. The efficient face style transfer method based on diffusion model according to claim 3, characterized in that: The acquiring text encoding information based on the face image to be stylized, the target style image and the text prompt model includes: Based on the PaliGemma model, respectively obtaining text information of the face image to be stylized and the target style image, and correspondingly obtaining face text information and style text information; Using the CLIP text encoder to encode the face text information and the style text information respectively, and obtain face encoding information and style encoding information accordingly; The face encoding information and the style encoding information are used together as the text encoding information.

5. The efficient face style transfer method based on diffusion model according to claim 2, characterized in that: The cross-attention mechanism is used to fuse the style feature information and the facial fusion information to obtain network fusion information, including: According to the corresponding layers of the first basic style transfer model and the second basic style transfer model, a cross attention mechanism is adopted to fuse the style feature information and the facial fusion information according to a preset fusion ratio to obtain the network fusion information.

6. The efficient face style transfer method based on diffusion model according to claim 5, characterized in that: The preset fusion ratio is 2:

8.

7. The efficient face style transfer method based on diffusion model according to claim 1, characterized in that: The first basic style transfer model and the second basic style transfer model both adopt the ControlNet network architecture. The first basic style transfer model is provided with a CLIP image encoder; the second basic style transfer model is provided with an Insight Face encoder.

8. The efficient face style transfer method based on diffusion model according to claim 1, characterized in that: The diffusion model adopts the Stable Diffusion XL model.

9. An efficient face style transfer device based on a diffusion model, characterized in that: The efficient face style migration device based on the diffusion model includes: an acquisition unit and a processing unit; The acquisition unit is used to: acquire the face image to be stylized and the target style image; The processing unit is used to: input the face image to be stylized and the target style image into a pre-trained style model, perform face stylization processing, and obtain a final face style image; the pre-trained style model is constructed based on a first basic style transfer model, a second basic style transfer model, a diffusion model, and a text prompt model, using a cross-attention mechanism; the first basic style transfer model is used to obtain style feature information of the target style image; the second basic style transfer model is used to obtain facial fusion information of the face image to be stylized and the target style image.

10. An efficient face style migration device based on a diffusion model, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the efficient face style migration device based on a diffusion model is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the efficient face style migration method based on a diffusion model as described in any one of claims 1 to 8.

Images