Image style transition method, device, electronic apparatus and storage medium

The non-learning image style transition method using attentional editing in a pre-trained diffusion model addresses inefficiencies and overfitting in existing methods, enhancing efficiency and generalizability for high-quality image style transitions.

JP2024161345AActive Publication Date: 2024-11-19BEIJING BAIDU NETCOM SCI & TECH CO LTD

Patent Information

Application Number
JP2024098387
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-05-23
Filing Date
2024-06-19
Publication Date
2024-11-19
Estimated Expiration
2044-06-19

AI Technical Summary

Technical Problem

Existing image style transition methods using fine-tuned diffusion models are inefficient, time-consuming, and suffer from overfitting and poor generalizability due to the need for extensive training data.

Method used

A non-learning image style transition method that employs attentional editing within a pre-trained diffusion model, utilizing cross-attention features to guide image generation, allowing for high-quality transitions without additional fine-tuning.

Benefits of technology

This approach improves efficiency and generalizability of image style transitions, reducing deployment and usage costs while maintaining high-quality results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161345000001_ABST
    Figure 2024161345000001_ABST
Patent Text Reader

Abstract

To provide an image style transition method, a device, an electronic apparatus and a storage medium in the present disclosure.SOLUTION: This disclosure relates to an artificial intelligence technology field, and relates to a technical field, such as deep learning, and an AIGC (Artificial Intelligence Generated Content). An image style transition method includes: acquiring a reference image and a description text; extracting a text feature of the description text; and calculating a first cross attention feature between a first image feature and the text feature in each time step of a diffusion model on the basis of the diffusion model trained in advance, acquiring a second cross attention feature between a second image feature of the reference image and the text feature, editing the first cross attention feature on the basis of the second cross attention feature, acquiring a third cross attention feature, generating a result image feature of the time step on the basis of the third cross attention feature and the text feature, decoding the result image feature in a last time step, and generating a target image by performing operation for generating the target image.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to technical fields such as deep learning, AIGC (artificial intelligence generated content), etc., and specifically to an image style transition method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Image style transition means obtaining a new image (i.e., target image) that combines the content of the original image and the new style by changing the style of the original image (i.e., reference image) while keeping the content of the original image almost constant. For example, the original image is a photo of a dog walking down the street (i.e., photo style), and the specified new style is an anime style. By performing style transition on the original image, a new image can be obtained that depicts a dog walking down the street in an anime style.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely by its inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention

[0004] The present disclosure provides an image style transition method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, an image style transition method is provided, the method comprising: acquiring a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; extracting text features from the description text; calculating a first cross-attention feature between a first image feature and the text feature at each time step of the diffusion model based on a pre-trained diffusion model, the first image feature at the first time step is an image feature of a preset initial image, and the first image feature at each time step after the second time step is a result image feature generated at a previous time step; acquiring a second cross-attention feature between a second image feature of the reference image and the text feature; editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; generating a result image feature of the time step based on the third cross-attention feature and the text feature; and decoding the result image feature of the last time step to generate the target image.

[0006] According to an aspect of the present disclosure, an image style transition device is provided, the image style transition device including: an acquisition module configured to acquire a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; an extraction module configured to extract text features of the description text; and a generation module configured to generate the target image based on a pre-trained diffusion model, the generation module including: an attention editing unit configured to calculate a first cross-attention feature between a first image feature and the text feature at each time step of the diffusion model, the first image feature at the first time step being an image feature of a preset initial image, and the first image feature at each time step after the second time step being a result image feature generated at a previous time step; obtain a second cross-attention feature between a second image feature and the text feature of the reference image, edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, and generate a result image feature of the time step based on the third cross-attention feature and the text feature; and a decoding unit configured to decode the result image feature of the last time step to generate the target image.

[0007] According to one aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0008] According to one aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method described above.

[0009] According to one aspect of the present disclosure, there is provided a computer program product comprising computer program instructions which, when executed by a processor, implements the method described above.

[0010] According to one or more embodiments of the present disclosure, a non-learning image style transition method based on attention editing is provided, which can achieve high-quality image style transition using a general-purpose pre-trained diffusion model, does not require further training (fine-tuning) of the diffusion model using a large amount of label data, improves the efficiency of image style transition, and has good generalization.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief description of the drawings]

[0012] The drawings illustratively illustrate embodiments, constitute a part of the specification, and together with the written description of the specification, serve to explain exemplary embodiments of the embodiments. The illustrated embodiments are for illustrative purposes only and are not intended to limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] FIG. 1 is a schematic diagram illustrating an example system capable of implementing the methods described herein, according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of an image style transition method according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a schematic diagram of a video style transition process according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of an image style transition device according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, but various details in the embodiments of the present disclosure contained therein are for the purpose of aiding understanding, and therefore should be considered merely as examples. Therefore, it should be recognized that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits the description of known functions and structures.

[0014] In this disclosure, unless otherwise stated, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the contextual description.

[0015] The terms used in the description of various examples of the present disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, and unless otherwise specifically limiting the number of elements, an element may be one or more. It should be noted that the term "and / or" used in the present disclosure covers any and all possible combinations of the listed items. "Plurality" means more than one.

[0016] In the technical solution disclosed herein, the acquisition, storage and application of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and good customs.

[0017] Image style transition means changing the style of an original image while keeping the content of the original image almost constant, to obtain a new image that combines the content of the original image and the new style. Depending on the number of images to be style-transitioned, the image transition task can be further divided into a single-image style transition task and a video style transition task.

[0018] In the related art, image style transition is generally realized using a fine-tuned diffusion model. That is, first, a pre-trained diffusion model capable of generating an image from a basic text is obtained. Then, the pre-trained diffusion model is fine-tuned using a large amount of training data for the image style transition task (i.e., label data consisting of a sample reference image, a sample style description text, and a sample target image), and the fine-tuned diffusion model is used to realize image style transition. Specifically, noise is added to the reference image to be style transitioned to obtain an initial image to be input to the diffusion model. The initial image and the style description text are input to the diffusion model, and the diffusion model performs noise removal on the initial image multiple times under the condition of the style description text, to obtain a target image after style transition.

[0019] In the above-mentioned related art, a fine-tuning step of the diffusion model is required to ensure the visual effect of the target image after the transition. However, training (fine-tuning) the diffusion model takes time and is inefficient, and the style transition effect of the diffusion model depends on the distribution of the training data, so it is prone to overfitting and has poor generalizability.

[0020] In response to the above problem, the present disclosure provides a non-learning image style transition method based on attention editing, which uses the image features of a reference image and the second cross-attention features of text features to edit the first cross-attention feature calculated in the image generation process of the diffusion model, so that the information in the reference image can be continuously introduced into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model, and ensuring that the generated target image can match the content of the reference image and have a specified style.

[0021] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without the need to further train (fine-tune) the diffusion model using a large amount of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and having good generalizability.

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0023] 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0024] In an embodiment of the present disclosure, the client devices 101, 102, 103, 104, 105, 106 and the server 120 may execute one or more services or software applications that enable the execution of the image style transition method.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software-as-a-Service (SaaS) model.

[0026] In the configuration shown in FIG. 1, the server 120 may include one or more assemblies that implement the functions performed by the server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed on one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from the system 100. Thus, FIG. 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] Client devices 101, 102, 103, 104, 105, and / or 106 may provide an interface through which a user of the client device may interact with the client device. The client device may also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0028] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, in-vehicle equipment, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT® Windows®, APPLE iOS, UNIX® or similar operating systems, Linux® or similar operating systems, and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android®, and the like. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), short message service (SMS) applications, and may use a variety of communication protocols.

[0029] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.

[0031] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server applications and / or mid-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0032] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0033] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or an intelligent cloud computing server or an intelligent cloud host equipped with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, and solves the defects of the traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0034] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files and video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.

[0035] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, a general-purpose repository supported by a file system, etc.

[0036] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the application of the various methods and apparatus described in accordance with this disclosure.

[0037] According to some embodiments, the client devices 101-106 can obtain a reference image and a description text input by a user, and the description text can include a content description text describing the reference image, e.g., "a dog is walking down the street", and a style description text describing the style of a target image to be generated, e.g., "anime style". The client devices 101-106 send an image style transition request to the server 120 based on the reference image and the description text input by the user. In response to the image style transition request from the client devices 101-106, the server 120 executes an image style transition method according to an embodiment of the present disclosure to generate a target image whose content matches the reference image specified by the user and has a specified style, and returns the generated target image to the client devices 101-106.

[0038] According to some embodiments, the client devices 101-106 may also execute the image style transition method according to the embodiment of the present disclosure. Specifically, the client devices 101-106 may execute the image style transition method according to the embodiment of the present disclosure to obtain a reference image and a description text input by a user, and generate a target image whose content matches the reference image specified by the user and has a specified style based on the reference image and the description text.

[0039] 2 is a flow chart of an image style transition method 200 according to an embodiment of the present disclosure. As mentioned above, the method 200 may be performed by a client device, such as the client devices 101-106 shown in FIG 1, or a server, such as the server 120 shown in FIG 1.

[0040] As shown in FIG. 2, the method 200 includes steps S210 to S270.

[0041] In step S210, a reference image and a description text are obtained, the description text including a content description text describing the content of the reference image and a style description text describing the style of the target image to be generated.

[0042] In step S220, text features of the written text are extracted.

[0043] Steps S230 to S270 are performed based on the pre-trained diffusion model to generate a target image.

[0044] In step S230, a first cross-attention feature between the first image feature and the text feature is calculated in each time step of the diffusion model, where the first image feature in the first time step is the image feature of a preset initial image, and the first image feature in each time step after the second time step is the result image feature generated in the previous time step.

[0045] In step S240, a second cross-attention feature of the second image feature and the text feature of the reference image is obtained.

[0046] In step S250, the first cross-attention feature is edited based on the second cross-attention feature to obtain a third cross-attention feature.

[0047] In step S260, a result image feature of the time step is generated based on the third cross attention feature and the text feature.

[0048] In step S270, the resulting image features of the last time step are decoded to generate a target image, the content of which matches the content of the reference image and has the style indicated by the style description text.

[0049] According to an embodiment of the present disclosure, a non-learning image style transition method based on attention editing is provided, which uses the image feature of a reference image and the second cross-attention feature of a text feature to edit a first cross-attention feature calculated in the image generation process of a diffusion model, so that the information in the reference image can be continuously introduced into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model, and ensuring that the generated target image can be consistent with the content of the reference image and have a specified style.

[0050] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without the need to further train (fine-tune) the diffusion model using a large amount of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and having good generalizability.

[0051] Each step of the method 200 is described in detail below.

[0052] In step S210, a reference image and a description text are obtained.

[0053] The reference image may be input by a user. According to some embodiments, the user may input a single image as the reference image. According to other embodiments, the user may input a reference video, and correspondingly, the reference image may be any image frame in the reference video.

[0054] The descriptive text further includes a content descriptive text and a style descriptive text.

[0055] The content description text is used to describe the content of the reference image. According to some embodiments, the content description text may be entered by a user. For example, a user may specify a reference image and enter the content description text of the reference image: "A dog is walking down the street".

[0056] According to another embodiment, the content description text may be automatically generated from the reference image, for example, by inputting a user-specified reference image into a trained image understanding model, and obtaining the content description text of the reference image output by the image understanding model, for example, the image understanding model may be a neural network model including a large language model, an image encoder and a text decoder, etc.

[0057] The style description text is used to describe the new style into which the reference image transitions, that is, the style of the target image to be generated, such as a photo style, an anime style, a sketch style, a sumi-e style, etc. The style description text can be input by a user.

[0058] In step S220, text features of the written text are extracted.

[0059] According to some embodiments, the entire description text can be input to a trained text encoder to obtain text features of the description text output by the text encoder. The text encoder may be, for example, a Contrastive Language-Image Pretraining (CLIP) text encoder, a Bidirectional Encoder Representations from Transformers (BERT) model, a word2vec model, etc. In general, the text encoder divides the description text into multiple tokens and encodes each token to obtain a feature vector for each token. The feature vectors of each token are concatenated to obtain text features of the description text.

[0060] As can be seen, the description text includes two parts, the content description text and the style description text, and accordingly the text features of the description text also include two parts, i.e. the text features of the description text include a first text feature of the content description text and a second text feature of the style description text. The first text feature includes a feature vector of each token in the content description text. The second text feature includes a feature vector of each token in the style description text.

[0061] According to some embodiments, step S220 may include steps S221 to S223.

[0062] In step S221, the content description text is encoded to obtain a first text feature of the content description text.

[0063] In step S222, the information of the reference image is introduced into the style description text to obtain an extended style description text.

[0064] In step S223, the extended style description text is encoded to obtain second text features of the extended style description text, where the text features of the description text include the first text feature and the second text feature.

[0065] According to the above embodiment, the content description text and the style description text are encoded separately, and information on the reference image is introduced into the style description text. This makes it possible to separately control the degree of preservation of the content of the reference image and the degree of application of the new style during the style transition process, thereby making the style transition process more controllable and smoother.

[0066] According to some embodiments, in step S221, the content description text may be input to a text encoder to obtain first text features of the content description text output from the text encoder. As can be seen, the first text features include a feature vector of each token in the content description text.

[0067] According to some embodiments, in step S222, a style description identifier of a reference image can be obtained, and the style description identifier of the reference image indicates the style of the reference image. The original style description text and the style description identifier of the reference image are concatenated to obtain an extended style description text. That is, the extended style description text includes the original style description text and the style description identifier of the reference image.

[0068] According to some embodiments, the style description identifier of the reference image may be a token that already exists in the lexicon, such as "photo", "sketch", etc. In this case, the style description identifier of the reference image may be obtained by recognition using a trained style recognition model. Specifically, the reference image may be input to the style recognition model, and the style type of the reference image output by the style recognition model may be obtained. The style recognition model may be, for example, a convolutional neural network.

[0069] In response to the style description identifier being a token already present in the lexicon, step S223 may include inputting the extended style description text into a text encoder to obtain second text features of the extended style description text output by the text encoder. As can be seen, the second text features include a feature vector of each token in the extended style description text.

[0070] According to some embodiments, the style description identifier of the reference image may be a visual identifier that does not appear in a lexicon, which may be represented as [S*], for example, and since the visual identifier has not appeared in the lexicon, a feature vector of the visual identifier cannot be obtained using a text encoder.

[0071] In response to the style description identifier being a visual identifier that does not appear in the lexicon, step S223 may include steps S2231 to S2233.

[0072] In step S2231, a text encoder is used to extract a first text sub-feature of the style description text.

[0073] In step S2232, a third image feature of the reference image is extracted using an image encoder, the image encoder and the text encoder being configured to map the image and the text, respectively, into the same feature space.

[0074] In step S2233, the third image feature is taken as the second text sub-feature of the style description identifier. The second text feature of the extended style description text includes the first text sub-feature and the second text sub-feature.

[0075] According to the above embodiment, the text information and image information in the extended style description text are encoded using a cross-modal text encoder and an image encoder, respectively, so that cross-modal features can be accurately extracted, thereby accurately representing the visual style features of the reference image and improving the accuracy of style transition.

[0076] According to some embodiments, the text encoder in step S2231 may be a CLIP text encoder, and the image encoder in step S2232 may be a CLIP image encoder, which can map text and images to the same feature space and achieve a cross-modal uniform feature representation.

[0077] According to some embodiments, if the reference image is an independent image without context, in step S2232, the reference image may be input to an image encoder to obtain a third image feature output by the image encoder.

[0078] According to some embodiments, when the reference image is any image frame in the reference video, in step S2232, an image encoder can be used to extract image features of one or more image frames in the reference video, and the image features can be used as the third image feature of the reference image. For example, the first image frame of the reference video can be input to an image encoder, and the image encoder can be used to obtain the image features of the output image frame. When performing style transition using each image frame in the reference video as a reference image, the image features of the first image frame in the reference video are used as the third image feature of the reference image.

[0079] According to the above-mentioned embodiment, each image frame in the same reference video can multiplex the same third image feature, which is advantageous to avoid duplicated calculation of the third image feature and improve the consistency of style transition of each image frame in the video style transition task.

[0080] An embodiment of the present disclosure utilizes a pre-trained diffusion model to generate a target image whose content matches the content of the reference image and has a style indicated by the style description text.

[0081] In the embodiments of the present disclosure, the pre-trained diffusion model has the basic text-to-image generation capability but is not fine-tuned for the image style transfer task.

[0082] The pre-trained diffusion model performs multiple denoising operations (i.e., de-diffusion operations) on a preset initial image conditioned on the text features to finally obtain a target image. Each denoising operation of the diffusion model corresponds to one time step.

[0083] The pre-trained diffusion model includes a cross-attention layer, and the image generation process of the diffusion model is as follows: At each time step t (t=T, T-1, T-2, ..., 2, 1, where T is a preset integer greater than 1, such as 50, 100, etc.), the cross-attention layer firstly generates the first image feature I t Using the text features Text of the description text as input, we extract the first image feature I t The first cross attention feature between and the text feature Text

number

[0084] For the first time step t=T, the first image feature IT is an image characteristic of the initial image. The initial image may be, for example, a random noise image or an image obtained by adding noise to a reference image. For each time step t after the second, the first image feature I t is the result image feature O generated in the previous time step (t+1). t+1 It is.

[0085] The embodiments of the present disclosure introduce an attention editing mechanism based on the pre-trained diffusion model, which realizes high-quality image style transition without further fine-tuning the diffusion model. Specifically, the second cross attention feature M t The first cross attention feature M t * By editing the reference image, the information in the reference image can be continuously introduced into the image generation process of the diffusion model, so as to effectively utilize the information in the reference image to guide the image generation of the diffusion model, and ensure that the generated target image can match the content of the reference image and have a specified style.

[0086] Steps S230 to S270 describe the process of generating a target image after introducing the attention editing mechanism into the pre-trained diffuse generative model.

[0087] In step S230, at each time step t (t=T, T−1, T−2, ..., 2, 1, where T is a preset integer greater than 1, e.g., 50, 100, etc.) of the diffusion model, a first image feature I t The first intersection attention feature M t * Calculate the first image feature I at the first time step t=T. Tis a preset image characteristic of an initial image. The initial image may be, for example, a random noise image or an image obtained by adding noise to a reference image. The image feature of the initial image can be extracted using an image encoder (for example, a CLIP image encoder). The first image feature I t is the result image feature O generated at the previous time step t+1. t+1 It is.

[0088] As mentioned above, the diffusion model includes a cross-attention layer, which is a layer that transforms the query matrix W Q , Key transformation matrix W K and the value transformation matrix W V It contains three parameters:

[0089] The cross-attention layer is the first image feature I t The input is the text feature Text of the description text. The query transformation matrix W Q Using the first image feature I t To obtain the query matrix Q t =I t ·W Q We obtain the key transformation matrix W K and the value transformation matrix W V We linearly transform the text feature Text using t =Text·W K and the value matrix V t =Text·W V The first cross attention feature M t * Calculate:

number

number

[0090] According to some embodiments, the diffusion model may also include a self-attention layer. The output terminal of the self-attention layer may be connected to the input terminal of the cross-attention layer. Correspondingly, step S230 may include steps S231 to S233.

[0091] In step S231, a self-attention feature of the first image feature is calculated.

[0092] In step S232, a fourth image feature is generated based on the self-attention feature and the first image feature.

[0093] In step S233, a first cross attention feature between the fourth image feature and the text feature is calculated.

[0094] According to the above embodiment, by realizing information aggregation within the first image feature by the self-attention mechanism, the correlation between pixel points can be captured, and the aggregated first image feature (i.e., the fourth image feature) can more accurately represent the information of the generated image. By calculating the first cross-attention feature using the fourth image feature, the first cross-attention feature can accurately represent the information of the generated image, thereby improving the quality of the generated target image.

[0095] In step S231, the first image feature I t Self-attention feature M s,t Specifically, the self-attention layer computes the query transformation matrix

number

number

number

number

number

number

number

number

[0096] In step S232, the self-attention feature M s,t The first image feature I t The value matrix V calculated by s,t The fourth image feature I s,t That is, the updated first image feature can be obtained. That is, in this embodiment, the fourth image feature I s,t is calculated by the following equation (3).

number

[0097] In step S2321, the self-attention feature M s,t Based on the historical self-attention feature corresponding to the self-attention feature M s,t Then, the adjusted self-attention feature M s,t The historical self-attention feature is obtained by performing style transition on the historical image frames of the reference image using a diffusion model. The position of the self-attention feature M s,t It is the same attention feature as

[0098] In step S2322, the adjusted self-attention feature M s,t ' and the first image feature I t Based on the fourth image feature I s,t The fourth image feature I s,t can be calculated using the following formula (4).

number

[0099] With regard to step S2321 above, it should be understood that each historical image frame corresponds to one historical self-attention feature. When there are multiple historical image frames, multiple historical self-attention features can be obtained.

[0100] According to some embodiments, the self-attention feature M s,t and the self-attention feature M s,t ' can be used.

[0101] According to another embodiment, the self-attention feature M s,t The adjusted self-attention feature M s,t The weight of each historical self-attention feature can be negatively correlated to the distance from the corresponding historical image frame to the reference image, i.e., the closer (smaller) the historical image frame is to the reference image, the larger the weight of the historical self-attention feature corresponding to the historical image frame.

[0102] In step S233, the fourth image feature I s,t and the text feature Text, and calculate the first attention cross feature M t * Specifically, the fourth image feature I s,t The cross attention feature of Text is the first image feature I mentioned above. t The calculation method is the same as the calculation method of the cross attention feature between the first image feature I t The fourth image feature I s,t All we need to do is replace

[0103] In step S240, a second cross-attention feature of the second image feature and the text feature of the reference image is obtained.

[0104] The second image feature F of the reference image can be extracted using an image encoder (eg, a CLIP image encoder).

[0105] The second intersection attention feature M between the second image feature F of the reference image and the text feature Text t can be obtained by using the cross-attention layer of the diffusion model. Specifically, the query transformation matrix W Q The second image feature F is linearly transformed using the query matrix Q = F W Q We obtain the key transformation matrix W K and the value transformation matrix W V The text feature Text is linearly transformed using the key matrix K = Text W K and the value matrix V = Text W V As can be seen, here K is the K t V is the same as above. t The second cross attention feature M t Calculate:

number

number

[0106] In step S250, the first cross attention feature is edited based on the second cross attention feature to obtain a third cross attention feature, it should be understood that the third cross attention feature is the edited first cross attention feature.

[0107] As mentioned above, the text feature of the description text includes two parts: the first text feature of the content description text and the second text feature of the style description text. Therefore, the first cross-attention feature, the second cross-attention feature, and the third cross-attention feature can each be divided into two sub-features, one sub-feature corresponding to the content description text and the other sub-feature corresponding to the style description text. Specifically, the first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text. The second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text. The third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text.

[0108] According to the above-mentioned sub-feature splitting embodiment, step S250 may further include steps S251 and S252.

[0109] In step S251, the first content sub-feature is modified based on the second content sub-feature to obtain a third content sub-feature.

[0110] In step S252, a third style sub-characteristic is determined based on the first style sub-characteristic.

[0111] According to the above embodiment, the content sub-features and the style sub-features are edited separately to avoid the reference image mainly affecting the content of the target image and excessively affecting the application of the new style.

[0112] The content of the target image depends on the content of the reference image. According to some embodiments, in step S251, the first content sub-feature can be replaced by the product of the second content sub-feature and the first factor. That is, the third content sub-feature is the product of the second content sub-feature and the first factor. The first factor indicates the degree of match between the content of the target image and the content of the reference image, i.e., the degree of retention of the reference image content.

[0113] According to some embodiments, the first factor is a positive number, and thus the value of the first factor has a positive correlation with the degree of retention of the reference image content. By adjusting the value of the first factor, the degree of retention of the reference image content in the style transition process can be controlled. Specifically, the larger the value of the first factor, the greater the degree of retention of the reference image content, and the higher the matching degree between the target image content and the reference image content. The smaller the value of the first factor, the smaller the degree of retention of the reference image content, and the lower the matching degree between the target image content and the reference image content.

[0114] According to some embodiments, in step S251, a weighted sum of the first content sub-feature and the second content sub-feature can be a third content sub-feature. The weight of the second content sub-feature can indicate the matching degree between the target image content and the reference image content, i.e., the retention degree of the reference image content. By adjusting the weight of the second content sub-feature, the retention degree of the reference image content during the style transition process can be controlled.

[0115] The style of the target image depends on the style description text and is less influenced by the reference image. Therefore, according to some embodiments, in step S252, the third style sub-feature can be determined based only on the first style sub-feature. It should be noted that, if the information of the reference image is introduced into the style description text in the above step S222, the extended style description text also contains the information of the reference image, so even if the third style sub-feature is determined based only on the first style sub-feature, the style of the target image is still guided by the reference image information, and the style transition of the target image is smooth with respect to the reference image, and does not cause too abrupt style change.

[0116] According to some embodiments, in step S252, a third style sub-feature may be a product of the first style sub-feature and a second factor, the second factor indicating the degree of application of the new style (i.e., the style indicated by the style description text).

[0117] According to some embodiments, the second factor is a positive number, and thus the value of the second factor has a positive correlation with the degree of application of the new style. By adjusting the value of the second factor, the degree of application of the new style in the style transition process can be controlled. Specifically, the larger the value of the second factor, the greater the degree of application of the new style. The smaller the value of the second factor, the less the degree of application of the new style.

[0118] According to some embodiments, the attention editing process of step S250 can be expressed as follows:

number

number

[0119] In step S260, a result image feature of the time step is generated based on the third cross attention feature and the text feature.

[0120] According to some embodiments, in step S260, a third attention feature M t ** Let V be the value matrix calculated from the text features. t The resulting image feature O is multiplied with t In this embodiment, the result image feature O t is calculated using the following equation (7).

number

number

[0121] In step S270, the resulting image features of the final time step are decoded to generate a target image whose content matches the content of the reference image and has the style indicated by the style description text.

[0122] According to some embodiments, the pre-trained diffusion model can include a decoder, which can decode the result image features of the last time step to obtain a target image after the style transition, whose content matches the content of the reference image and has the style indicated by the style description text.

[0123] 3 is a schematic diagram of a video style transition process according to an embodiment of the present disclosure. The video style transition process is realized by a video style transition service. As shown in FIG. 3, the video style transition service includes a text feature mapping module 310, a note editing module 320, and a video generation module 330.

[0124] The text feature mapping module 310 receives a description text D="X,Y" given by a user, where X is a content description text, which is used to describe the content in the reference video V, and Y is a style description text, which is used to describe the style of the target video to be generated. As shown in Fig. 3, the content description text X="There is a dog that is walking on the street" and the style description text Y="anime style".

[0125] In order to retain more content characteristics of the reference video V, we introduce the content information of the reference video V into the description text D and update the description text D to D*="X,Y*", where Y*="Y of [S*]"="anime style of [S*]", [S*] is the style description identifier of the reference video V.

[0126] The text feature mapping module 310 uses the pre-trained CLIP model to encode the updated description text D* to obtain text features of D*. Specifically, for natural language tokens in D*, i.e., the “anime style of” parts in X and Y*, the text features are extracted using the text encoder 312 in the CLIP model. For the style description identifier [S*], which is a token that does not appear in the lexicon, the visual encoder (i.e., image encoder) 314 in the CLIP model is used to extract visual features of the first frame of the reference video V. Since the CLIP model can map visual features and text features to the same feature space, the feature is taken as the text feature corresponding to the style description identifier [S*]. Then, the two partial features are stitched together to obtain the complete text feature of the description text D*.

[0127] The attention editing module 320 takes the text features of the description text D* and the reference video V as input, edits the cross-attention mechanism between the reference video V and the description text to obtain new attention features in the video generation process during style transition, and introduces the new attention features into the inference process of the pre-trained basic model, i.e., the Stable Diffusion model 334.

[0128] At the t-th time step of the inference process, the cross attention feature M t * , and calculate the cross attention feature M t Get the.

number

[0129] In the process of generating the video, the attention editing module 320 generates cross attention features M t * When computing, the feature encoding of the reference video V is replaced by the cross attention feature between the text features of X and M t * Generate the feature encoding of the video frames and the attention feature portion of the X text features, i.e.:

number

[0130] The video generation module 330 generates a post-style transition video V' based on the stable diffusion model 334. The video generation module 330 encodes the reference video V using an encoder 332 to obtain a feature encoding f1 of the reference video V. The encoder 332 may be, for example, a CLIP visual encoder. A feature encoding f2 is obtained by adding noise (e.g., random noise conforming to a Gaussian distribution) to the feature encoding f1. The feature encoding f2 is taken as the initial image feature of the stable diffusion model 334, i.e., the starting point of the de-diffusion operation.

[0131] In the process of generating each video frame, the edited cross-attention feature is used to replace the original cross-attention feature calculated in the stable diffusion model 334. At the same time, the self-attention feature of the historical video frame is used to replace the original self-attention feature calculated in the stable diffusion model 334 to build the association of each video frame, so that the generated video is better represented in timing consistency.

[0132] The video style transition process shown in FIG.

[0133] 1. It does not require training models using large amounts of data, reducing the costs of deploying and using the service.

[0134] 2. Users can customize the stylization levels α and β.

[0135] 3. The generated style transition videos have excellent timing consistency.

[0136] According to an embodiment of the present disclosure, an image style transition device is also provided. Fig. 4 is a block diagram showing a configuration of an image style transition device 400 according to an embodiment of the present disclosure. As shown in Fig. 4, the device 400 includes an acquisition module 410, an extraction module 420, and a generation module 430.

[0137] The acquiring module 410 is configured to acquire a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated.

[0138] The extraction module 420 is configured to extract text features of the written text.

[0139] The generation module 430 is configured to generate the target image based on a pre-trained diffusion model. The generation module 430 further includes an attention editing unit 432 and a decoding unit 434.

[0140] The attention editing unit 432 is configured to calculate a first cross-attention feature between a first image feature and the text feature in each time step of the diffusion model, where the first image feature in the first time step is an image feature of a preset initial image, and the first image feature in each time step from the second time step onwards is a result image feature generated in the previous time step, obtain a second cross-attention feature between a second image feature of the reference image and the text feature, edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, and generate a result image feature of the time step based on the third cross-attention feature and the text feature.

[0141] A decoding unit 434 is configured to decode the resulting image features of the last time step to generate the target image.

[0142] According to an embodiment of the present disclosure, a non-learning image style transition device based on attention editing is provided, which edits a first cross-attention feature calculated in the image generation process of a diffusion model using the image feature of a reference image and a second cross-attention feature of a text feature, so that the information in the reference image can be continuously introduced into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model, and ensuring that the generated target image can be consistent with the content of the reference image and have a specified style.

[0143] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without the need to further train (fine-tune) the diffusion model using a large amount of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and having good generalizability.

[0144] According to some embodiments, the first cross attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, the third cross attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and the attention editing unit includes a content editing subunit configured to modify the first content sub-feature based on the second content sub-feature to obtain the third content sub-feature, and a style editing subunit configured to determine the third style sub-feature based on the first style sub-feature.

[0145] According to some embodiments, the content editing subunit is further configured to replace the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating a degree of match between the content of the target image and the content of the reference image.

[0146] According to some embodiments, the style editing subunit is further configured to determine the third style sub-feature as a product of the first style sub-feature and a second factor, the second factor indicating the degree of application of the style.

[0147] According to some embodiments, the extraction module includes a first encoding unit configured to encode the content description text to obtain first text features of the content description text, an introduction unit configured to introduce information of the reference image into the style description text to obtain an extended style description text, and a second encoding unit configured to encode the extended style description text to obtain second text features of the extended style description text, where the text features include the first text features and the second text features.

[0148] According to some embodiments, the extended style description text comprises the style description text and a style description identifier of the reference image, and the second encoding unit comprises a first encoding subunit configured to extract a first text sub-feature of the style description text using a text encoder, and a second encoding subunit configured to extract a third image feature of the reference image using an image encoder, the image encoder and the text encoder being configured to map image and text respectively into the same feature space, and a determination subunit configured to determine the third image feature as a second text sub-feature of the style description identifier, the second text feature comprising the first text sub-feature and the second text sub-feature.

[0149] According to some embodiments, the reference image is any image frame in a reference video, wherein the second encoding subunit is further configured to utilize the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

[0150] According to some embodiments, the attention editing unit includes a first calculation subunit configured to calculate a self-attention feature of the first image feature, a generation subunit configured to generate a fourth image feature based on the self-attention feature and the first image feature, and a second calculation subunit configured to calculate a first cross-attention feature between the fourth image feature and the text feature.

[0151] According to some embodiments, the reference image is any image frame other than a first image frame in a reference video, where the generation subunit is further configured to adjust the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, where the historical self-attention feature is an attention feature having a position the same as that of the self-attention feature, obtained by style transitioning historical image frames of the reference image using the diffusion model, and generate the fourth image feature based on the adjusted self-attention feature and the first image feature.

[0152] It should be understood that each module and unit of the apparatus 400 shown in Figure 4 may correspond to each step of the method 200 described with reference to Figure 2. Thus, the operations, features, and advantages described above with respect to the method 200 are equally applicable to the apparatus 400 and the modules and units included therein. For the sake of brevity, some operations, features, and advantages will not be described here.

[0153] Although particular functionality is discussed above with reference to particular modules, it should be noted that the functionality of each module discussed herein may be split into multiple modules and / or at least some of the functionality of multiple modules may be combined into a single module.

[0154] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Each unit described with respect to FIG. 4 may be realized in hardware or in hardware combined with software and / or firmware. For example, these units may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of the modules 410-430 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (e.g., a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) that may optionally execute received program code and / or include embedded firmware to perform functions.

[0155] According to an embodiment of the present disclosure, there is further provided an electronic device, the electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, enabling the at least one processor to perform an image style transition method according to an embodiment of the present disclosure.

[0156] According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform an image style transition method according to an embodiment of the present disclosure.

[0157] According to an embodiment of the present disclosure, there is also provided a computer program product including computer program instructions which, when executed by a processor, implements an image style transition method according to an embodiment of the present disclosure.

[0158] Next, referring to FIG. 5, a configuration block diagram of an electronic device 500 functioning as a server or client of the present disclosure will be described, which is an example of a hardware device applicable to each aspect of the present disclosure. The electronic device represents various forms of digital electronic computers, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0159] 5, the electronic device 500 includes a computing unit 501 that can perform various appropriate operations and processes according to a computer program stored in a read only memory (ROM) 502 or loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0160] A number of components in the electronic device 500, including an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509, are connected to the I / O interface 505. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input numeric or character information and generate key signal inputs for user settings and / or function control of the electronic device, including but not limited to a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 507 may be any type of device capable of presenting information, including but not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include but is not limited to a magnetic disk, an optical disk. The communications unit 509 enables the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver, and / or a chipset (e.g., a Bluetooth® device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communications device, and / or the like).

[0161] The computing unit 501 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 executes each of the methods and processes described above, such as method 200. For example, in some embodiments, the method 200 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above can be executed. Alternatively, in other embodiments, the computing unit 501 is configured to perform these methods 200 in any other suitable manner (eg, by firmware).

[0162] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0163] The program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, such that when executed by the processor or controller, the program code performs the functions / operations specified in the flow charts and / or block diagrams. The program code may be entirely executed on a machine, partly executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0164] In the context of this disclosure, a machine-readable medium may be a tangible medium, which may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above contents. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above contents.

[0165] To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and may receive input from the user in any form (including sound input, speech input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system including a backstage component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage component, middleware component, or front-end component. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0167] The computer system may include a client side and a server. The client side and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client side and the server is generated by running computer programs having a client side-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0168] It should be understood that the steps may be reordered, added, or removed using the various forms of flow described above. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, and the present disclosure is not limited thereto as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0169] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements of the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. 1. A method for image style transition, comprising: obtaining a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; extracting text features of the written text; Based on a pre-trained diffusion model, At each time step of the diffusion model, Calculate a first cross-attention feature between a first image feature and the text feature, the first image feature in a first time step is an image feature of a preset initial image, and the first image feature in each time step after the second time step is a result image feature generated in the previous time step; Obtain a second cross attention feature between a second image feature of the reference image and the text feature; Edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; Generate a result image feature for the time step based on the third cross attention feature and the text feature; and performing the operations of decoding the resulting image features of the last time step to generate a target image, thereby generating the target image.

2. the first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, and the third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, modifying the first content sub-feature based on the second content sub-feature to obtain the third content sub-feature; and determining the third style sub-characteristic based on the first style sub-characteristic.

3. Modifying the first content sub-feature based on the second content sub-feature includes: The method of claim 2 , further comprising replacing the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating a degree of match between the content of the target image and the content of the reference image.

4. determining the third style sub-characteristic based on the first style sub-characteristic, The method of claim 2 , further comprising: determining the third style sub-feature as a product of the first style sub-feature and a second factor, the second factor indicating a degree of application of the style.

5. Extracting text features of the written text includes: encoding the content description text to obtain first text features of the content description text; introducing information of the reference image into the style description text to obtain an extended style description text; and encoding the expanded style description text to obtain second text features of the expanded style description text; The method of claim 1 , wherein the text features include the first text feature and the second text feature.

6. The extended style description text includes the style description text and a style description identifier of the reference image, and encoding the extended style description text to obtain second text features of the extended style description text includes: extracting first text sub-features of said style description text utilizing a text encoder; extracting a third image feature of the reference image using an image encoder, the image encoder and the text encoder configured to map images and text, respectively, into the same feature space; and wherein the third image feature is a second text sub-feature of the style description identifier; The method of claim 5 , wherein the second text feature comprises the first text sub-feature and the second text sub-feature.

7. The reference image is any image frame in a reference video, and wherein extracting a third image feature of the reference image utilizing the image encoder includes: The method of claim 6 , comprising utilizing the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

8. Calculating a first cross attention feature of the first image feature and the text feature includes: Computing a self-attention feature of the first image feature; generating a fourth image feature based on the self-attention feature and the first image feature; and computing a first cross-attention feature of the fourth image feature and the text feature.

9. the reference image is any image frame other than a first image frame in a reference video, and wherein generating a fourth image feature based on the self-attention feature and the first image feature includes: Adjust the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, where the historical self-attention feature is an attention feature obtained by performing style transition on a historical image frame of the reference image using the diffusion model, and has a position identical to that of the self-attention feature; and generating the fourth image feature based on the adjusted self-attention feature and the first image feature.

10. An image style transition device, comprising: an acquisition module configured to acquire a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; an extraction module configured to extract text features of the written text; and a generation module configured to generate the target image based on a pre-trained diffusion model, the generation module comprising: At each time step of the diffusion model, Calculate a first cross-attention feature between a first image feature and the text feature, the first image feature in a first time step is an image feature of a preset initial image, and the first image feature in each time step after the second time step is a result image feature generated in the previous time step; Obtain a second cross attention feature between a second image feature of the reference image and the text feature; Edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; an attention editing unit configured to generate a result image feature of the time step based on the third cross attention feature and the text feature; and a decoding unit configured to decode the resulting image features of the last time step to generate the target image.

11. The first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, the third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and the attention editing unit: a content editing subunit configured to modify the first content sub-characteristic based on the second content sub-characteristic to obtain the third content sub-characteristic; and a style editing subunit configured to determine the third style sub-characteristic based on the first style sub-characteristic.

12. The content editing subunit further comprises: The apparatus of claim 11 , configured to replace the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating a degree of match between the content of the target image and the content of the reference image.

13. The style editing subunit further comprises: The apparatus of claim 11 , further configured to define the third style sub-feature as a product of the first style sub-feature and a second factor, the second factor indicating a degree of application of the style.

14. The extraction module includes: a first encoding unit configured to encode the content description text to obtain first text features of the content description text; an introduction unit configured to introduce information of the reference image into the style description text to obtain an extended style description text; a second encoding unit configured to encode the extended style description text to obtain second text features of the extended style description text, The apparatus of claim 10 , wherein the text features include the first text feature and the second text feature.

15. The extended style description text includes the style description text and a style description identifier of the reference image, and the second encoding unit: a first encoding subunit configured to extract first text sub-features of said style description text using a text encoder; a second encoding subunit configured to extract a third image feature of the reference image using an image encoder, the image encoder and the text encoder configured to map the image and the text, respectively, into the same feature space; a determining subunit configured to determine the third image feature as a second text sub-feature of the style description identifier; The apparatus of claim 14 , wherein the second text feature comprises the first text sub-feature and the second text sub-feature.

16. The reference image is any image frame in a reference video, where the second encoding sub-unit further comprises: The apparatus of claim 15 , configured to utilize the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

17. The attention editing unit includes: a first computation subunit configured to compute a self-attention feature of the first image feature; a generating subunit configured to generate a fourth image feature based on the self-attention feature and the first image feature; and a second computation subunit configured to compute a first cross-attention feature of the fourth image feature and the text feature.

18. The reference image is any image frame other than a first image frame in a reference video, where the generating subunit further comprises: Adjusting the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, where the historical self-attention feature is an attention feature obtained by style transition of a historical image frame of the reference image using the diffusion model, the position of which is the same as that of the self-attention feature; The apparatus of claim 17 , configured to generate the fourth image feature based on the adjusted self-attention feature and the first image feature.

19. An electronic device, At least one processor; a memory communicatively coupled to the at least one processor; An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 9.

21. A computer program product comprising computer program instructions which, when executed by a processor, implements the method of any one of claims 1 to 9.

Citation Information

Patent Citations

  • Method of changing character part with image, computer equipment, and computer program

    JP2022107580A

  • Model training method and library creation method, device, equipment, and storage medium

    JP2023039889A

  • Three-dimensional medical image recognition method and apparatus, and device, storage medium and product

    WO2023160157A1

  • Generating images using sequences of generative neural networks

    WO2023225344A1

Cited By

  • Diffusion model-based camouflage target generation method and related device

    CN119919522A

  • Multi-condition guide text image generation method based on decoupling and multi-domain guide strategy

    CN120047565A