Image style transition method, device, electronic device and storage medium

Abstract

To provide an image style transition method, a device, an electronic apparatus and a storage medium in the present disclosure.SOLUTION: This disclosure relates to an artificial intelligence technology field, and relates to a technical field, such as deep learning, and an AIGC (Artificial Intelligence Generated Content). An image style transition method includes: acquiring a reference image and a description text; extracting a text feature of the description text; and calculating a first cross attention feature between a first image feature and the text feature in each time step of a diffusion model on the basis of the diffusion model trained in advance, acquiring a second cross attention feature between a second image feature of the reference image and the text feature, editing the first cross attention feature on the basis of the second cross attention feature, acquiring a third cross attention feature, generating a result image feature of the time step on the basis of the third cross attention feature and the text feature, decoding the result image feature in a last time step, and generating a target image by performing operation for generating the target image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, AIGC (artificial intelligence generated content), etc., and specifically to an image style transition method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Image style transition means keeping the content of an original image (i.e., a reference image) nearly constant while changing the style of the original image to obtain a new image (i.e., a target image) that combines the content of the original image and the new style. For example, the original image is a photo of a dog walking down the street (i.e., a photo style), and the specified new style is an anime style. By style transitioning the original image, a new image depicting a dog walking down the street in an anime style can be obtained.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely by virtue of its inclusion in this section. Similarly, unless otherwise noted, any problem addressed in this section should not be considered an admission of prior art. Summary of the Invention

[0004] The present disclosure provides an image style transition method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, there is provided an image style transition method, the method including: acquiring a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; extracting text features from the description text; calculating, based on a pre-trained diffusion model, a first cross-attention feature between a first image feature and the text feature at each time step of the diffusion model, the first image feature at the first time step being an image feature of a preset initial image, and the first image feature at each time step after the second time step being a result image feature generated at the previous time step; acquiring a second cross-attention feature between a second image feature of the reference image and the text feature; editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; generating a result image feature for the time step based on the third cross-attention feature and the text feature; and decoding the result image feature for the final time step to generate the target image.

[0006] According to an aspect of the present disclosure, there is provided an image style transition device, the image style transition device including: an acquisition module configured to acquire a reference image and a description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; an extraction module configured to extract text features of the description text; and a generation module configured to generate the target image based on a pre-trained diffusion model, the generation module including: an attention editing unit configured to: calculate, at each time step of the diffusion model, a first cross-attention feature between a first image feature and the text feature, the first image feature at the first time step being an image feature of a predetermined initial image, and the first image feature at each time step after the second time step being a result image feature generated at a previous time step; obtain a second cross-attention feature between a second image feature of the reference image and the text feature, edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, and generate a result image feature for the time step based on the third cross-attention feature and the text feature; and a decoding unit configured to decode the result image feature of the last time step to generate the target image.

[0007] According to one aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0008] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method described above.

[0009] According to one aspect of the present disclosure, there is provided a computer program product comprising computer program instructions which, when executed by a processor, implements the method described above.

[0010] According to one or more embodiments of the present disclosure, a non-learning image style transition method based on attentional editing is provided, which can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, does not require further training (fine-tuning) of the diffusion model using a large amount of label data, improves the efficiency of image style transition, and has good generalization.

[0011] It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of protection of the present disclosure. Other features of the present disclosure will be easily understood from the following description. [Brief explanation of the drawings]

[0012] The drawings illustrate exemplary embodiments, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary system capable of implementing the methods described herein, according to an embodiment of the present disclosure. [Figure 2] 1 is a flowchart of an image style transition method according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram of a video style transition process according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of an image style transition device according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram illustrating an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0033] Exemplary embodiments of the present disclosure will be described below with reference to the drawings. However, various details of the embodiments of the present disclosure included therein are intended to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description omits descriptions of known functions and structures.

[0014] In this disclosure, unless otherwise specified, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of an element, or in some cases, may refer to different instances based on the context.

[0015] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, and unless specifically limiting the number of elements, an element may be one or more. Note that, as used in this disclosure, the term "and / or" covers any and all possible combinations of the listed items. "Plurality" means more than one.

[0016] In the technical solution disclosed herein, the acquisition, storage and application of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0017] Image style transition means changing the style of an original image while keeping its content roughly constant, resulting in a new image that combines the content of the original image with the new style. Depending on the number of images to be style-transitioned, the image transition task can be further divided into a single-image style transition task and a video style transition task.

[0018] Related technologies generally achieve image style transition using a fine-tuned diffusion model. That is, first, a pre-trained diffusion model capable of generating images from basic text is obtained. Next, the pre-trained diffusion model is fine-tuned using a large amount of training data for the image style transition task (i.e., label data consisting of sample reference images, sample style description text, and sample target images), and the fine-tuned diffusion model is used to achieve image style transition. Specifically, noise is added to the reference image to be style transitioned to obtain an initial image to be input to the diffusion model. The initial image and the style description text are input to the diffusion model, and the diffusion model performs noise removal on the initial image multiple times based on the style description text to obtain a target image after style transition.

[0019] The above-mentioned related techniques require a fine-tuning step for the diffusion model to ensure the visual effect of the target image after the transition. However, training (fine-tuning) the diffusion model takes time and is inefficient. In addition, the style transition effect of the diffusion model depends on the distribution of training data, which makes it prone to overfitting and poor generalization.

[0020] To address the above problem, the present disclosure provides a non-learning image style transition method based on attention editing. By editing the first cross-attention feature calculated in the image generation process of the diffusion model using the image features of the reference image and the second cross-attention feature of the text feature, the information in the reference image can be continuously introduced into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model and ensuring that the generated target image is consistent with the content of the reference image and has a specified style.

[0021] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without requiring further training (fine-tuning) of the diffusion model using large amounts of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and achieving good generalization.

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0023] 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0024] In an embodiment of the present disclosure, the client devices 101, 102, 103, 104, 105, 106 and the server 120 may execute one or more services or software applications that enable the execution of the image style transition method.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0026] In the configuration shown in FIG. 1 , server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof, executable on one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with server 120 to access services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Thus, FIG. 1 is intended to be illustrative and not limiting of a system for implementing various methods described herein.

[0027] Client devices 101, 102, 103, 104, 105, and / or 106 may provide an interface through which a user of the client device interacts with the client device. The client device may also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will understand that the present disclosure can support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, in-vehicle equipment, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT® Windows®, APPLE iOS, UNIX®-like operating systems, Linux® or Linux®-like operating systems, or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, and Android®. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), and the like. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client device may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), and short message service (SMS) applications, and may use a variety of communication protocols.

[0029] Network 110 may be any type of network known to those skilled in the art, which may use any one of several available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframes, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0032] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0033] In some embodiments, server 120 may be a server in a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and poor business scalability.

[0034] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of a variety of types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may be used by an application to store data for the application. The databases used by the application may be various types of databases, such as a key-value repository, an object repository, or a general-purpose repository supported by a file system.

[0036] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0037] According to some embodiments, the client devices 101-106 can acquire a reference image and description text input by a user, where the description text can include content description text describing the reference image, e.g., "A dog is walking down the street," and style description text describing the style of a target image to be generated, e.g., "Anime style." The client devices 101-106 send an image style transition request to the server 120 based on the reference image and description text input by the user. In response to the image style transition request from the client devices 101-106, the server 120 executes an image style transition method according to an embodiment of the present disclosure to generate a target image whose content matches the reference image specified by the user and has the specified style, and returns the generated target image to the client devices 101-106.

[0038] According to some embodiments, the client devices 101-106 may also execute an image style transition method according to an embodiment of the present disclosure. Specifically, the client devices 101-106 may execute the image style transition method according to an embodiment of the present disclosure to obtain a reference image and description text input by a user, and generate a target image based on the reference image and description text, the target image having a specified style and content that matches the reference image specified by the user.

[0039] 2 is a flowchart of an image style transition method 200 according to an embodiment of the present disclosure. As mentioned above, the method 200 may be performed by a client device, such as client devices 101-106 shown in FIG. 1, or by a server, such as server 120 shown in FIG. 1.

[0040] As shown in FIG. 2, the method 200 includes steps S210 to S270.

[0041] In step S210, a reference image and description text are obtained, where the description text includes a content description text that describes the content of the reference image and a style description text that describes the style of the target image to be generated.

[0042] In step S220, text features of the written text are extracted.

[0043] Steps S230 to S270 are performed based on the pre-trained diffusion model to generate a target image.

[0044] In step S230, a first cross-attention feature between the first image feature and the text feature is calculated at each time step of the diffusion model. The first image feature at the first time step is the image feature of a preset initial image, and the first image feature at each time step after the second time step is the result image feature generated at the previous time step.

[0045] In step S240, a second image feature of the reference image and a second cross-attention feature of the text feature are obtained.

[0046] In step S250, the first cross-attention feature is edited based on the second cross-attention feature to obtain a third cross-attention feature.

[0047] In step S260, a result image feature of the time step is generated based on the third cross attention feature and the text feature.

[0048] In step S270, the resulting image features of the last time step are decoded to generate a target image whose content matches the content of the reference image and has the style indicated by the style description text.

[0049] According to an embodiment of the present disclosure, a non-learning image style transition method based on attention editing is provided, which edits a first cross-attention feature calculated in the image generation process of a diffusion model using the image features of a reference image and a second cross-attention feature of text features, thereby continuously introducing information in the reference image into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model and ensuring that the generated target image is consistent with the content of the reference image and has a specified style.

[0050] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without requiring further training (fine-tuning) of the diffusion model using large amounts of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and achieving good generalization.

[0051] Each step of the method 200 is described in detail below.

[0052] In step S210, a reference image and a description text are obtained.

[0053] The reference image may be input by a user. According to some embodiments, the user may input a single image as the reference image. According to other embodiments, the user may input a reference video, and correspondingly, the reference image may be any image frame in the reference video.

[0054] The description text further includes content description text and style description text.

[0055] The content description text is used to describe the content of the reference image. According to some embodiments, the content description text may be entered by a user. For example, a user can specify a reference image and enter the content description text for the reference image: "A dog is walking down the street."

[0056] In another embodiment, the content description text may be automatically generated from a reference image. For example, a user-specified reference image may be input to a trained image understanding model, and the image understanding model may output the content description text for the reference image. The image understanding model may be, for example, a neural network model including a large language model, an image encoder, and a text decoder.

[0057] The style description text is used to describe the new style that the reference image will transition into, i.e., the style of the target image to be generated, such as a photo style, an anime style, a sketch style, a sumi-e style, etc. The style description text can be input by the user.

[0058] In step S220, text features of the written text are extracted.

[0059] According to some embodiments, the entire description text can be input to a trained text encoder to obtain text features of the description text output by the text encoder. The text encoder may be, for example, a Contrastive Language-Image Pretraining (CLIP) text encoder, a Bidirectional Encoder Representations from Transformers (BERT) model, a word2vec model, etc. Generally, the text encoder divides the description text into multiple tokens and encodes each token to obtain a feature vector for each token. The feature vectors for each token are concatenated to obtain text features of the description text.

[0060] As can be seen, the description text includes two parts, the content description text and the style description text, and accordingly the text features of the description text also include two parts, i.e. the text features of the description text include a first text feature of the content description text and a second text feature of the style description text. The first text feature includes a feature vector of each token in the content description text. The second text feature includes a feature vector of each token in the style description text.

[0061] According to some embodiments, step S220 may include steps S221 to S223.

[0062] In step S221, the content description text is encoded to obtain a first text feature of the content description text.

[0063] In step S222, the information of the reference image is introduced into the style description text to obtain an extended style description text.

[0064] In step S223, the extended style description text is encoded to obtain second text features of the extended style description text, where the text features of the description text include the first text feature and the second text feature.

[0065] According to the above embodiment, the content description text and the style description text are coded separately, and reference image information is introduced into the style description text. This allows the degree of preservation of the reference image content and the degree of application of the new style to be controlled separately during the style transition process, making the style transition process more controllable and smoother.

[0066] According to some embodiments, in step S221, the content description text may be input to a text encoder to obtain first text features of the content description text output from the text encoder. As can be seen, the first text features include a feature vector of each token in the content description text.

[0067] According to some embodiments, in step S222, a style description identifier of the reference image can be obtained, where the style description identifier of the reference image indicates the style of the reference image. The original style description text and the style description identifier of the reference image are concatenated to obtain an extended style description text. That is, the extended style description text includes the original style description text and the style description identifier of the reference image.

[0068] According to some embodiments, the style description identifier of the reference image may be a token already present in the lexicon, such as "photo" or "sketch." In this case, the style description identifier of the reference image can be obtained by recognition using a trained style recognition model. Specifically, the reference image can be input to the style recognition model, and the style type of the reference image output by the style recognition model can be obtained. The style recognition model may be, for example, a convolutional neural network.

[0069] If the style description identifier is a token already present in the glossary, step S223 may include inputting the extended style description text into a text encoder to obtain second text features of the extended style description text output by the text encoder. As can be seen, the second text features include a feature vector for each token in the extended style description text.

[0070] According to some embodiments, the style description identifier of the reference image may be a visual identifier that does not appear in a lexicon, which may be represented as [S*], for example, and since the visual identifier has not appeared in the lexicon, a feature vector for the visual identifier cannot be obtained using a text encoder.

[0071] If the style description identifier is a visual identifier that does not appear in the lexicon, step S223 may include steps S2231 to S2233.

[0072] In step S2231, a text encoder is used to extract first text sub-features of the style description text.

[0073] In step S2232, a third image feature of the reference image is extracted using an image encoder, the image encoder and the text encoder being configured to map the image and the text, respectively, into the same feature space.

[0074] In step S2233, the third image feature is set as the second text sub-feature of the style description identifier, and the second text feature of the extended style description text includes the first text sub-feature and the second text sub-feature.

[0075] According to the above embodiment, the text information and image information in the extended style description text are encoded using a cross-modal text encoder and an image encoder, respectively, and cross-modal features can be accurately extracted, thereby accurately representing the visual style features of the reference image and improving the accuracy of style transition.

[0076] According to some embodiments, the text encoder in step S2231 may be a CLIP text encoder, and the image encoder in step S2232 may be a CLIP image encoder, which can map text and images to the same feature space and achieve a cross-modal, uniform feature representation.

[0077] According to some embodiments, if the reference image is an independent image without context, in step S2232, the reference image can be input to an image encoder to obtain a third image feature output by the image encoder.

[0078] According to some embodiments, if the reference image is any image frame in the reference video, in step S2232, an image encoder can be used to extract image features of one or more image frames in the reference video, and the extracted image features can be used as the third image features of the reference image. For example, the first image frame of the reference video can be input to the image encoder, and the image encoder can be used to obtain the image features of the output image frame. When performing style transition using each image frame in the reference video as a reference image, the image features of the first image frame in the reference video are used as the third image features of the reference image.

[0079] According to the above-mentioned embodiment, each image frame in the same reference video can multiplex the same third image feature, which is advantageous for avoiding redundant calculation of the third image feature and improving the consistency of style transition of each image frame in the video style transition task.

[0080] An embodiment of the present disclosure utilizes a pre-trained diffusion model to generate a target image whose content matches the content of the reference image and has a style indicated by the style description text.

[0081] In the embodiments of the present disclosure, the pre-trained diffusion model has the ability to generate images from basic text, but is not fine-tuned for the image style transfer task.

[0082] The pre-trained diffusion model performs multiple denoising operations (i.e., de-diffusion operations) on a pre-defined initial image based on text features to finally obtain a target image. Each denoising operation of the diffusion model corresponds to one time step.

[0083] The pre-trained diffusion model includes a cross-attention layer. The image generation process of the diffusion model is as follows: At each time step t (t=T, T-1, T-2, ..., 2, 1, where T is a preset integer greater than 1, such as 50, 100, etc.), the cross-attention layer generates the first image feature I t Using the text features Text of the written text as input, we extract the first image feature I using the cross-attention mechanism. t and the text feature Text.

number

[0084] Note that for the first time step t=T, the first image feature IT is an image characteristic of the initial image. The initial image may be, for example, a random noise image or an image obtained by adding noise to a reference image. For each time step t after the second, the first image characteristic I t is the result image feature O generated in the previous time step (t+1). t+1 is.

[0085] The embodiments of the present disclosure introduce an attention editing mechanism based on a pre-trained diffusion model, which achieves high-quality image style transition without further fine-tuning the diffusion model. Specifically, the second cross attention feature M t The first cross-attention feature M calculated in the image generation process of the diffusion model using t * By editing, the information in the reference image can be continuously introduced into the image generation process of the diffusion model, thereby effectively utilizing the information in the reference image to guide the image generation of the diffusion model, and ensuring that the generated target image can match the content of the reference image and have a specified style.

[0086] Steps S230 to S270 describe the process of generating a target image after introducing the attention editing mechanism into the pre-trained diffusion generative model.

[0087] In step S230, at each time step t (t=T, T-1, T-2, ..., 2, 1, where T is a preset integer greater than 1, e.g., 50, 100, etc.) of the diffusion model, a first image feature I t and the text feature Text, the first intersection attention feature M t * Calculate the first image feature I at the first time step t=T. Tare image characteristics of a preset initial image. The initial image may be, for example, a random noise image or an image obtained by adding noise to a reference image. The image features of the initial image can be extracted using an image encoder (e.g., a CLIP image encoder). The first image feature I at each of the second and subsequent time steps t=T-1, T-2, ..., 2, 1 is t is the result image feature O generated at the previous time step t+1. t+1 is.

[0088] As mentioned above, the diffusion model includes a cross-attention layer, which is a layer that transforms a query matrix W Q , Key transformation matrix W K and the value transformation matrix W V It contains three parameters:

[0089] The cross-attention layer focuses on the first image feature I t The input is the text feature Text of the description text. The query transformation matrix W Q Using the first image feature I t to obtain the query matrix Q t =I t ·W Q The key transformation matrix W K and the value transformation matrix W V The text feature Text is linearly transformed using the key matrix K t =Text·W K and the value matrix V t =Text·W V The first cross attention feature M t * Calculate:

number

number

[0090] According to some embodiments, the diffusion model may also include a self-attention layer. The output terminal of the self-attention layer may be connected to the input terminal of the cross-attention layer. Correspondingly, step S230 may include steps S231 to S233.

[0091] In step S231, the self-attention feature of the first image feature is calculated.

[0092] In step S232, a fourth image feature is generated based on the self-attention feature and the first image feature.

[0093] In step S233, a first cross attention feature between the fourth image feature and the text feature is calculated.

[0094] According to the above embodiment, by realizing information aggregation within the first image feature using the self-attention mechanism, correlations between pixel points can be captured, and the aggregated first image feature (i.e., the fourth image feature) can more accurately represent the information of the generated image. By calculating the first cross-attention feature using the fourth image feature, the first cross-attention feature can accurately represent the information of the generated image, thereby improving the quality of the generated target image.

[0095] In step S231, the first image feature I is extracted using the self-attention layer. t The self-attention feature M s,t Specifically, the self-attention layer computes the query transformation matrix

number

number

number

number

number

number

number

number

[0096] In step S232, the self-attention feature M s,t The first image feature I t The value matrix V calculated by s,t The fourth image feature I is obtained by multiplying s,t That is, the updated first image feature can be obtained. In other words, in this embodiment, the fourth image feature I s,t is calculated by the following equation (3):

number

[0097] In step S2321, the self-attention feature M s,t Based on the historical self-attention feature corresponding to s,t and adjust the adjusted self-attention feature M s,t The historical self-attention feature is obtained by performing style transition on the historical image frame of the reference image using a diffusion model. The position of the self-attention feature M s,t It is the same attention feature as

[0098] In step S2322, the adjusted self-attention features M s,t ' and the first image feature I t Based on the fourth image feature I s,t The fourth image feature I s,t can be calculated using the following formula (4).

number

[0099] With respect to step S2321 above, it should be understood that each historical image frame corresponds to one historical self-attention feature. If there are multiple historical image frames, multiple historical self-attention features can be obtained.

[0100] According to some embodiments, the self-attention feature M s,t and the self-attention feature M after adjusting the average value of each historical self-attention feature. s,t ' can be used.

[0101] According to another embodiment, the self-attention feature M s,t and the weighted sum of each historical self-attention feature, and the adjusted self-attention feature M s,t The weight of each historical self-attention feature may be negatively correlated with the distance from the corresponding historical image frame to the reference image, i.e., the closer (smaller) the historical image frame is to the reference image, the larger the weight of the historical self-attention feature corresponding to that historical image frame.

[0102] In step S233, the fourth image feature I is extracted using the cross-attention layer. s,t and the text feature Text, and calculate the first attention cross feature M t * Specifically, the fourth image feature I s,t The cross attention feature of Text is the first image feature I mentioned above. t The calculation method for the intersection attention feature between the first image feature I and the text feature Text is the same as that for the intersection attention feature between the first image feature I and the text feature Text. The difference is that the first image feature I t The fourth image feature I s,t The only thing to do is to replace

[0103] In step S240, a second image feature of the reference image and a second cross-attention feature of the text feature are obtained.

[0104] The second image feature F of the reference image can be extracted using an image encoder (eg, a CLIP image encoder).

[0105] The second intersection attention feature M between the second image feature F of the reference image and the text feature Text. t can be similarly obtained using the cross-attention layer of the diffusion model. Specifically, the query transformation matrix W Q The second image feature F is linearly transformed using the query matrix Q = F W Q The key transformation matrix W K and the value transformation matrix W V The text feature Text is linearly transformed using the key matrix K=Text·W K and the value matrix V=Text·W V As can be seen, here K is the K t V is the same as the V above. t The second cross attention feature M t Calculate:

number

number

[0106] In step S250, the first cross attention feature is edited based on the second cross attention feature to obtain a third cross attention feature, which should be understood to be the edited first cross attention feature.

[0107] As described above, the text feature of the description text includes two parts: a first text feature of the content description text and a second text feature of the style description text. Therefore, the first cross-attention feature, the second cross-attention feature, and the third cross-attention feature can each be divided into two sub-features, one sub-feature corresponding to the content description text and the other sub-feature corresponding to the style description text. Specifically, the first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text. The second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text. The third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text.

[0108] According to the above-mentioned sub-feature dividing embodiment, step S250 may further include steps S251 and S252.

[0109] In step S251, the first content sub-feature is modified based on the second content sub-feature to obtain a third content sub-feature.

[0110] In step S252, a third style sub-feature is determined based on the first style sub-feature.

[0111] According to the above embodiment, the content sub-features and the style sub-features are edited separately, so that the reference image mainly affects the content of the target image and avoids excessive influence on the application of the new style.

[0112] The content of the target image depends on the content of the reference image. According to some embodiments, in step S251, the first content sub-feature can be replaced by the product of the second content sub-feature and the first factor. That is, the third content sub-feature is the product of the second content sub-feature and the first factor. The first factor indicates the degree of match between the content of the target image and the content of the reference image, that is, the degree of retention of the reference image content.

[0113] According to some embodiments, the first factor is a positive number, and thus the value of the first factor is positively correlated with the degree of reference image content retention. By adjusting the value of the first factor, the degree of reference image content retention during the style transition process can be controlled. Specifically, the larger the value of the first factor, the greater the degree of reference image content retention, and the higher the degree of match between the target image content and the reference image content. The smaller the value of the first factor, the less the degree of reference image content retention, and the lower the degree of match between the target image content and the reference image content.

[0114] According to some embodiments, in step S251, the weighted sum of the first content sub-feature and the second content sub-feature can be a third content sub-feature. The weight of the second content sub-feature can indicate the degree of match between the target image content and the reference image content, i.e., the degree of retention of the reference image content. By adjusting the weight of the second content sub-feature, the degree of retention of the reference image content during the style transition process can be controlled.

[0115] The style of the target image depends on the style description text and is less influenced by the reference image. Therefore, according to some embodiments, in step S252, the third style sub-feature can be determined based only on the first style sub-feature. Note that if information about the reference image is introduced into the style description text in step S222, the extended style description text also contains information about the reference image. Therefore, even if the third style sub-feature is determined based only on the first style sub-feature, the style of the target image can still be guided by the reference image information, and the style transition of the target image relative to the reference image can be smooth, without causing an excessively abrupt style change.

[0116] According to some embodiments, in step S252, a third style sub-feature may be a product of the first style sub-feature and a second factor, where the second factor indicates the degree of application of the new style (i.e., the style indicated by the style description text).

[0117] According to some embodiments, the second factor is a positive number, and therefore, the value of the second factor has a positive correlation with the degree of application of the new style. By adjusting the value of the second factor, the degree of application of the new style in the style transition process can be controlled. Specifically, the larger the value of the second factor, the greater the degree of application of the new style. The smaller the value of the second factor, the less the degree of application of the new style.

[0118] According to some embodiments, the attention editing process of step S250 can be expressed as follows:

number

number

[0119] In step S260, a result image feature of the time step is generated based on the third cross attention feature and the text feature.

[0120] According to some embodiments, in step S260, a third attention feature M t ** is the value matrix V calculated from the text features. t By multiplying with the resulting image feature O t In this embodiment, the result image feature O t is calculated using the following equation (7).

number

number

[0121] In step S270, the resulting image features of the final time step are decoded to generate a target image whose content matches the content of the reference image and has the style indicated by the style description text.

[0122] According to some embodiments, the pre-trained diffusion model may include a decoder, which may decode the resulting image features of the last time step to obtain a target image after the style transition, whose content matches the content of the reference image and has the style indicated by the style description text.

[0123] 3 is a schematic diagram of a video style transition process according to an embodiment of the present disclosure. The video style transition process is implemented by a video style transition service. As shown in FIG. 3, the video style transition service includes a text feature mapping module 310, a note editing module 320, and a video generation module 330.

[0124] The text feature mapping module 310 receives a description text D="X,Y" given by a user, where X is a content description text used to describe the content in the reference video V, and Y is a style description text for describing the style of the target video to be generated. As shown in Figure 3, the content description text X="There is a dog that is walking on the street" and the style description text Y="anime style".

[0125] In order to retain more content characteristics of the reference video V, the content information of the reference video V is introduced into the description text D, and the description text D is updated to D*="X,Y*", where Y*="Y of [S*]"="anime style of [S*]", [S*] is the style description identifier of the reference video V.

[0126] The text feature mapping module 310 uses the pre-trained CLIP model to encode the updated written text D* to obtain text features for D*. Specifically, for natural language tokens in D*, i.e., the "anime style of" portion in X and Y*, the text features are extracted using the CLIP model's text encoder 312. For the style description identifier [S*], which is a token that does not appear in the lexicon, the visual encoder (i.e., image encoder) 314 in the CLIP model is used to extract visual features of the first frame of the reference video V. Because the CLIP model can map visual features and text features to the same feature space, the feature is used as the text feature corresponding to the style description identifier [S*]. Then, the two partial features are concatenated to obtain the complete text features for the written text D*.

[0127] The attention editing module 320 takes the text features of the description text D* and the reference video V as input, edits the cross-attention mechanism between the reference video V and the description text, obtains new attention features in the video generation process during style transition, and introduces the new attention features into the inference process of the pre-trained basic model, i.e., the stable diffusion model 334.

[0128] At the t-th time step of the inference process, the intersection attention feature M between the feature code of the currently generated video frame and the text features of D* is calculated. t * , and calculate the intersection attention feature M t Get.

number

[0129] During the process of generating the video, the attention editing module 320 generates the intersection attention features M between the feature encoding of the generated video frames and the text features of D*. t * When computing, the feature coding of the reference video V is replaced by the intersection attention feature between the text features of X and M t * Generate the feature encoding of the video frame and the attention feature part of the X text features, i.e.:

number

[0130] The video generation module 330 generates a post-style transition video V' based on the stable diffusion model 334. The video generation module 330 encodes the reference video V using an encoder 332 to obtain a feature encoding f1 of the reference video V. The encoder 332 may be, for example, a CLIP visual encoder. A feature encoding f2 is obtained by adding noise (e.g., random noise conforming to a Gaussian distribution) to the feature encoding f1. The feature encoding f2 is used as the initial image feature of the stable diffusion model 334, i.e., the starting point of the de-diffusion operation.

[0131] In the process of generating each video frame, the edited cross-attention features are used to replace the original cross-attention features calculated in the stable diffusion model 334. At the same time, the self-attention features of the historical video frames are used to replace the original self-attention features calculated in the stable diffusion model 334 to build the association of each video frame, so that the generated video is better represented in timing consistency.

[0132] The video style transition process shown in FIG. 3 has the following advantages:

[0133] 1. It does not require training models using large amounts of data, reducing the cost of deploying and using the service.

[0134] 2. Users can customize the stylization levels α and β.

[0135] 3. The generated videos after style transition have excellent timing consistency.

[0136] According to an embodiment of the present disclosure, an image style transition device is also provided. Figure 4 is a block diagram showing the configuration of an image style transition device 400 according to an embodiment of the present disclosure. As shown in Figure 4, the device 400 includes an acquisition module 410, an extraction module 420, and a generation module 430.

[0137] The acquiring module 410 is configured to acquire a reference image and description text, where the description text includes a content description text that describes the content of the reference image and a style description text that describes the style of a target image to be generated.

[0138] The extraction module 420 is configured to extract text features of the written text.

[0139] The generation module 430 is configured to generate the target image based on a pre-trained diffusion model. The generation module 430 further includes an attention editing unit 432 and a decoding unit 434.

[0140] The attention editing unit 432 is configured to calculate a first cross-attention feature between a first image feature and the text feature at each time step of the diffusion model, where the first image feature at the first time step is an image feature of a preset initial image, and the first image feature at each time step from the second time step onwards is a result image feature generated at the previous time step, obtain a second cross-attention feature between a second image feature of the reference image and the text feature, edit the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, and generate a result image feature of the time step based on the third cross-attention feature and the text feature.

[0141] A decoding unit 434 is configured to decode the resulting image features of the last time step to generate the target image.

[0142] According to an embodiment of the present disclosure, a non-learning image style transition device based on attention editing is provided, which edits a first cross-attention feature calculated in the image generation process of a diffusion model using the image features of a reference image and a second cross-attention feature of a text feature, thereby continuously introducing information in the reference image into the image generation process of the diffusion model, thereby effectively utilizing information in the reference image to guide the image generation of the diffusion model, and ensuring that the generated target image is consistent with the content of the reference image and has a specified style.

[0143] The present disclosure can achieve high-quality image style transition using a general-purpose, pre-trained diffusion model, without requiring further training (fine-tuning) of the diffusion model using large amounts of label data, improving the efficiency of image style transition, reducing the deployment and usage costs of the image style transition service, and achieving good generalization.

[0144] According to some embodiments, the first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, the third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and the attention editing unit includes a content editing subunit configured to modify the first content sub-feature based on the second content sub-feature to obtain the third content sub-feature, and a style editing subunit configured to determine the third style sub-feature based on the first style sub-feature.

[0145] According to some embodiments, the content editing subunit is further configured to replace the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating the degree of match between the content of the target image and the content of the reference image.

[0146] According to some embodiments, the style editing subunit is further configured to set the third style sub-feature as a product of the first style sub-feature and a second factor, the second factor indicating the degree of application of the style.

[0147] According to some embodiments, the extraction module includes a first encoding unit configured to encode the content description text to obtain first text features of the content description text, an introduction unit configured to introduce information of the reference image into the style description text to obtain extended style description text, and a second encoding unit configured to encode the extended style description text to obtain second text features of the extended style description text, wherein the text features include the first text features and the second text features.

[0148] According to some embodiments, the extended style description text comprises the style description text and a style description identifier of the reference image, and the second encoding unit comprises: a first encoding subunit configured to extract first text sub-features of the style description text using a text encoder; a second encoding subunit configured to extract third image features of the reference image using an image encoder, wherein the image encoder and the text encoder are configured to map image and text to the same feature space, respectively; and a determination subunit configured to determine the third image features as second text sub-features of the style description identifier, wherein the second text features comprise the first text sub-features and the second text sub-features.

[0149] According to some embodiments, the reference image is any image frame in a reference video, wherein the second encoding subunit is further configured to utilize the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

[0150] According to some embodiments, the attention editing unit includes a first calculation subunit configured to calculate a self-attention feature of the first image feature, a generation subunit configured to generate a fourth image feature based on the self-attention feature and the first image feature, and a second calculation subunit configured to calculate a first cross-attention feature between the fourth image feature and the text feature.

[0151] According to some embodiments, the reference image is any image frame other than a first image frame in a reference video, wherein the generation subunit is further configured to adjust the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, wherein the historical self-attention feature is an attention feature having the same position as the self-attention feature, obtained by style transitioning historical image frames of the reference image using the diffusion model, and generate the fourth image feature based on the adjusted self-attention feature and the first image feature.

[0152] It should be understood that each module and unit of the apparatus 400 shown in Figure 4 may correspond to each step of the method 200 described with reference to Figure 2. Accordingly, the operations, features, and advantages described above with respect to the method 200 are equally applicable to the apparatus 400 and the modules and units included therein. For the sake of brevity, some operations, features, and advantages will not be described here.

[0153] Although particular functionality is discussed above with reference to particular modules, it should be noted that the functionality of each module discussed herein may be split into multiple modules and / or at least some of the functionality of multiple modules may be combined into a single module.

[0154] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Each unit described with respect to FIG. 4 may be realized in hardware or in hardware in combination with software and / or firmware. For example, these units may be implemented as computer program code / instructions configured to run on one or more processors and stored on a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of modules 410-430 may be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (e.g., a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) that may optionally execute received program code and / or include embedded firmware to perform functions.

[0155] According to an embodiment of the present disclosure, there is further provided an electronic device, the electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, enabling the at least one processor to perform an image style transition method according to an embodiment of the present disclosure.

[0156] According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform an image style transition method according to an embodiment of the present disclosure.

[0157] According to an embodiment of the present disclosure, there is also provided a computer program product including computer program instructions that, when executed by a processor, implements an image style transition method according to an embodiment of the present disclosure.

[0158] Next, referring to FIG. 5 , a block diagram of an electronic device 500 functioning as a server or client of the present disclosure will be described, which is an example of a hardware device applicable to each aspect of the present disclosure. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stage computers, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0159] 5, the electronic device 500 includes a computing unit 501 that can perform various appropriate operations and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data necessary for the operation of the electronic device 500. The computing unit 501, the ROM 502, and the RAM 503 are connected to one another via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0160] The electronic device 500 includes multiple components, including an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509, connected to the I / O interface 505. The input unit 506 may be any type of device capable of inputting information to the electronic device 500. The input unit 506 may receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device, including, but not limited to, a mouse, keyboard, touchscreen, trackboard, trackball, control lever, microphone, and / or remote control. The output unit 507 may be any type of device capable of presenting information, including, but not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 508 may include, but is not limited to, a magnetic disk or an optical disk. The communication unit 509 enables the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset (e.g., a Bluetooth® device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like).

[0161] The computing unit 501 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs each of the methods and processes described above, such as method 200. For example, in some embodiments, method 200 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, some or all of the computer program may be loaded and / or installed into the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, it can perform one or more steps of the method 200 described above. Alternatively, in other embodiments, the computing unit 501 is configured to perform these methods 200 in any other suitable manner (eg, by firmware).

[0162] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0163] Program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flowcharts and / or block diagrams. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0164] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may include or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection through one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] To provide for user interaction, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be used to provide for user interaction, for example, providing feedback to a user in any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and receiving input from a user in any form (including sound input, speech input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique implementation), or any combination of backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0167] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other via a communication network. The relationship between the client and the server is created by running computer programs having a client-server relationship on corresponding computers. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0168] It should be understood that the various forms of flow described above may be used to rearrange, add, or remove steps, and for example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the technical solutions disclosed in this disclosure can achieve the desired results, and the present disclosure is not limited thereto.

[0169] Although embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and that the scope of the present disclosure is not limited by these embodiments or examples, but only by the scope of the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by equivalent elements. Furthermore, steps may be performed in an order different from that described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. 1. A method for image style transition, comprising: obtaining a reference image and description text, the description text including content description text describing the content of the reference image and style description text describing the style of a target image to be generated; extracting text features from the written text; Based on a pre-trained diffusion model, At each time step of the diffusion model, Calculate a first cross-attention feature between a first image feature and the text feature, wherein the first image feature at a first time step is an image feature of a preset initial image, and the first image feature at each time step after the second time step is a result image feature generated at the previous time step; Obtain a second intersection attention feature between a second image feature of the reference image and the text feature; Editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; Generate a result image feature for the time step based on the third cross-attention feature and the text feature; and performing the operations of decoding the resultant image features of the last time step to generate a target image, thereby generating the target image.

2. the first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, and the third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature, modifying the first content sub-feature based on the second content sub-feature to obtain the third content sub-feature; and determining the third style sub-characteristic based on the first style sub-characteristic.

3. Modifying the first content sub-feature based on the second content sub-feature includes:

3. The method of claim 2, further comprising replacing the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating a degree of match between the content of the target image and the content of the reference image.

4. determining the third style sub-characteristic based on the first style sub-characteristic, The method of claim 2 , further comprising: determining the third style sub-feature as a product of the first style sub-feature and a second factor, the second factor indicating the degree of application of the style.

5. Extracting text features of the written text includes: encoding the content description text to obtain first text features of the content description text; introducing information of the reference image into the style description text to obtain an extended style description text; encoding the expanded style description text to obtain second text features of the expanded style description text; The method of claim 1 , wherein the text features include the first text feature and the second text feature.

6. The extended style description text includes the style description text and a style description identifier of the reference image, and encoding the extended style description text to obtain second text features of the extended style description text includes: extracting first text sub-features of the style description text using a text encoder; extracting a third image feature of the reference image using an image encoder, the image encoder and the text encoder configured to map the image and the text, respectively, into the same feature space; and wherein the third image feature is a second text sub-feature of the style description identifier; The method of claim 5 , wherein the second text feature comprises the first text sub-feature and the second text sub-feature.

7. The reference image is an arbitrary image frame in a reference video, and wherein extracting a third image feature of the reference image using the image encoder includes: The method of claim 6 , comprising utilizing the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

8. Calculating a first cross attention feature of the first image feature and the text feature includes: Computing a self-attention feature of the first image feature; generating a fourth image feature based on the self-attention feature and the first image feature; and computing a first cross-attention feature of the fourth image feature and the text feature.

9. the reference image is any image frame other than a first image frame in a reference video, and wherein generating a fourth image feature based on the self-attention feature and the first image feature includes: adjusting the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, wherein the historical self-attention feature is an attention feature obtained by style transitioning a historical image frame of the reference image using the diffusion model, and has the same position as the self-attention feature; and generating the fourth image feature based on the adjusted self-attention feature and the first image feature.

10. 1. An image style transition device, comprising: an acquiring module configured to acquire a reference image and description text, the description text including a content description text describing a content of the reference image and a style description text describing a style of a target image to be generated; an extraction module configured to extract text features of the written text; a generation module configured to generate the target image based on a pre-trained diffusion model, the generation module comprising: At each time step of the diffusion model, Calculate a first cross-attention feature between a first image feature and the text feature, wherein the first image feature at a first time step is an image feature of a preset initial image, and the first image feature at each time step after the second time step is a result image feature generated at the previous time step; Obtain a second intersection attention feature between a second image feature of the reference image and the text feature; Editing the first cross-attention feature based on the second cross-attention feature to obtain a third cross-attention feature; an attention editing unit configured to generate a result image feature for the time step based on the third cross attention feature and the text feature; a decoding unit configured to decode the resulting image features of the last time step to generate said target image.

11. The first cross-attention feature includes a first content sub-feature corresponding to the content description text and a first style sub-feature corresponding to the style description text, the second cross-attention feature includes a second content sub-feature corresponding to the content description text and a second style sub-feature corresponding to the style description text, the third cross-attention feature includes a third content sub-feature corresponding to the content description text and a third style sub-feature corresponding to the style description text, and the attention editing unit: a content editing subunit configured to modify the first content sub-feature based on the second content sub-feature to obtain the third content sub-feature; and a style editing subunit configured to determine the third style sub-characteristic based on the first style sub-characteristic.

12. The content editing subunit further comprises: The apparatus of claim 11 , configured to replace the first content sub-feature with a product of the second content sub-feature and a first factor, the first factor indicating a degree of match between content of the target image and content of the reference image.

13. The style editing subunit further comprises: The apparatus of claim 11 , wherein the third style sub-feature is a product of the first style sub-feature and a second factor, the second factor indicating the degree of application of the style.

14. The extraction module: a first encoding unit configured to encode the content description text to obtain first text features of the content description text; an introducing unit configured to introduce information of the reference image into the style description text to obtain an extended style description text; a second encoding unit configured to encode the extended style description text to obtain second text features of the extended style description text, The apparatus of claim 10 , wherein the text features include the first text feature and the second text feature.

15. The extended style description text includes the style description text and a style description identifier of the reference image, and the second encoding unit: a first encoding sub-unit configured to extract first text sub-features of said style description text using a text encoder; a second encoding subunit configured to extract third image features of the reference image using an image encoder, the image encoder and the text encoder configured to map image and text, respectively, into the same feature space; a determining subunit configured to determine the third image feature as a second text sub-feature of the style description identifier; The apparatus of claim 14 , wherein the second text features include the first text sub-feature and the second text sub-feature.

16. The reference image is an arbitrary image frame in a reference video, wherein the second encoding sub-unit further comprises: The apparatus of claim 15 , configured to utilize the image encoder to extract image features of one or more image frames in the reference video as third image features of the reference image.

17. The attention editing unit: a first computing subunit configured to compute a self-attention feature of the first image feature; a generating subunit configured to generate a fourth image feature based on the self-attention feature and the first image feature; and a second computing subunit configured to compute a first cross-attention feature of the fourth image feature and the text feature.

18. The reference image is any image frame other than the first image frame in the reference video, wherein the generating subunit further comprises: adjusting the self-attention feature based on a historical self-attention feature corresponding to the self-attention feature to obtain an adjusted self-attention feature, where the historical self-attention feature is an attention feature obtained by style transitioning historical image frames of the reference image using the diffusion model and having the same position as the self-attention feature; The apparatus of claim 17 , configured to generate the fourth image feature based on the adjusted self-attention feature and the first image feature.

19. An electronic device, at least one processor; a memory communicatively coupled to the at least one processor; An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 9.

21. A computer program comprising computer program instructions which, when executed by a processor, implement the method of any one of claims 1 to 9.

Images