Image generation method and apparatus based on artificial intelligence, electronic device, computer readable storage medium, and computer program product

Abstract

The present application provides an image generation method and apparatus based on artificial intelligence, an electronic device, a computer readable storage medium, and a computer program product. The method comprises: acquiring a content text and acquiring a style image having a target style; carrying out text encoding processing on the content text to obtain a content text code of the content text, and extracting a style code from the style image; and on the basis of a dual-path cross attention mechanism corresponding to the style code and the content text code, carrying out inverse diffusion processing on a noise image to obtain a target image, wherein the target image is matched with the content of the content text, and the target image has the target style.

Description

Artificial intelligence-based image generation method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202310820471.6 and application date July 5, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to artificial intelligence technology, and in particular to an image generation method, device, electronic device, computer-readable storage medium, and computer program product based on artificial intelligence. Background Art

[0004] Artificial Intelligence (AI) is a comprehensive field of computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0005] Style transfer technology has been applied to various image editing and image generation scenarios. The image content involved in the style transfer scheme in related technologies usually involves a specified content image, that is, the style of the existing image itself is transferred. Moreover, the effect of style transfer is relatively coarse-grained, mainly at the color level, which makes it impossible to efficiently and accurately generate images that meet the content and style requirements.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an artificial intelligence-based image generation method, device, electronic device, computer-readable storage medium, and computer program product, which can efficiently generate images that meet both content and style requirements.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] The present invention provides an artificial intelligence-based image generation method, including:

[0010] Get the content text and get the style image with the target style;

[0011] Performing text encoding processing on the content text to obtain a content text encoding of the content text, and extracting the style encoding from the style image;

[0012] Based on a two-way cross-attention mechanism corresponding to the style code and the content text code, inverse diffusion processing is performed on the noise image to obtain a target image;

[0013] The target image matches the content of the content text, and the target image has the target style.

[0014] The present invention provides an artificial intelligence-based image generation device, comprising:

[0015] an acquisition module configured to acquire content text and acquire a style image having a target style;

[0016] an encoding module configured to perform text encoding processing on the content text to obtain a content text encoding of the content text, and extract a style encoding from the style image;

[0017] The inverse diffusion module is configured to perform inverse diffusion processing on the noisy image based on a two-way cross-attention mechanism corresponding to the style encoding and the content text encoding to obtain a target image; wherein the target image matches the content of the content text and the target image has the target style.

[0018] An embodiment of the present application provides an electronic device, including:

[0019] a memory for storing computer-executable instructions;

[0020] The processor is used to implement the artificial intelligence-based image generation method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the artificial intelligence-based image generation method provided in an embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the artificial intelligence-based image generation method provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] The content text is obtained, and a style image with a target style is obtained. The content text is subjected to text encoding processing to obtain a content text encoding of the content text, and the style image is subjected to style encoding processing to obtain a style encoding. The content text encoding and the style encoding are integrated into the process of inverse diffusion processing of the noise image through a two-way cross attention mechanism, so that a target image that matches the content text and has a target style can be obtained at one time, thereby improving the efficiency of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of the structure of an artificial intelligence-based image generation system provided in an embodiment of the present application;

[0026] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0027] 3A-3D are flowcharts of an artificial intelligence-based image generation method according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram of a model of an image generation method based on artificial intelligence provided in an embodiment of the present application;

[0029] FIG5 is a schematic diagram of a model of an artificial intelligence-based image generation method provided in an embodiment of the present application;

[0030] 6A to 6C are schematic diagrams showing the effects of the artificial intelligence-based image generation method provided in an embodiment of the present application;

[0031] FIG7 is a schematic diagram showing the effect of the artificial intelligence-based image generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0036] The embodiments of the present application relate to artificial intelligence (AI), natural language processing and computer vision technology.

[0037] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0038] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] Natural language processing (NLP) is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. Natural language processing involves natural language, the language people use in daily life, and is closely related to linguistics research; it also involves computer science and mathematics. Pre-training models, an important technology for model training in the field of artificial intelligence, are developed from large language models (LLMs) in the field of NLP. After fine-tuning, large language models can be widely used in downstream tasks. Natural language processing technologies generally include text processing, semantic understanding, machine translation, robot question answering, knowledge graphs, and other technologies.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) U-Net: A commonly used convolution-based deep learning network architecture with a U-shaped feature connection method. U-Net is often used to perform image segmentation tasks.

[0042] 2) Stable diffusion model (SD): The working principle of the diffusion model is to learn the information attenuation due to noise and then use the learned pattern to generate images.

[0043] 3) Embedding: Embedding technology converts high-dimensional vectors into a relatively low-dimensional space, making machine learning easier and more efficient. It is primarily used in natural language processing and machine learning. It involves converting a high-dimensional, sparse vector into a low-dimensional, dense vector of real numbers. This process, also known as word embedding or vector embedding, encodes semantic information into this low-dimensional vector. This conversion is typically accomplished through deep learning network training.

[0044] Related technologies include traditional style transfer methods and diffusion model-based style generation methods. Traditional style transfer methods use inputs consisting of a content image and a style image. After extracting features from both, an additional style mapper is used to transfer the style of the style image to the content image. Diffusion model-based style generation methods learn placeholders for one or more style images in the text space, or these style images are input into a diffusion model to repeatedly fine-tune the diffusion model to train a diffusion model corresponding to a specific style. This allows the diffusion model corresponding to a specific style to be called and, using a prompt word as input, generate an image with that style.

[0045] Traditional style transfer methods rely on a specific content image. Their style transfer results are relatively coarse-grained, primarily focusing on color, and they transfer style to the existing image itself. Diffusion-based style generation methods, on the other hand, require training a separate model for each style, resulting in high training costs. Consequently, these solutions fail to achieve both high performance and high efficiency.

[0046] Based on the above technical problems, the embodiments of the present application provide an image generation method, device, electronic device, computer-readable storage medium and computer program product, which can efficiently generate images with specified semantics and reference style.

[0047] The image generation method provided in the embodiments of the present application can be implemented independently by the terminal / server; it can also be implemented collaboratively by the terminal and the server. For example, the terminal alone undertakes the following image generation method, or the terminal sends an image generation request (carrying content text and style image) to the server, and the server executes the image generation method according to the received image generation request, performs text encoding on the content text to obtain the content text encoding of the content text, and performs style encoding on the style image to obtain the style encoding; based on the two-way cross-attention mechanism corresponding to the style encoding and the content text encoding, the noise image is subjected to inverse diffusion processing to obtain a target image that matches the content text and has the target style, and the server returns the target image to the terminal.

[0048] The electronic device for executing the image generation method provided in the embodiments of the present application can be various types of terminal devices or servers, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0049] Taking servers as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.

[0050] Refer to Figure 1, which is a schematic diagram of the architecture of the image generation system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0051] Terminal 400 (running a clipping client) can be used to obtain an image generation request. For example, a user inputs content text and a style image through the input interface of terminal 400 (controls corresponding to different styles are triggered by selection operations. After triggering controls of any style, multiple style images corresponding to the style are obtained) to generate an image generation request. Terminal 400 sends the image generation request to server 200. Server 200 performs text encoding on the content text to obtain a content text encoding of the content text, and performs style encoding on the style image to obtain a style encoding. Based on the two-way cross-attention mechanism corresponding to the style encoding and the content text encoding, the noise image is subjected to inverse diffusion processing to obtain a target image that matches the content of the content text and has a target style. Server 200 returns the target image to terminal 400.

[0052] In some embodiments, an image generation plug-in may be embedded in the client running in the terminal to implement the image generation method locally on the client. For example, after receiving the image generation request, the terminal 400 calls the image generation plug-in to implement the image generation method, performs text encoding processing on the content text to obtain the content text encoding, and performs style encoding processing on the style image to obtain the style encoding; based on the two-way cross-attention mechanism corresponding to the style encoding and the content text encoding, performs inverse diffusion processing on the noise image to obtain a target image that matches the content text and has the target style.

[0053] Referring to FIG. 2 , FIG. 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The terminal 400 shown in FIG. 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It will be understood that the bus system 440 is used to implement connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG. 2 , all various buses are labeled as the bus system 440.

[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0057] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0059] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0060] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0061] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0062] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0063] In some embodiments, the image generation device provided in the embodiments of the present application can be implemented using software. FIG2 shows an image generation device 455 stored in memory 450. This device can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 4551, an encoding module 4552, and an inverse diffusion module 4553. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0064] As previously mentioned, the image generation method provided in the embodiments of the present application can be implemented by various types of electronic devices. The following description uses a terminal as an example execution subject. See FIG3A , which is a schematic flow chart of the image generation method provided in the embodiments of the present application. The description will be made in conjunction with steps 101 to 103 shown in FIG3A .

[0065] In step 101 , content text is obtained, and a style image with a target style is obtained.

[0066] As an example, the content text here is used to control the content of image generation. For example, if the content text is "motorcycle", then the content text can be used to guide the generation of a target image including a motorcycle, that is, the target image matches the content of the content text, for example, the target image includes the object mentioned in the content text.

[0067] As an example, the style image here can be one or more. If there are multiple style images, these style images all have the same style, which can be comic style, Van Gogh style, oil painting style, etc. The same style means that multiple style images all belong to the same style, for example, all belong to comic style, which can guide the generation of a target image with a target style.

[0068] In some embodiments, referring to FIG. 3B , obtaining a style image having a target style in step 101 may be implemented through steps 1011 to 1013 shown in FIG. 3B .

[0069] In step 1011, at least one original style image having the target style is obtained.

[0070] Steps 1012 to 1013 are performed for each of the original style images.

[0071] In step 1012, block segmentation processing is performed on the original style image to obtain multiple image blocks of the original style image.

[0072] In step 1013, a plurality of image blocks of the original style image are shuffled and spliced ​​to obtain a style image having the target style.

[0073] By using the embodiments of the present application, the original style image is scrambled based on image blocks, thereby interfering with the semantic information of the original style image while retaining style details such as texture and brush strokes in the original style image.

[0074] In step 102, text encoding processing is performed on the content text to obtain a content text encoding of the content text, and the style encoding is extracted from the style image.

[0075] As an example, the text encoding processing of the content text here can be implemented by the text model in the text image contrast model (Contrastive Language-Image Pre-training, CLIP). The training data of CLIP is text-image pairs (an image and its corresponding text description). Here, it is hoped that through contrast learning, the CLIP model can learn the matching relationship between text-image pairs. CLIP includes two models: a text model and a visual model. The text model is used to extract the features of the text and can adopt the Transformer model commonly used in NLP; while the visual model is used to extract the features of the image and can adopt a common convolutional neural network model or a visual Transformer model. In some embodiments, referring to Figure 3C, the extraction of style coding from the style image in step 102 can be implemented by executing steps 1021 to 1024 shown in Figure 3C for each of the style images.

[0076] In step 1021 , image coding processing is performed on the style image to obtain image coding of the style image.

[0077] In some embodiments, in step 1021, the style image is subjected to image coding processing to obtain the image coding of the style image, which can be achieved by the following technical solution: performing the following processing on each style image: obtaining a semantic coding that characterizes the semantic type of the style image; obtaining multiple image blocks of the style image; performing a second embedding process on each image block to obtain a visual coding of each image block, and performing a third embedding process on each image block to obtain a position coding of each image block; splicing the semantic coding with the visual coding of multiple image blocks to obtain a second splicing result; superimposing the second splicing result with the position coding of each image block corresponding to the image block to obtain an image coding. By encoding the style image at the semantic, positional, and visual levels according to the embodiment of the present application, the image coding obtained can have a better guiding effect.

[0078] As an example, the second embedding process and the third embedding process are actually mapping processes for image blocks, but the mapping parameters used are different. The style image is mapped through the visual model, where the visual model comes from the text image contrast model. The visual model is pre-trained. After mapping the style image through the visual model, the semantic encoding representing the semantic type of the style image and the visual encoding of each image block are output, referring to formula (1):

[0079] Among them, E Ir is the image encoding, E cls is the semantic encoding that represents the semantic type of the style image, E pps is the position code of each image block, is the non-overlapping image block obtained by dividing the style image, and E represents the mapping parameter used in the second embedding process. Here E pos The position coding of each image block in the style image is represented. The semantic coding does not participate in the superposition process. The position coding of the image block in the style image is superimposed on the visual coding of the image block.

[0080] In some embodiments, before performing an attention-based encoding process on the image encoding of the style image to obtain the attention image encoding of the style image, the following processing is performed on each of the style images: removing the semantic encoding of the semantic type representing the style image from the image encoding of the style image, and updating the removal result to the image encoding of the style image.

[0081] As an example, the semantic encoding E cls It is closely related to the semantic information in the style image, so the embodiment of the present application removes the semantic code E cls By discarding the semantic information in the style image, the embodiment of the present application can improve the representation capability of style characteristics and reduce the influence of semantic information on image generation.

[0082] The present invention can achieve category embedding suppression. Since our goal is to suppress the semantic information in the style image to avoid compromising the content fidelity of the output image, semantically relevant information is eliminated when extracting features from the style image using a visual model.

[0083] In step 1022, the image encoding of the style image is subjected to encoding processing based on the attention mechanism to obtain the attention image encoding of the style image.

[0084] As an example, the encoding processing based on the attention mechanism is implemented through the transformer network structure. The transformer network structure is composed of multiple cascaded self-attention modules, and the data processing of the self-attention module is implemented through the Attention formula of the self-attention mechanism.

[0085] In step 1023, when there are multiple style images, the attention image codes of the multiple style images are spliced ​​to obtain a first splicing result, and the first splicing result is embedded in a first manner to obtain the style code.

[0086] In step 1024, when the number of the style image is one, a first embedding process is performed on the attention image code of the style image to obtain the style code.

[0087] As an example, referring to FIG4 , when there are multiple style images, the attention images of the multiple style images are encoded and spliced ​​to obtain a first splicing result f r , and then the first splicing result f r A first embedding process is performed to obtain a style code. When the number of style images is one, the first embedding process can be directly performed on the attention image code of the style image to obtain the style code.

[0088] As an example, the following describes the process of the first embedding process, and in conjunction with Figure 5, introduces the style embedding network (StyleEmb) involved in the embodiment of this application. It is a Transformer structure composed of multiple attention modules. Its input is the first splicing result f r and a learnable embedding feature f m After passing through the Transformer structure, and Through the learnable matrix M s Mapped to f s (Style encoding) to participate in the subsequent generation process.

[0089] Through the embodiments of the present application, the attention image encodings of multiple style images can be fused, thereby obtaining the style information of multiple style images and improving the accuracy of style encoding.

[0090] In some embodiments, the text encoding processing and the image encoding processing are implemented by calling a text image contrast model, which includes a visual model and a text model, obtaining multiple first text samples and first image samples matching each first text sample; performing image encoding processing on each first image sample through the visual model of the text image contrast model to obtain the image encoding of each first image sample; performing text encoding processing on each first text sample through the text model of the text image contrast model to obtain the text encoding of each first text sample; determining the text image contrast loss based on the text encoding of each first text sample, the image encoding of each first image sample, and the matching relationship between each first text sample and each first image sample; and updating the parameters of the text image contrast model based on the text image contrast loss.

[0091] As an example, the text-image contrast model first inputs the matching first image sample and first text sample into a visual model image_encoder and a text model text_encoder respectively, obtaining the vector representations If and T_f of the first image sample and the first text sample. The vector representations of the first image sample and the first text sample are then mapped to a multimodal space to obtain the vector representations I_e and T_e of the first image sample and the first text sample that can be directly compared. This is a method in multimodal learning. There may be a gap between the data representations of different modalities, making direct comparison impossible. Therefore, mapping the data of different modalities to the same multimodal space first is conducive to subsequent similarity calculations and other operations. The cosine similarity between the vector representations of the first image sample and the first text sample is calculated. The objective function of contrastive learning is to make the similarity of positive sample pairs (matching first image samples and first text samples) higher and the similarity of negative sample pairs (mismatching first image samples and first text samples) lower.

[0092] In step 103, based on the two-way cross-attention mechanism corresponding to the style encoding and the content text encoding, the noise image is subjected to inverse diffusion processing to obtain a target image.

[0093] As an example, the target image matches the content of the content text, and the target image has the target style.

[0094] In some embodiments, referring to FIG3D , the two-way cross-attention mechanism based on the style encoding and the content text encoding described in step 103 performs inverse diffusion processing on the noise image to obtain the target image, which can be achieved through steps 1031 to 1032 shown in FIG3D .

[0095] In step 1031, through the n-th inverse diffusion network in N cascaded inverse diffusion networks, perform inverse diffusion processing on the input of the n-th inverse diffusion network based on the dual-path cross-attention mechanism, and transmit the n-th inverse diffusion result output by the n-th inverse diffusion network to the (n + 1)-th inverse diffusion network to continue performing inverse diffusion processing based on the dual-path cross-attention mechanism, so as to obtain the (n + 1)-th inverse diffusion result corresponding to the (n + 1)-th inverse diffusion network.

[0096] In step 1032, generate the target image based on the N-th inverse diffusion result corresponding to the N-th inverse diffusion network.

[0097] As an example, n is an integer variable that increases incrementally starting from 1, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th inverse diffusion network is the noise image, the content text encoding, and the style encoding. When n takes the value of 2 ≤ n < N, the input of the n-th inverse diffusion network is the (n - 1)-th inverse diffusion result output by the (n - 1)-th inverse diffusion network, the content text encoding, and the style encoding.

[0098] As an example, there are N cascaded inverse diffusion networks, which is equivalent to performing T times of inverse diffusion processing. Each time, inverse diffusion processing and random sampling processing are performed based on the noise image obtained from the previous inverse diffusion (i.e., the inverse diffusion result obtained from the previous inverse diffusion), and then input into the next inverse diffusion network for inverse diffusion processing and random sampling processing. n is an integer variable that increases incrementally starting from 1, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th inverse diffusion network is the noise image (or the latent noise image encoding), the content text encoding, and the style encoding. When n takes the value of 2 ≤ n < N, the input of the n-th inverse diffusion network is the (n - 1)-th inverse diffusion result output by the (n - 1)-th inverse diffusion network, the content text encoding, and the style encoding. [[ID=1l]]

[0099] As an example, taking N as 3 for illustration, perform inverse diffusion processing on the noise image (or latent space noise encoding), the content text encoding, and the style encoding through the 1st inverse diffusion network to obtain the 1st inverse diffusion result. Perform inverse diffusion processing on the 1st inverse diffusion result and the content text encoding through the 2nd inverse diffusion network to obtain the 2nd inverse diffusion result. Perform inverse diffusion processing on the 2nd inverse diffusion result, the content text encoding, and the style encoding through the 3rd inverse diffusion network to obtain the 3rd inverse diffusion result. Each inverse diffusion result obtained through the above method is also an image (or a latent space encoding). The inverse diffusion processing performed by each inverse diffusion network is equivalent to the inverse diffusion processing of one time step.

[0100] As an example, when the Nth inverse diffusion result is the latent space encoding, a random distribution is generated based on the Nth inverse diffusion result, and then a latent space image encoding is sampled from the random distribution. Then, the latent space image encoding is decoded by a decoder to obtain the target image. After each inverse diffusion process performed by the inverse diffusion network, the obtained data is used as the mean of the random distribution, and the variance is set data, so as to obtain the random distribution corresponding to the inverse diffusion result. Then, the random distribution is sampled to obtain the inverse diffusion result, which is input into the next inverse diffusion network. When the Nth inverse diffusion result is an image, the Nth inverse diffusion result is directly used as the target image.

[0101] Through the embodiments of the present application, denoising processing can be gradually performed on the latent space noise encoding or the noise image, so as to obtain the denoising result of the latent space or the denoising result of the real space. The denoising process is implemented on the latent space, reducing the data processing amount and improving the denoising speed. Denoising in the real space can be visually evaluated during the denoising process, optimizing the denoising effect and saving computing resources.

[0102] In some embodiments, the nth inverse diffusion network includes M cascaded sampling networks, where the value of M satisfies 2 ≤ M. In step 1031, through the nth inverse diffusion network in the N cascaded inverse diffusion networks, inverse diffusion processing based on the dual-path cross-attention mechanism is performed on the input of the nth inverse diffusion network. This can be achieved through the following technical solution: through the mth sampling network in the M cascaded sampling networks, sampling processing based on the dual-path cross-attention mechanism is performed on the input of the mth sampling network to obtain the mth sampling result corresponding to the mth sampling network. The mth sampling result corresponding to the mth sampling network is transmitted to the (m + 1)th sampling network to continue the sampling processing based on the dual-path cross-attention mechanism to obtain the (m + 1)th sampling result corresponding to the (m + 1)th sampling network. The sampling result output by the Mth sampling network is used as the nth inverse diffusion result. Through the embodiments of the present application, the dual-path cross-attention mechanism is combined with the sampling process. A single sampling is actually a process of extracting features using convolution, while sampling based on the dual-path cross-attention mechanism is a process of extracting features using two attention mechanisms, which can achieve targeted sampling and improve the representation ability of the model.

[0103] As an example, m is an integer variable that starts from 1 and increases incrementally, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the mth sampling network is the input of the nth inverse diffusion network, the content text encoding, and the style encoding. When m takes the value of 2 ≤ m < M, the input of the mth sampling network is the (m - 1)th sampling result output by the (m - 1)th sampling network, the content text encoding, and the style encoding.

[0104] As an example, taking the second inverse diffusion network as an example, the inverse diffusion network may include three downsampling networks and three upsampling networks. The first inverse diffusion result, content text encoding and style encoding are downsampled by the three cascaded downsampling networks to obtain the downsampling result of the second inverse diffusion network; the downsampling result, content text encoding and style encoding of the second inverse diffusion network are upsampled by the three cascaded upsampling networks to obtain the upsampling result of the second inverse diffusion network as the noise estimation result of the second inverse diffusion network; the first inverse diffusion result is subjected to noise elimination processing based on the noise estimation result of the second inverse diffusion network to obtain the second inverse diffusion result corresponding to the second inverse diffusion network.

[0105] As an example, the input of the first downsampling network is downsampled through the first downsampling network to obtain the downsampling result corresponding to the first downsampling network, and the downsampling result corresponding to the first downsampling network is transmitted to the second downsampling network to continue the downsampling process to obtain the second downsampling result corresponding to the second downsampling network; the input of the second downsampling network is downsampled through the second downsampling network to obtain the downsampling result corresponding to the second downsampling network, and the downsampling result corresponding to the second downsampling network is transmitted to the third downsampling network to continue the downsampling process to obtain the third downsampling result corresponding to the third downsampling network, and the third downsampling result output by the third downsampling network is used as the second inverse diffusion result. Here, the input of each downsampling network includes not only the output of the previous network, but also content text encoding and style encoding.

[0106] In some embodiments, the above-mentioned transmission of the mth sampling result corresponding to the mth sampling network to the m+1th sampling network to continue the sampling processing based on the dual-path cross-attention mechanism to obtain the m+1th sampling result corresponding to the m+1th sampling network can be achieved by the following technical solutions: performing self-attention processing on the mth sampling result of the mth sampling network to obtain the self-attention processing result of the m+1th sampling network; performing cross-attention processing on the self-attention processing result of the m+1th sampling network and the content text encoding to obtain the text cross-attention processing result of the m+1th sampling network; performing cross-attention processing on the self-attention processing result of the m+1th sampling network and the style encoding to obtain the style cross-attention processing result of the m+1th sampling network; performing fusion processing on the style cross-attention processing result of the m+1th sampling network and the text cross-attention processing result of the m+1th sampling network to obtain the m+1th sampling result corresponding to the m+1th sampling network. Through the embodiments of the present application, style guidance and content guidance can be integrated, so that the generation effect of the target image can be ensured in two aspects.

[0107] As an example, self-attention processing is implemented through the Attention formula of the self-attention mechanism. Subsequently, the self-attention processing results are cross-attention processed with the content text encoding and the style encoding to obtain two cross-attention processing results. The two cross-attention processing results are then fused through the parameter λ to obtain the m+1th sampling result of the m+1th sampling network, see formula (2):

[0108] in, is the m+1th sampling result of the m+1th sampling network, λ is a trainable fusion parameter, Attention(Q s , K s , V S ) is the result of style cross attention processing, Attention(Q t , K t , V t ) is the result of text cross attention processing.

[0109] In some embodiments, the above-mentioned cross-attention processing of the self-attention processing result of the m+1th sampling network and the content text encoding to obtain the text cross-attention processing result of the m+1th sampling network can be achieved by the following technical solutions: mapping processing based on the first query parameter on the self-attention processing result of the m+1th sampling network to obtain a first query matrix; mapping processing based on the first key parameter on the content text encoding to obtain a first key matrix; mapping processing based on the first value parameter on the content text encoding to obtain a first value matrix; performing attention calculation on the first query matrix, the first key matrix and the first value matrix to obtain the text cross-attention processing result of the m+1th sampling network. Through the embodiments of the present application, the content text encoding can be gradually integrated with the self-attention processing result, so that the target content can be gradually characterized during the inverse diffusion processing.

[0110] As an example, see formula (3):

[0111] Among them, W Qt is the first query parameter, y is the output of the self-attention module, W Kt is the first key parameter, W Vt is the first value parameter, f t is the style feature, d is the dimension, Q t is the first query matrix, K t is the first bond matrix, V t is the first value matrix, Attention(Q t , K t , Vt ) is the output of the text cross-attention module (text cross-attention processing result).

[0112] In some embodiments, the above-mentioned cross-attention processing of the self-attention processing result of the m+1th sampling network and the style coding to obtain the style cross-attention processing result of the m+1th sampling network can be achieved by the following technical solutions: mapping processing based on a second query parameter on the self-attention processing result of the m+1th sampling network to obtain a second query matrix; mapping processing based on a second key parameter on the style coding to obtain a second key matrix; mapping processing based on a second value parameter on the style coding to obtain a second value matrix; performing attention calculation on the second query matrix, the second key matrix and the second value matrix to obtain the style cross-attention processing result of the m+1th sampling network. Through the embodiments of the present application, the style coding can be gradually integrated with the self-attention processing result, so that the target style can be gradually characterized during the inverse diffusion process.

[0113] As an example, see formula (4):

[0114] Among them, W Qs is the second query parameter, y is the output of the self-attention module, W Ks Is the second key parameter, W Vs is the second value parameter, f s is the text feature, d is the dimension, Q S is the second query matrix, K s is the second bond matrix, V s is the second value matrix, Attention(Q s , K s , V s ) is the output of the style cross-attention module.

[0115] The parameters involved in the embodiments of the present application are all parameters obtained through training, such as the second key parameter, the first key parameter and other parameters.

[0116] The content text is obtained, and a style image with a target style is obtained. The content text is subjected to text encoding processing to obtain a content text encoding of the content text, and the style image is subjected to style encoding processing to obtain a style encoding. The content text encoding and the style encoding are integrated into the process of inverse diffusion processing of the noise image through a two-way cross attention mechanism, so that a target image that matches the content text and has a target style can be obtained at one time, thereby improving the efficiency of image generation.

[0117] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0118] A terminal (running an image editing client) can be used to obtain an image generation request. For example, a user inputs content text and a style image through the terminal's input interface (selection operations trigger controls corresponding to different styles. After triggering controls of any style, multiple style images corresponding to the style are obtained) to generate an image generation request. The terminal sends the image generation request to the server. The server performs text encoding on the content text to obtain a content text encoding of the content text, and performs style encoding on the style image to obtain a style encoding. Based on a two-way cross-attention mechanism corresponding to the style encoding and the content text encoding, the noise image is inversely diffused to obtain a target image that matches the content text and has a target style. The server returns the target image to the terminal.

[0119] The embodiments of the present application mainly provide a style adapter for style image generation. The diffusion model implemented based on the two-way cross-attention mechanism and the semantic style decoupling strategy of the style image enable the style adapter to generate user-desired content and images of the user's desired style based on the content text (prompt words) and the style reference image (i.e., the style image described above).

[0120] The following describes the architectural diagram of the style adapter provided in an embodiment of the present application in conjunction with FIG4 .

[0121] The style adapter includes a text model for parsing content text, a vision model for parsing style reference images, a learnable style embedding network (StyleEmb), and a diffusion model based on a two-way attention mechanism.

[0122] Given a content text P and a series of style reference images R = {I0, I1, ..., I K-1}, which are first parsed into text features f by textmodel and visionmodel respectively t and visual features {f0, f1, ..., f K-1}. Then the style embedding network transforms the visual features {f0, f1, ..., f K-1} fused into style features s . t and f s They are input into the two-way cross attention module in the diffusion model respectively. Specifically, f t Input to T-CrossA, T-CrossA represents the text cross attention module, f s Input to S-CrossA, S-CrossA represents the style cross attention module, which enables the diffusion model to generate images with content consistent with the content text and style consistent with the style reference image Io .

[0123] The network structures that need to update parameters during the training phase are the style cross-attention module and the style embedding network. The remaining network structures are obtained after the pre-training is completed.

[0124] The textmodel and visionmodel involved in the embodiments of the present application can be a contrastive language-image pre-training model (CLIP). Through simple text-image dual-tower contrast learning and a large amount of text-image corpus, the model has a significant ability to align text-image features, and is very effective in zero-sample image classification and cross-modal retrieval.

[0125] The diffusion model involved in the embodiment of the present application is composed of a cascade of multiple inverse diffusion (denoising) networks. Each inverse diffusion network includes multiple sampling layers (up-sampling layers and down-sampling layers). Each sampling layer includes a self-attention module and a two-way cross attention module. The two-way cross attention module consists of two parallel cross attention modules T-CrossA and S-CrossA, which process the text feature f respectively. t and style features f t Their processing results are combined through the parameter λ, see formula (5) to formula (7):

[0126] Among them, W Qt is the first query parameter, y is the output of the self-attention module, W Kt is the first key parameter, W Vt is the first value parameter, f t is the style feature, d is the dimension, Q t is the first query matrix, K t is the first bond matrix, V t is the first value matrix, Attention(Q t , K t , V t ) is the output of the text criss-cross attention module.

[0127] Among them, W Qs is the second query parameter, y is the output of the self-attention module, W Ks Is the second key parameter, W Vs is the second value parameter, f s is the text feature, d is the dimension, Q s is the second query matrix, K s is the second bond matrix, V sis the second value matrix, Attention(Q s , K s , V s ) is the output of the style cross-attention module.

[0128] in, is the output of the two-way cross attention module, λ is a trainable fusion parameter, Attention(Q s , K s , V s ) is the output of the style cross attention module, Attention(Q t , K t , V t ) is the output of the text criss-cross attention module.

[0129] The following describes the style embedding network (StyleEmb) involved in the embodiment of the present application in conjunction with Figure 5. It is a Transformer structure composed of multiple attention modules. Its input consists of the visual features {f0, f1, ..., f K-1}The concatenated feature f r and a learnable embedding feature f m After passing through the Transformer structure, and Through the learnable matrix M s Mapped to f s to participate in the subsequent generation process.

[0130] The following introduces the semantic style decoupling strategy of style images.

[0131] In order to decouple the semantics and style in the style image and alleviate the interference of the semantics in the style image on the generated image content, the embodiments of the present application propose three effective decoupling strategies.

[0132] First, the style reference image is shuffled based on blocks, which interferes with the semantic information of the style reference image while preserving the style details such as texture and strokes in the reference image.

[0133] Second, remove the class embedding information in the visual model used to parse the style image, see formula (8):

[0134] Among them, E cls is the class embedding (associated with the category) obtained by the visual model, E pos Represent the location information of each image block, is the non-overlapping image block obtained by dividing the style reference image. Formula (8) is the formula for the visual model to process the style reference image. E cls It is closely related to the semantic information in the style image, so the embodiment of the present application removes E cls To discard the semantic information in the style image.

[0135] Third, we provide multiple style reference images with strong semantic diversity. This means that the style reference consists of multiple images of the same style, but with as different semantics as possible within each image. This approach allows the model to prioritize capturing the common stylistic features across these references when generating images, while discarding inconsistent semantic features.

[0136] The style adapter proposed in the embodiment of the present application uses a two-way cross-attention mechanism and three decoupling strategies to generate a variety of images that are consistent with the content of the content text and the style of the reference reference using the same model.

[0137] The following describes the experimental results of the style adapter provided in the embodiment of the present application.

[0138] As shown in Figure 6A , given multiple style graphs, the style adapter provided by this embodiment of the present application can generate a target image that matches the style and content in a single pass. Furthermore, this embodiment of the present application is compatible with other controllable conditions, such as sketches. As shown in the results in the last column of Figure 6A , with the guidance of additional sketches, the shape of the generated content is more controllable.

[0139] See Figure 6B and Table 1, which shows the scores of various related technologies and the present solution at the two levels of text similarity and style similarity. Related Technology 1 and Related Technology 2 only perform coarse-grained color conversion. Related Technology 3 fails to produce satisfactory stylization because the text extracted from the style image performs poorly. Related Technology 4 relies on text inversion and achieves better results in content-based style conversion than hint-based style conversion, but its style is still far from the style image. In contrast, the target image generated by the style adapter provided in the embodiment of the present application is more faithful to the target style of the style image, especially in terms of strokes, lines, etc. The embodiment of the present application also produces a target image that better matches the content text and achieves a better balance between content fidelity and style fidelity.

[0140] Table 1 Text similarity and style similarity score table of each scheme

[0141] Referring to Figure 6C, an embodiment of the present application conducts experiments to evaluate the effectiveness of the two-way cross-attention mechanism module and the three decoupling strategies. Figure 8 shows the results. The target image obtained by the non-present scheme is obtained using a single style image, and the target image corresponding to the present scheme is obtained using multiple style images. The related art cannot generate cats based on content text, but a two-way cross-attention mechanism can generate cats. Furthermore, by losing semantic information, the cat becomes more visible. By shuffling image blocks, the embodiment of the present application achieves a high-fidelity generation effect. The embodiment of the present application introduces a trainable parameter λ, which can produce a better balance between content fidelity and style fidelity.

[0142] Referring to Figure 7, the dual-path cross-attention module provided by an embodiment of the present application fuses the information of the content text and the style reference through λ, which is an adaptive parameter that controls the trade-off between the guidance from the content text and the guidance from the style image. When λ is reduced by a factor less than 1.0, the guidance from the style image gradually disappears, and the generated target image becomes more natural. On the other hand, when λ is enlarged by a factor greater than 1.0, the style in the generated target image becomes more prominent, but the dog in the image also loses its natural appearance. Therefore, the embodiment of the present application can customize the generated target image by adjusting λ according to preference.

[0143] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0144] The following continues to describe an exemplary structure of the image generating device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the software modules stored in the image generating device 455 of the memory 450 may include: an acquisition module 4551, configured to acquire content text and acquire a style image with a target style; an encoding module 4552, configured to perform text encoding processing on the content text to obtain a content text encoding of the content text, and extract the style encoding from the style image; an inverse diffusion module 4553, configured to perform inverse diffusion processing on the noise image based on a two-way cross-attention mechanism corresponding to the style encoding and the content text encoding to obtain a target image; wherein the target image matches the content of the content text, and the target image has the target style.

[0145] In some embodiments, the acquisition module 4551 is further configured to: acquire at least one original style image having the target style; perform the following processing on each original style image: perform block segmentation processing on the original style image to obtain multiple image blocks of the original style image; perform shuffle splicing processing on the multiple image blocks of the original style image to obtain a style image having the target style.

[0146] In some embodiments, the encoding module 4552 is further configured to: perform the following processing on each of the style images: perform image encoding processing on the style image to obtain the image encoding of the style image; perform encoding processing based on the attention mechanism on the image encoding of the style image to obtain the attention image encoding of the style image; when the number of the style images is multiple, perform splicing processing on the attention image encodings of the multiple style images to obtain a first splicing result, and perform a first embedding processing on the first splicing result to obtain the style encoding; when the number of the style images is one, perform a first embedding processing on the attention image encoding of the style image to obtain the style encoding.

[0147] In some embodiments, the text encoding processing and the image encoding processing are implemented by calling a text-image contrast model, and the encoding module 4552 is further configured to: obtain multiple first text samples and first image samples matching each first text sample; perform image encoding processing on each first image sample through the visual model of the text-image contrast model to obtain the image encoding of each first image sample; perform text encoding processing on each first text sample through the text model of the text-image contrast model to obtain the text encoding of each first text sample; determine the text-image contrast loss based on the text encoding of each first text sample, the image encoding of each first image sample, and the matching relationship between each first text sample and each first image sample; and update the parameters of the text-image contrast model based on the text-image contrast loss.

[0148] In some embodiments, the encoding module 4552 is further configured to: perform the following processing for each of the style images: obtain a semantic code representing the semantic type of the style image; obtain multiple image blocks of the style image; perform a second embedding process on each of the image blocks to obtain a visual code of each of the image blocks, and perform a third embedding process on each of the image blocks to obtain a position code of each of the image blocks; perform a splicing process on the semantic code and the visual code of the multiple image blocks to obtain a second splicing result; perform a superposition process on the second splicing result and the position code of each of the image blocks corresponding to the image block to obtain the image code of the style image.

[0149] In some embodiments, before performing encoding processing based on an attention mechanism on the image encoding of the style image to obtain the attention image encoding of the style image, the encoding module 4552 is further configured to perform the following processing for each of the style images: removing the semantic encoding representing the semantic type of the style image from the image encoding of the style image, and updating the removal result as the image encoding of the style image.

[0150] In some embodiments, the inverse diffusion module 4553 is further configured to: perform inverse diffusion processing based on the two-way cross-attention mechanism on the input of the nth inverse diffusion network through the nth inverse diffusion network in N cascaded inverse diffusion networks, and transmit the nth inverse diffusion result output by the nth inverse diffusion network to the (n + 1)th inverse diffusion network to continue performing inverse diffusion processing based on the two-way cross-attention mechanism, so as to obtain the (n + 1)th inverse diffusion result corresponding to the (n + 1)th inverse diffusion network; generating the target image based on the Nth inverse diffusion result corresponding to the Nth inverse diffusion network; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth inverse diffusion network is the noise image, the content text encoding, and the style encoding. When 2 ≤ n < N, the input of the nth inverse diffusion network is the (n - 1)th inverse diffusion result output by the (n - 1)th inverse diffusion network, the content text encoding, and the style encoding.

[0151] In some embodiments, the nth inverse diffusion network includes M cascaded sampling networks, and the value of M satisfies 2 ≤ M; the inverse diffusion module 4553 is further configured to: perform sampling processing based on the two-way cross-attention mechanism on the input of the mth sampling network through the mth sampling network in M cascaded sampling networks, so as to obtain the mth sampling result corresponding to the mth sampling network, and transmit the mth sampling result corresponding to the mth sampling network to the (m + 1)th sampling network to continue performing sampling processing based on the two-way cross-attention mechanism, so as to obtain the (m + 1)th sampling result corresponding to the (m + 1)th sampling network; using the sampling result output by the Mth sampling network as the nth inverse diffusion result; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the mth sampling network is the input of the nth inverse diffusion network. When 2 ≤ m < M, the input of the mth sampling network is the (m - 1)th sampling result output by the (m - 1)th sampling network, the content text encoding, and the style encoding.

[0152] In some embodiments, the inverse diffusion module 4553 is further configured to: perform self-attention processing on the mth sampling result of the mth sampling network to obtain the self-attention processing result of the m+1th sampling network; perform cross-attention processing on the self-attention processing result of the m+1th sampling network and the content text encoding to obtain the text cross-attention processing result of the m+1th sampling network; perform cross-attention processing on the self-attention processing result of the m+1th sampling network and the style encoding to obtain the style cross-attention processing result of the m+1th sampling network; perform fusion processing on the style cross-attention processing result of the m+1th sampling network and the text cross-attention processing result of the m+1th sampling network to obtain the m+1th sampling result corresponding to the m+1th sampling network.

[0153] In some embodiments, the inverse diffusion module 4553 is further configured to: perform mapping processing on the self-attention processing result of the m+1th sampling network based on the first query parameter to obtain a first query matrix; perform mapping processing on the content text encoding based on the first key parameter to obtain a first key matrix; perform mapping processing on the content text encoding based on the first value parameter to obtain a first value matrix; perform attention calculation on the first query matrix, the first key matrix and the first value matrix to obtain the text cross-attention processing result of the m+1th sampling network.

[0154] In some embodiments, the inverse diffusion module 4553 is further configured to: perform mapping processing on the self-attention processing result of the m+1th sampling network based on a second query parameter to obtain a second query matrix; perform mapping processing on the style code based on a second key parameter to obtain a second key matrix; perform mapping processing on the style code based on a second value parameter to obtain a second value matrix; perform attention calculation on the second query matrix, the second key matrix and the second value matrix to obtain the style cross-attention processing result of the m+1th sampling network.

[0155] The present invention provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the instructions, causing the electronic device to perform the image generation method described in the present invention.

[0156] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the image generation method provided by the embodiment of the present application.

[0157] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0158] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0159] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0160] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0161] To summarize, through the embodiments of the present application, content text is obtained, and a style image with a target style is obtained, text encoding processing is performed on the content text to obtain content text encoding of the content text, and style encoding processing is performed on the style image to obtain style encoding, and the content text encoding and style encoding are integrated into the process of inverse diffusion processing of the noise image through a two-way cross attention mechanism, so that a target image that matches the content text and has a target style can be obtained at one time, thereby improving image generation efficiency.

[0162] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image generation method based on artificial intelligence, the method being performed by an electronic device, the method comprising: Get the content text and get the style image with the target style; Performing text encoding processing on the content text to obtain a content text encoding of the content text, and extracting a style encoding from the style image; Based on a two-way cross attention mechanism corresponding to the style code and the content text code, the noise image is subjected to inverse diffusion processing to obtain a target image; The target image matches the content of the content text, and the target image has the target style.

2. The method according to claim 1, wherein: The step of obtaining a style image having a target style includes: Acquire at least one original style image having the target style; The following processing is performed for each of the original style images: Performing block segmentation processing on the original style image to obtain multiple image blocks of the original style image; The multiple image blocks of the original style image are shuffled and spliced ​​to obtain a style image with the target style.

3. The method according to claim 1 or 2, wherein: The extracting the style code from the style image comprises: The following processing is performed for each style image: Performing image coding processing on the style image to obtain image coding of the style image; Performing an attention-based encoding process on the image encoding of the style image to obtain an attention-based image encoding of the style image; When there are multiple style images, performing a splicing process on the attention image codes of the multiple style images to obtain a first splicing result, and performing a first embedding process on the first splicing result to obtain the style code; When the number of the style images is one, a first embedding process is performed on the attention image code of the style image to obtain the style code.

4. The method according to any one of claims 1 to 3, wherein: The text encoding process and the image encoding process are implemented by calling a text image contrast model, wherein the text image contrast model includes a visual model and a text model. The method further includes: Acquire a plurality of first text samples and a first image sample matching each of the first text samples; Performing image coding processing on each of the first image samples by using the visual model to obtain an image code of each of the first image samples; Performing text encoding processing on each of the first text samples by using the text model to obtain a text encoding of each of the first text samples; Based on the text encoding of each of the first text samples, the image encoding of each of the first image samples, and the matching relationship between each of the first text samples and each of the first image samples, determine Fixed text image contrast loss; Parameters of the text image contrast model are updated based on the text image contrast loss.

5. The method according to claim 3, wherein: The performing image encoding processing on the style image to obtain the image encoding of the style image includes: The following processing is performed for each style image: Obtaining a semantic code representing a semantic type of the style image; Acquire multiple image blocks of the style image; Perform a second embedding process on each of the image patches to obtain the visual encoding of each image patch, and perform a third embedding process on each of the image patches to obtain the position encoding of each image patch; Concatenate the semantic encoding with the visual encodings of multiple image patches to obtain a second concatenation result; Perform an overlay process corresponding to each image patch on the second concatenation result and the position encoding of each image patch to obtain the image encoding of the style image.

6. The method according to claim 3, wherein: Before performing an encoding process based on the attention mechanism on the image encoding of the style image to obtain the attention image encoding of the style image, the method further includes: Perform the following process for each of the style images: Remove the semantic encoding representing the semantic type of the style image from the image encoding of the style image, and update the removal result as the image encoding of the style image.

7. The method according to any one of claims 1 to 6, wherein: The inverse diffusion process for the noise image based on the dual-path cross-attention mechanism corresponding to the style encoding and the content text encoding to obtain the target image includes: Through the n-th inverse diffusion network in N cascaded inverse diffusion networks, perform an inverse diffusion process based on the dual-path cross-attention mechanism on the input of the n-th inverse diffusion network, and transmit the n-th inverse diffusion result output by the n-th inverse diffusion network to the (n + 1)-th inverse diffusion network to continue the inverse diffusion process based on the dual-path cross-attention mechanism to obtain the (n + 1)-th inverse diffusion result corresponding to the (n + 1)-th inverse diffusion network; Generate the target image based on the N-th inverse diffusion result corresponding to the N-th inverse diffusion network; Where n is an integer variable starting from 1 and increasing, the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th inverse diffusion network is the noise image, the content text encoding, and the style encoding. When 2 ≤ n < N, the input of the n-th inverse diffusion network is the (n - 1)-th inverse diffusion result output by the (n - 1)-th inverse diffusion network, the content text encoding, and the style encoding.

8. The method according to claim 7, wherein: The n-th inverse diffusion network includes M cascaded sampling networks, and the value of M satisfies 2 ≤ M; The process of performing an inverse diffusion process based on the dual-path cross-attention mechanism on the input of the n-th inverse diffusion network through the n-th inverse diffusion network in N cascaded inverse diffusion networks includes: Through the m-th sampling network in M cascaded sampling networks, perform a sampling process based on the dual-path cross-attention mechanism on the input of the m-th sampling network to obtain the m-th sampling result corresponding to the m-th sampling network, and transmit the m-th sampling result corresponding to the m-th sampling network to the (m + 1)-th sampling network to continue the sampling process based on the dual-path cross-attention mechanism to obtain the (m + 1)-th sampling result corresponding to the (m + 1)-th sampling network; Take the sampling result output by the M-th sampling network as the n-th inverse diffusion result; Wherein, m is an integer variable starting from 1 and increasing incrementally, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th sampling network is the input of the n-th inverse diffusion network, the content text encoding, and the style encoding. When m takes the value of 2 ≤ m < M, the input of the m-th sampling network is the (m - 1)-th sampling result output by the (m - 1)-th sampling network, the content text encoding, and the style encoding.

9. The method according to claim 8, wherein: The step of transmitting the m-th sampling result corresponding to the m-th sampling network to the (m + 1)-th sampling network to continue the sampling process based on the dual-path cross-attention mechanism to obtain the (m + 1)-th sampling result corresponding to the (m + 1)-th sampling network includes: Performing self-attention processing on the m-th sampling result of the m-th sampling network to obtain the self-attention processing result of the (m + 1)-th sampling network; Performing cross-attention processing on the self-attention processing result of the (m + 1)-th sampling network and the content text encoding to obtain the text cross-attention processing result of the (m + 1)-th sampling network; Performing cross-attention processing on the self-attention processing result of the (m + 1)-th sampling network and the style encoding to obtain the style cross-attention processing result of the (m + 1)-th sampling network; Performing fusion processing on the style cross-attention processing result and the text cross-attention processing result of the (m + 1)-th sampling network to obtain the (m + 1)-th sampling result corresponding to the (m + 1)-th sampling network.

10. The method according to claim 9, wherein: The step of performing cross-attention processing on the self-attention processing result of the (m + 1)-th sampling network and the content text encoding to obtain the text cross-attention processing result of the (m + 1)-th sampling network includes: Performing mapping processing on the self-attention processing result of the (m + 1)-th sampling network based on the first query parameter to obtain the first query matrix; Performing mapping processing on the content text encoding based on the first key parameter to obtain the first key matrix; Performing mapping processing on the content text encoding based on the first value parameter to obtain the first value matrix; Performing attention calculation on the first query matrix, the first key matrix, and the first value matrix to obtain the text cross-attention processing result of the (m + 1)-th sampling network.

11. The method according to claim 9, wherein: The step of performing cross-attention processing on the self-attention processing result of the (m + 1)-th sampling network and the style encoding to obtain the style cross-attention processing result of the (m + 1)-th sampling network includes: Performing mapping processing on the self-attention processing result of the (m + 1)-th sampling network based on the second query parameter to obtain the second query matrix; Performing mapping processing on the style encoding based on the second key parameter to obtain the second key matrix; Performing mapping processing on the style encoding based on the second value parameter to obtain the second value matrix; Performing attention calculation on the second query matrix, the second key matrix, and the second value matrix to obtain the style cross-attention processing result of the (m + 1)-th sampling network.

12. An image generation device based on artificial intelligence, the device includes: An acquisition module, configured to acquire content text and acquire a style image having a target style; The encoding module is configured to perform text encoding processing on the content text to obtain the content text encoding the text, and extracting the style code from the style image; The inverse diffusion module is configured to perform inverse diffusion processing on the noise image based on a two-way cross attention mechanism corresponding to the style encoding and the content text encoding to obtain a target image; wherein the target image matches the content of the content text and the target image has the target style.

13. An electronic device, comprising: A memory for storing computer executable instructions; A processor, used to implement the artificial intelligence-based image generation method described in any one of claims 1 to 11 when executing the computer-executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the artificial intelligence-based image generation method according to any one of claims 1 to 11.

15. A computer program product, comprising computer executable instructions, which, when executed by a processor, implement the artificial intelligence-based image generation method according to any one of claims 1 to 11.

Images