Image generation method, and training method and system of image generation model

Through tokenization processing of target text and target style images and combined with image generation model, the problem of difficult generation of complex style images in the existing technology is solved, and the target image is generated from the two dimensions of style and content is realized, which improves the accuracy of image generation Sex and reliability.

CN118941441APending Publication Date: 2024-11-12ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411108186.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to generate images of complex styles because complex styles are difficult to describe in language, resulting in only simple styles of images being generated.

Method used

By obtaining the target text and target style images of the target image, determining the token of the target text and target style, combining the image generation model to generate the target image, and using the style encoder, style word participle, text word participle, text encoder and diffusion model to achieve Generate the target image from both style and content dimensions.

Benefits of technology

It realizes the generation of target images from the two dimensions of style and content to avoid confusion between style and text. The generated image content is consistent with the content indicated by the target text, and the style is consistent with the style indicated by the target style image, which improves image generation accuracy and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941441A_ABST
    Figure CN118941441A_ABST
Patent Text Reader

Abstract

The invention provides an image generation method, and a training method and system of an image generation model. The image generation method comprises the steps that a target text and at least one target style image are obtained, the target text is used for indicating the content of a target image, the at least one target style image is used for indicating the style of the target image, a target text token of the target text and a target style token of the at least one target style image are determined, the target text token is used for representing the content in the target text, the target style token is used for representing the style in the at least one target style image, and a target image is generated according to the target text token and the target style token. By introducing the target style image indicating the style of the target image, the style can be described more accurately and the image of the style can be generated. The target image is generated from the style dimension and the content dimension in a token mode, and confusion between the style and the text can be avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image generation method, an image generation model training method and a system. Background Art

[0002] With the development of artificial intelligence technology, image generation not only focuses on the content of the image, but also on the style of the image.

[0003] In the related art, a user can determine an image generation text based on his / her needs to specify his / her desired image content and image style through the image generation text. The image generation system obtains the image generation text input by the user and generates an image including the image content and image style expected by the user according to the image generation text.

[0004] However, complex styles are difficult to describe in words, so this method can only generate images with simple styles.

[0005] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0006] This specification provides an image generation method, an image generation model training method and a system to avoid the above-mentioned technical problems.

[0007] In a first aspect, this specification provides an image generation method, comprising:

[0008] Obtaining target text and at least one target style image for generating a target image, wherein the target text is used to indicate the content of the target image, and the at least one target style image is used to indicate the style of the target image;

[0009] Determining a target text word-meta token of the target text and a target style token of the at least one target style image, wherein the target text token is used to represent the content in the target text, and the target style token is used to represent the style in the at least one target style image; and

[0010] The target image is generated according to the target text token and the target style token.

[0011] In some embodiments, determining a target style token of the at least one target style image includes:

[0012] Encoding the at least one target style image to obtain a target style feature; and

[0013] The target style feature is tokenized to obtain the target style token.

[0014] In some embodiments, the at least one target style image includes a first target style image and a second target style image; encoding the at least one target style image to obtain a target style feature includes:

[0015] Encoding the first target style image and the second target style image respectively to obtain style features corresponding to each of them; and

[0016] The target style feature is determined according to the respective corresponding style features.

[0017] In some embodiments, determining the target style feature according to the respective corresponding style features includes:

[0018] Based on the preset weights of the respective corresponding style features, the respective corresponding style features are weightedly fused to obtain the target style feature.

[0019] In some embodiments, generating the target image according to the target text token and the target style token includes:

[0020] Concatenate the target text token and the target style token to obtain a target token;

[0021] Encoding the target token to obtain a target feature vector; and

[0022] The target image is generated according to the target feature vector.

[0023] In some embodiments, the method is implemented based on an image generation model, wherein the image generation model is trained to generate an image having the style of the style image and the content of the text;

[0024] The image generation model includes: a style encoder, a style word segmenter, a text word segmenter, a text encoder, and a diffusion model; wherein,

[0025] The style encoder is used to encode the at least one target style image to obtain a target style feature;

[0026] The style tokenizer is used to tokenize the target style feature to obtain the target style token;

[0027] The text tokenizer is used to determine the target text token of the target text;

[0028] The text encoder is used to encode the target token to obtain a target feature vector;

[0029] The diffusion model is used to generate the target image according to the target feature vector.

[0030] In some embodiments, the target text token and the target style token are in the same word vector space.

[0031] In a second aspect, this specification provides a method for training an image generation model, comprising:

[0032] Obtaining a plurality of sample data pairs, wherein one sample data pair includes a sample style image and a sample text for generating a predicted image, wherein the sample text is used to indicate the content of the predicted image, and the sample style image is used to indicate the style of the predicted image;

[0033] Determine a sample text token of the sample text and a sample style token of the sample style image, wherein the sample text token is used to characterize the content in the sample text, and the sample style token is used to characterize the style in the sample style image;

[0034] generating the predicted image according to the sample text token and the sample style token; and

[0035] The image generation model is obtained by minimizing the difference between the predicted image and the pre-labeled real image as a training goal.

[0036] In some embodiments, determining a sample style token of the sample style image includes:

[0037] Encoding the sample style image to obtain sample style features; and

[0038] The style features of the sample are tokenized to obtain the style token of the sample.

[0039] In some embodiments, generating the predicted image according to the sample text token and the sample style token includes:

[0040] Concatenating the sample text token and the sample style token to obtain a sample token, wherein the sample text token and the sample style token are in the same word vector space;

[0041] Encoding the sample token to obtain a sample feature vector; and

[0042] The predicted image is generated according to the sample feature vector.

[0043] In some embodiments, the determining the sample style token of the sample style image includes: encoding the sample style image based on a style encoder to obtain a sample style feature;

[0044] The performing style lemma on the sample style feature to obtain the sample style token includes: performing style lemma on the sample style feature based on a style lemma to obtain the sample style token;

[0045] The determining of the sample text token of the sample text includes: tokenizing the sample text based on a text tokenizer to obtain the sample text token;

[0046] The encoding of the sample token to obtain a sample feature vector includes: encoding the sample token based on a text encoder to obtain the sample feature vector;

[0047] The generating the predicted image according to the sample feature vector: generating the predicted image according to the sample feature vector based on a diffusion model.

[0048] In some embodiments, the image generation model is obtained by taking minimizing the difference between the predicted image and the pre-labeled real image as the training goal, including:

[0049] Taking minimizing the difference between the predicted image and the real image as a training goal, the parameters of the style segmentor are adjusted to obtain the image generation model.

[0050] In some embodiments, the style encoder is trained through multiple training samples, wherein one training sample includes a training style image and annotation information corresponding to the training style image, and the annotation information represents actual style features of the training style image.

[0051] In some embodiments, the training process of the style encoder includes:

[0052] Inputting the training style image into the style encoder to obtain predicted style features; and

[0053] The parameters of the style encoder are updated with minimizing the difference between the actual style feature and the predicted style feature as a training goal.

[0054] In a third aspect, this specification provides an image generation system, comprising:

[0055] at least one storage medium storing at least one set of instructions for generating an image;

[0056] At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and implements the method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0057] In a fourth aspect, this specification provides a training system for an image generation model, comprising:

[0058] at least one storage medium storing at least one instruction set for training an image generation model;

[0059] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of the second aspects is implemented according to the instructions of the at least one instruction set.

[0060] In a fifth aspect, the present specification provides a computer-readable non-temporary storage medium, wherein the computer-readable non-temporary storage medium stores at least one instruction set, and the at least one instruction set is executed by at least one processor to implement a method as described in any one of the first aspect or the second aspect.

[0061] It can be seen from the above technical solutions that the image generation method, image generation model training method and system provided in this specification. Among them, as far as the image generation method is concerned, it is possible to generate a target image from two dimensions: style and content. Exemplarily, from the dimension of style, this specification introduces a target style image that indicates the style of the target image, so that the style is described by using a style image instead of text instructions, which can more accurately and easily describe the specified style and generate an image of the style. From the two dimensions of style and content, the target image is generated from the two dimensions of style and content by tokenization, which can avoid confusion between style and text, thereby achieving that the content in the target image is the content indicated by the target text, and the style is the style indicated by the style image.

[0062] The image generation method, image generation model training method and other functions of the system provided in this specification will be partially listed in the following description. The creative aspects of the image generation method, image generation model training method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A schematic diagram of the principle of generating an image provided by one embodiment of this specification;

[0065] Figure 2 A schematic diagram of the principle of generating an image provided by another embodiment of this specification;

[0066] Figure 3 A schematic diagram of an application scenario of the image generation method according to an embodiment of this specification;

[0067] Figure 4 A schematic diagram of the structure of an image generation system provided in an embodiment of this specification;

[0068] Figure 5 A schematic diagram of a flow chart of an image generation method provided in an embodiment of this specification;

[0069] Figure 6 A schematic diagram of the principle of an image generation method provided by one embodiment of this specification;

[0070] Figure 7 A schematic diagram of the principle of an image generation method provided by one embodiment of this specification;

[0071] Figure 8 Schematic diagram of the flow of the training method of the image generation model of the embodiment of this specification. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0073] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0074] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0075] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar.

[0076] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0077] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0078] To facilitate the reader's understanding of this specification, at least some of the terms in this specification are explained as follows:

[0079] Deep Learning (DL) is a subfield of Machine Learning (ML). It is about learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds.

[0080] Diffusion models are generative models that are used in the fields of deep learning and machine learning to generate complex high-dimensional data, such as images. That is, the diffusion module can generate realistic images. This model is inspired by the molecular diffusion process in physics. It gradually "diffuses" the data into noise through a series of steps, and then learns how to reverse this process, so that the structure of the data can be restored from the noise.

[0081] Low-Rank Adaptation (LoRA) technology is an efficient technology for fine-tuning pre-trained models. It adapts to new tasks by adding and training low-rank matrices without comprehensively updating the weights of the entire model, thereby greatly reducing computing and storage requirements.

[0082] To facilitate readers' understanding of this specification, the application scenarios of this specification are now introduced.

[0083] The technical solution provided in this specification is applicable to application scenarios of generating images from two dimensions, style and content, so that the generated images have both the style and content expected by the user. The application scenarios include but are not limited to artistic creation, games, advertising, design, architecture, education, health care, and news.

[0084] Take the art creation scene as an example: the image generation system can control the generation of artworks from two dimensions, style and content, through the technical solution provided in this specification. For example, paintings and illustrations. Specifically, the image generation system can generate paintings with the oil painting style expected by the user, as well as paintings with the rice field scenery content expected by the user.

[0085] Taking the educational scenario as an example: the image generation system can control the generation of images of characters in the game from two dimensions: style and content through the technical solution provided in this specification. For example, the image generation system generates images with both the cartoon style expected by the user and the boy content expected by the user.

[0086] It is worth noting that the above examples are only used to illustrate the application scenarios to which the technical solutions of this specification can be applied, and cannot be understood as limiting the application scenarios of the technical solutions of this specification.

[0087] In practical applications, images can be generated in the following three ways:

[0088] Method 1: Train a diffusion model to generate an image through the diffusion model.

[0089] For example, the diffusion model can obtain image-generated text including a user-specified style. Figure 1 As shown in the figure, the user specifies the style as sketch style, and the image generates a text representation of a sketch-style image of a little boy. Correspondingly, the diffusion model output is as follows Figure 1 Shown is an image of a boy in sketch style.

[0090] Since complex styles are difficult to describe in words, this method can only generate images with simple styles.

[0091] Method 2: Use a small number of data with similar styles to fine-tune the diffusion model using LoRA technology to generate images based on the fine-tuned diffusion model.

[0092] Using this method, each time an image of a specified style is generated, a corresponding diffusion model needs to be fine-tuned separately, which has too high a cost and usage threshold.

[0093] Method 3: When generating an image based on a diffusion model, an additional adapter is used to control the weight parameters corresponding to the content and style to obtain the output image.

[0094] For example, Figure 2 As shown, the style reference image represents the style that the output image should have - sketch style, and the reference text is used to represent the content that the output image should have - boy. The diffusion model obtains the style features of the style reference image and the text features of the reference text, and determines the output image based on the style features, text features, and the weights corresponding to the reference text and the style reference image.

[0095] Continue reading Figure 2 , λ is the weight of the style reference image. If λ = 0.2, the output image is Figure 2 The output image A described in ; if λ = 1, the output image is Figure 2 The output image B described in .

[0096] That is to say, when λ=0.2, the output image is more affected by the reference text and less affected by the style reference image. As λ increases, the output image is less affected by the reference text and more affected by the style reference image, until the output image B is obtained, whose display content and style are both derived from the style reference image.

[0097] That is, in this method, when the diffusion model generates an image, it is simultaneously controlled by two control signals, namely, text features and style features. These two control signals are easily confused, and one control signal may even be interfered with by the other control signal and lose its control effect.

[0098] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification.

[0099] Therefore, in order to avoid at least one of the above technical problems, this specification proposes a technical conception that has been creatively worked on: when a target image needs to be generated, a target text for indicating the content of the target image and a target style image for indicating the style of the target image are obtained; tokens corresponding to the target style image and the target text are obtained, and the target image is generated according to the tokens corresponding to the target style image and the target text. A token can be understood as the smallest meaningful unit, which can be a word, a number, a symbol, or other string fragments that can be processed separately.

[0100] Exemplarily, the technical solution provided in this specification can be understood as that the image generation system can generate a target image from two dimensions: style and content. From the perspective of style, this specification introduces a target style image indicating the style of the target image, and the image generation system determines the target style token of the target style image. From the perspective of text, the image generation system determines the target content token (or target text token) of the target text. Accordingly, the image generation system generates a target image by combining the target style token and the target text token.

[0101] On the one hand, by using style images instead of text instructions to describe the style, it is possible to more accurately and easily describe the specified style and generate images of that style. On the other hand, by generating the target image from the two dimensions of style and content through tokenization, it is possible to avoid confusion between style and text, so that the content in the target image is the content indicated by the target text, and the style is the style indicated by the style image.

[0102] Figure 3 FIG. 1 is a schematic diagram of an application scenario of the image generation method of an embodiment of the present specification, wherein the image generation method of the present specification can be applied to Figure 3 The application scenario 300 shown. Figure 3 As shown, the application scenario 300 may include a target user 301 , a client 302 , a server 303 , and a network 304 .

[0103] The target user 301 may be a user who initiates a request to generate a target image. For example, the target user 301 may perform a target operation on the client 302 to trigger the generation of the target image. In some embodiments, the target user 301 may upload a target text and a target style image on the client 302 to trigger the generation of the target image.

[0104] The client 302 may be an electronic device that provides interactive functions to the target user 301. For example, the client 302 may provide an interactive interface to the target user 301, and the target user 301 may perform interactive operations in the interactive page. In some embodiments, the client 302 executes the image generation method described in this specification in response to detecting the target operation of generating a target image triggered by the target user 301. At this time, the client 302 may store data or instructions for executing the image generation method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 302 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work, so as to execute the image generation method described in this specification.

[0105] In some embodiments, the client 302 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the built-in device in a motor vehicle may include an onboard computer, an onboard TV, or the like. In some embodiments, the client 302 may include a collection device for collecting target text and target style images.

[0106] In some embodiments, the client 302 may be installed with one or more applications (Application, APP). APP can provide the target user 301 with the ability and interface to interact with the outside world through the network 304. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, the client 302 may be installed with a target APP. The target APP can collect target text and target style images for the client 302.

[0107] like Figure 3 As shown, the client 302 can be connected to the server 303 for communication. The server 303 can be connected to one client 302 for communication, or can be connected to multiple clients 302 for communication. In some embodiments, the client 302 can interact with the server 303 through the network 304 to receive or send messages, etc. For example, the client 302 can interact with the server 303 through the network 304 to send the target text and the target style image to the server 303.

[0108] The server 303 may be a server that provides various services. For example, the server 303 may be a server that provides image generation services to the client 302. The server 303 may be a cloud server or a local server. The server 303 may be connected to a client 302 for communication and receive data sent by the client 302, or may be connected to multiple clients 302 for communication and receive data sent by each client 302.

[0109] In some embodiments, the image generation method described in this specification can be executed on the server 303. In this case, the server 303 can store data or instructions for executing the image generation method described in this specification, and can execute or be used to execute the data or instructions. The server 303 can include a hardware device with a data information processing function and the necessary programs required to drive the hardware device to work.

[0110] The network 304 is a medium for providing a communication connection between the client 302 and the server 303. The network 304 can facilitate the exchange of information or data. Figure 3 As shown, the client 302 and the server 303 can be connected to the network 304 respectively, and transmit information or data to each other through the network 304.

[0111] In some embodiments, the network 304 may be any type of wired or wireless network, or a combination thereof. For example, the network 304 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0112] In some embodiments, network 304 may include one or more network access points. For example, network 304 may include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of client 302 and server 303 may connect to network 304 to exchange data or information.

[0113] It is worth mentioning that Figure 3 The number of the client 302, server 303 and network 304 is only illustrative. According to the implementation requirements, there can be any number of clients 302, servers 303 and networks 304. The image generation method provided in this specification can be executed completely on the client 302, can be executed completely on the server 303, or can be executed partially on the client 302 and partially on the server 303.

[0114] That is to say, Figure 3 and for Figure 3The above description is only used to exemplify the application scenarios to which the image generation method of this specification may be applicable, and should not be understood as a limitation on the application scenarios.

[0115] Figure 4 The hardware structure diagram of an image generation system 400 provided according to an embodiment of the present specification is shown. The image generation system 400 can execute the image generation method described in the present specification. The image generation method is introduced in other parts of the present specification. When the image generation method is executed on the client 302, the image generation system 400 can be the client 302. When the image generation method is executed on the server 303, the image generation system 400 can be the server 303. When the image generation method is partially executed on the client 302 and partially executed on the server 303, the image generation system 400 can include the client 302 and the server 303.

[0116] like Figure 4 As shown, the image generation system 400 may include at least one storage medium 403 and at least one processor 402. In some embodiments, the image generation system 400 may further include a communication port 404 and an internal communication bus 401. The image generation system 400 may further include an I / O component 405.

[0117] The internal communication bus 401 can connect different system components. For example, the internal communication bus 401 can connect the storage medium 403, the processor 402, the communication port 404 and the I / O component 405.

[0118] I / O components 405 support input / output between image generation system 400 and other components.

[0119] The communication port 404 is used for data communication between the image generation system 400 and the outside world. For example, the communication port 404 can be used for data communication between the image generation system 400 and the network 304. The communication port 404 can be a wired communication port or a wireless communication port.

[0120] The storage medium 403 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 4031, a read-only storage medium (ROM) 4032, or a random access storage medium (RAM) 4033. The storage medium 403 also includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the image generation method provided in this specification.

[0121] At least one processor 402 may be connected in communication with at least one storage medium 403. At least one processor 402 is used to execute the above-mentioned at least one instruction set. When the image generation system 400 is running, at least one processor 402 reads the at least one instruction set and executes the image generation method provided in this specification according to the instructions of the at least one instruction set. The processor 402 may execute all the steps included in the image generation method. The processor 402 may be in the form of one or more processors. In some embodiments, the processor 402 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0122] For the purpose of illustration only, only one processor 402 is shown in the image generation system 400 in the figure. However, it should be noted that the image generation system 400 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 402 of the image generation system 400 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 402 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0123] See also Figure 5 , Figure 5 This is a flow chart of the image generation method provided in the embodiment of this specification. Figure 5 As shown, the method includes the following S501 to S503:

[0124] S501: Obtain target text and at least one target style image for generating a target image, wherein the target text is used to indicate the content of the target image, and the at least one target style image is used to indicate the style of the target image.

[0125] Exemplarily, the target image can be understood as an image that the target user expects to obtain.

[0126] The target text includes the image content in the target image that the target user expects. Figure 6As shown in , the target text is "an image of a little boy". That is, the target user expects the image generation system to generate a target image with the image content of a little boy. Figure 7 As shown, the target text is " Figure 1 That is, the target user expects the image generation system to generate a target image with the image content of a dog.

[0127] The number of target style images may be one or more. When the number of target style images is more than one, different target style images represent different styles. The target style image includes the image style of the image content in the target image that the target user expects.

[0128] For example, Figure 6 As shown, the number of target style images is one, and the style of the target style image is a sketch style. That is, the target user expects the image generation system to generate a target image with a sketch style.

[0129] For example, Figure 7 As shown, the number of target style images is multiple, and specifically can be 2. Among them, the style of one target style image is sketch style, and the style of one target style image is color style. That is, the target user expects the image generation system to generate a target image with both sketch style and color style.

[0130] This embodiment does not limit the way in which the image generation system obtains the target text and the target style image. Take the image generation system obtaining the target style image as an example:

[0131] In one example, the image generation system can be connected to other systems and receive target style images sent by other systems.

[0132] For example, Figure 3 As an example of the application scenario shown in FIG. 1 , the image generation system can be as follows: Figure 3 The server 303 shown in FIG. 1 may be a server 303, and other devices may be Figure 3 In the client 302 shown, the target user 301 can input the target style image on the client 302 through an APP or a non-APP method to trigger the client 302 to send the target style image to the server 303.

[0133] In another example, the image generation system may provide a tool for loading a target style image, so that the target style image is transmitted to the image generation system through the tool for loading a target style image.

[0134] Among them, the tool for loading the target style image can be an interface for connecting to an external device, such as an interface for connecting to other storage devices, through which the target style image transmitted by the external device is obtained; the tool for loading the target style image can also be a display device, such as the image generation system can output an interface for loading the target style image function on the display device, so as to import the target style image into the image generation system through the interface.

[0135] S502: Determine a target text token of a target text and a target style token of at least one target style image, wherein the target text token is used to characterize the content in the target text, and the target style token is used to characterize the style in at least one target style image.

[0136] This step can be understood as follows: on the one hand, the image generation system determines the token of the text dimension (ie, the target text token) based on the target text, and extracts information for generating the content in the target image from the target text in a meaningful and small-granular manner.

[0137] On the other hand, the image generation system determines a token of the style dimension (i.e., a target style token) based on the target style image, and extracts information for generating the style of the target image from the target style image in a meaningful and small-granular manner.

[0138] In some embodiments, the image generation system determines the target style token of the at least one target style image including the following steps 1 and 2:

[0139] Step 1: Encode the at least one target style image to obtain target style features.

[0140] The target style features may be understood as features of the target style image, such as color distribution, texture, shape, structure, edge, light and dark contrast, so as to characterize the style information of the target style image through these features.

[0141] like Figure 6 and Figure 7 As shown, the image generation system can be implemented based on an image generation model, and the image generation model is trained to generate an image with the style of the style image and the content of the text. Corresponding to this embodiment, the image generation model is trained to generate a target image with both the image style represented by the target style image and the image content represented by the target text. Figure 6 and Figure 7 As shown, the image generation model may include a style encoder.

[0142] For example, the at least one target style image is taken as an example. Figure 6 As shown, the target style image is the input I of the style encoder S The style encoder encodes the target style image and obtains the target style feature f S .

[0143] In combination with the above analysis, it can be known that in other embodiments, the number of the at least one target style image can also be multiple. Taking the at least one target style image as two, which are the first target style image and the second target style image, as an example, step 1 may include the following sub-steps 1 and 2:

[0144] Sub-step 1: Encode the first target style image and the second target style image respectively to obtain their corresponding style features.

[0145] Sub-step 2: Determine the target style features according to their corresponding style features.

[0146] For example, combining the above examples and Figure 7 , the input of the style encoder includes the first target style image and the second target style image. The style encoder encodes the first target style image to obtain the first style feature f S1 , encode the second target style image and obtain the second style feature f S2 , and according to the first style feature f S1 and the second style feature f S2 Determine the target style feature f S For example, the first style feature f can be fused or concatenated S1 and the second style feature f S2 , get the target style feature f S .

[0147] In some other embodiments, when the number of the at least one target style image is multiple, the number of style encoders may also be multiple, such as the number of target style images is the same as the number of style encoders.

[0148] For example, if the number of the at least one target style image is 2, and they are the first target style image and the second target style image respectively. The number of style encoders may also be 2. One style encoder encodes the first target style image to obtain a first style feature. Another style encoder encodes the first target style image to obtain a second style feature. After the first style feature and the second style feature are fused, the target style feature is obtained.

[0149] In this embodiment, the first style feature is the style feature of the first target style image, and the second style feature is the style feature of the second target style image. The target style feature includes both the style feature of the first target style image and the style feature of the second target style image. Therefore, when an image is generated based on the target style feature, a target image with multiple styles can be obtained.

[0150] In some embodiments, the above sub-step 2 may include: based on the preset weights of the respective corresponding style features, performing weighted fusion on the respective corresponding style features to obtain the target style feature.

[0151] This embodiment does not limit the preset weights. For example, the preset weights may be pre-configured by the target user in the image processing system.

[0152] In combination with the above analysis, the corresponding style features include a first style feature and a second style feature. Accordingly, the preset weight may include a first weight corresponding to the first style feature and a second weight corresponding to the second style feature. The first style feature and the second style feature are weighted and fused with the first weight and the second weight as weighting coefficients to obtain the target style feature.

[0153] Among them, different weights represent different style tendencies of the target image. For example, if the first weight is greater than the second weight, it means that the style of the target image is more inclined to the style of the first target style image. On the contrary, if the first weight is less than the second weight, it means that the style of the target image is more inclined to the style of the second target style image.

[0154] like Figure 7 As shown in the figure, if the first weight of the color style indicated by the first target style image is 0.8, and the second weight of the sketch style indicated by the second target style image is 0.2, the style of the target image is more inclined to the color style. If the first weight of the color style indicated by the first target style image is 0.2, and the second weight of the sketch style indicated by the second target style image is 0.8, the style of the target image is more inclined to the sketch style.

[0155] In this embodiment, the image generation system determines the target style features by combining preset weights, so that, for example, the target user can adjust the importance of different styles according to specific needs, thereby generating a more personalized style fusion result.

[0156] Step 2: Perform style tokenization on the target style features to obtain the target style token.

[0157] Among them, lemmatization can be understood as breaking down the corresponding object into a series of units, which can be words, characters or specific tags. Combined with the above analysis, these units are called tokens.

[0158] Corresponding to this embodiment, style tokenization can be understood as decomposing the target style features into a series of units, which can be words, characters or specific tags, to obtain the target style token.

[0159] Continue reading Figure 6 and Figure 7 The image generation model also includes a text tokenizer and a style tokenizer. The input of the text tokenizer is the target text, and the output is the target text token (e t ). The input of the style segmenter is the target style feature f S , the output is the target style token (e S ).

[0160] Combining the above analysis of step 1 and step 2, it can be seen that in this embodiment, the target style feature captures and quantifies the key visual attributes of the target style image, including but not limited to color, texture, shape and composition. Therefore, the target style token obtained by style tokenization can reliably and effectively represent the style information of the target style image.

[0161] In some embodiments, the target text token and the target style token are in the same word vector space.

[0162] Exemplarily, the style tokenizer determines a target style token of the target style feature, and maps the target style token to a word embedding space aligned with the target text token, so that the target style token and the target text token are in the same word vector space.

[0163] The same word vector space can be understood as the same coordinate system. Accordingly, the target style token and the target text token can be compared and fused in the same coordinate system.

[0164] Therefore, in this embodiment, by aligning the target style token and the target text token to the same vector space, the operations on the image style and the image content are performed in the same coordinate system. Therefore, the image style and the image content can be better understood to avoid confusion between the image style and the image content. That is, in the process of generating the target image, no interference between the image style and the image content can be achieved, so that the content of the target image is highly consistent with the image content indicated by the target text, and the style of the target image is highly consistent with the image style indicated by the target style image.

[0165] S503: Generate a target image according to the target text token and the target style token.

[0166] Combined with the above analysis, this step can be understood as: the image generation system combines the content information (i.e., the token of the text dimension) and the style information (i.e., the token of the style dimension) to jointly generate the target image. Figure 6 and Figure 7 As shown, the content in the target image is highly consistent with the content represented by the target text, and the style of the target image is highly consistent with the style represented by the at least one target style image. That is, the accuracy and reliability of generating the target image can be achieved.

[0167] In some embodiments, S503 may include the following steps 1 to 3:

[0168] Step 1: Concatenate the target text token and the target style token to get the target token.

[0169] For example, Figure 6 and Figure 7 As shown in the figure, the target text token and the target style token are added to obtain the target token. That is, the target token includes both the target text token representing the image content and the target style token representing the image style.

[0170] Step 2: Encode the target token to obtain the target feature vector.

[0171] For example, Figure 6 and Figure 7 As shown, the image generation model also includes a text encoder. The input of the text encoder is the target token, and the output is the target feature vector.

[0172] Step 3: Generate a target image based on the target feature vector.

[0173] For example, Figure 6 and Figure 7 As shown in FIG, the image generation model also includes a diffusion model. The input of the diffusion model is the target feature vector, and the output is I o is the target image.

[0174] Combined with the analysis of steps 1 to 3 above, it can be seen that the target token includes both the token that represents the image content and the token that represents the image style. Therefore, the target feature vector generated based on the target token includes both the feature vector that represents the image content and the feature vector that represents the image style. Therefore, the target image generated based on the target feature vector can reliably represent the image content indicated by the target text and the image style indicated by the target style image. And because the target image is generated based on tokens (including style tokens and content tokens), the confusion between image content and image style can be avoided, that is, the decoupling between image content and image style can be achieved, thereby improving the accuracy and reliability of the target image.

[0175] Combined with the above analysis, it can be seen that the image generation method of the above example can be implemented based on the image generation model. Accordingly, the present disclosure also provides a training method for the image generation model (hereinafter referred to as the training method). Figure 8 As shown, the training method includes the following S801-S804:

[0176] S801: Obtain multiple sample data pairs, wherein one sample data pair includes a sample style image and sample text for generating a predicted image, the sample text is used to indicate the content of the predicted image, and the sample style image is used to indicate the style of the predicted image.

[0177] Among them, the execution subject of this embodiment can be a training system of the image generation model (hereinafter referred to as the training system). The training system can be the same system as the image generation system, or it can be a different system. If the training system is a system different from the image generation system, after the training system obtains the image generation model based on the training method of this specification, the image generation model can be transmitted to the image generation system; or the training system can provide the image generation system with an interface for calling the image generation model, and so on.

[0178] It is understandable that, in order to avoid tedious description, the present embodiment will not repeat the same or similar technical features as those in the above embodiment. For example, regarding the structure of the training system, reference may be made to the above description of the structure of the image generation system. For another example, regarding the implementation principle of S801-S803, reference may be made to the description of S501 to S503 of S801 to S803 in the above example.

[0179] S802: Determine a sample text token of the sample text and a sample style token of the sample style image, wherein the sample text token is used to characterize the content in the sample text, and the sample style token is used to characterize the style in the sample style image.

[0180] In some embodiments, determining a sample style token of a sample style image includes: encoding the sample style image to obtain a sample style feature, and performing style tokenization on the sample style feature to obtain a sample style token.

[0181] Combined with the above analysis, it can be seen that the training method of this description can be applied to a preset network, which includes: a style encoder, a style segmenter, a text segmenter, a text encoder, and a diffusion model.

[0182] In some embodiments, encoding the sample style image to obtain the sample style feature includes: encoding the sample style image based on a style encoder to obtain the sample style feature.

[0183] Performing style tokenization on the sample style feature to obtain the sample style token includes: performing style tokenization on the sample style feature based on a style tokenizer to obtain the sample style token.

[0184] In some embodiments, determining a sample text token of the sample text includes: tokenizing the sample text based on a text tokenizer to obtain the sample text token.

[0185] S803: Generate a predicted image according to the sample text token and the sample style token.

[0186] In some embodiments, S803 includes: concatenating a sample text token and a sample style token to obtain a sample token, wherein the sample text token and the sample style token are in the same word vector space; encoding the sample token to obtain a sample feature vector; and generating a predicted image according to the sample feature vector.

[0187] In some embodiments, encoding the sample token to obtain the sample feature vector includes: encoding the sample token based on a text encoder to obtain the sample feature vector.

[0188] In some embodiments, the predicted image is generated according to the sample feature vector: the predicted image is generated according to the sample feature vector based on a diffusion model.

[0189] S804: Taking minimizing the difference between the predicted image and the pre-labeled real image as the training goal, an image generation model is obtained.

[0190] In some embodiments, S804 includes: taking minimizing the difference between the predicted image and the real image as a training goal, adjusting the parameters of the style segmentor to obtain an image generation model.

[0191] Exemplarily, the parameters of the style encoder, text segmenter, text encoder, and diffusion model are frozen, that is, during the training process, the parameters of the style encoder, text segmenter, text encoder, and diffusion model are not adjusted. The parameters of the style segmenter are adjusted until the difference between the predicted image and the real image is minimized, or the number of training iterations reaches a preset threshold, and the image generation model is obtained.

[0192] In contrast, the style segmenter of this embodiment is used to infer the style in the image to obtain the corresponding style token. There is no need to combine the language description, nor is there a need to train a special model for images of different styles. That is, through the style segmenter of the embodiment, any image of any style can be tokenized to map to a word vector space aligned with the token of the corresponding text.

[0193] In some embodiments, the style encoder is trained by multiple training samples, wherein one training sample includes a training style image and annotation information corresponding to the training style image, and the annotation information represents actual style features of the training style image.

[0194] Exemplarily, the training style image can be an image of any artistic style, such as impressionism, abstract art, realism, etc. The annotation information is a label or description of the style characteristics of the training style image. The annotation information can be manually marked or obtained from other means.

[0195] In this embodiment, the style encoder constructs a mapping function by learning a large number of training samples, which can map any input image to a vector space of fixed dimension. The points of this vector space represent the style features of the image.

[0196] In some embodiments, the training process of the style encoder includes the following steps 1 and 2:

[0197] Step 1: Input the training style image into the style encoder to obtain the predicted style features.

[0198] Step 2: Update the parameters of the style encoder with the goal of minimizing the difference between the actual style features and the predicted style features.

[0199] Exemplarily, the training style image is input into the style encoder, and the style encoder generates a predicted style feature vector (i.e., predicted style feature). The predicted style feature is compared with the annotation information (i.e., the actual style feature), and the loss between the two is calculated, such as the mean square error (MSE) or other similarity metrics. According to the calculated loss, the parameters of the style encoder are adjusted using the gradient descent method or its variants (such as the Adam optimizer) to minimize the loss until the style encoder can achieve satisfactory performance on the training sample.

[0200] It is worth noting that the above examples are only used to illustrate the possible implementation methods of the image generation method and the image generation model training method of this specification, and cannot be understood as limiting the implementation methods of the image generation method and the image generation model training method of this specification. Exemplarily, based on the above technical concept, some of the above technical features can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and the order of the above examples can be adjusted to obtain a new embodiment, etc., which will not be listed here one by one.

[0201] Based on the above technical concept, this specification also provides a computer-readable non-temporary storage medium, which stores at least one instruction set. When the at least one instruction set is executed by a processor, the steps of the image generation method and the image generation model training method described in this specification are implemented.

[0202] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes a program code. When the program product is run on the image generation system 400, the program code is used to make the image generation system 400 perform the steps of the image generation method described in this specification. The program product for implementing the above method may include program code using a portable compact disk read-only memory (CD-ROM) and may be run on the image generation system 400. However, the program product of this specification is not limited to this. In this specification, a readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system or used in combination with it. The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the image generation system 400, partially on the image generation system 400, as a stand-alone software package, partially on the image generation system 400 and partially on a remote image generation system 400, or entirely on a remote image generation system 400.

[0203] When the program product is run on a training system, the program code is used to enable the training system to execute the steps of the training method for the image generation model described in this specification. The specific implementation principle can be found in the above examples and will not be repeated here.

[0204] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0206] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0207] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0208] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0209] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for generating an image, comprising: Obtaining target text and at least one target style image for generating a target image, wherein the target text is used to indicate the content of the target image, and the at least one target style image is used to indicate the style of the target image; Determining a target text word-meta token of the target text and a target style token of the at least one target style image, wherein the target text token is used to characterize the content in the target text, and the target style token is used to characterize the style in the at least one target style image; and The target image is generated according to the target text token and the target style token.

2. The method according to claim 1, wherein: Determining a target style token of the at least one target style image includes: Encoding the at least one target style image to obtain a target style feature; and The target style feature is tokenized to obtain the target style token.

3. The method according to claim 2, wherein: The at least one target style image includes a first target style image and a second target style image; and encoding the at least one target style image to obtain a target style feature includes: Encoding the first target style image and the second target style image respectively to obtain style features corresponding to each of them; and The target style feature is determined according to the respective corresponding style features.

4. The method according to claim 3, wherein: The determining the target style feature according to the respective corresponding style features includes: Based on the preset weights of the respective corresponding style features, the respective corresponding style features are weightedly fused to obtain the target style feature.

5. The method according to claim 2, wherein: The generating the target image according to the target text token and the target style token includes: Concatenate the target text token and the target style token to obtain a target token; Encoding the target token to obtain a target feature vector; and The target image is generated according to the target feature vector.

6. The method according to claim 5, wherein: The method is implemented based on an image generation model, wherein the image generation model is trained to generate an image having the style of the style image and the content of the text; The image generation model includes: a style encoder, a style word segmenter, a text word segmenter, a text encoder, and a diffusion model; wherein, The style encoder is used to encode the at least one target style image to obtain a target style feature; The style tokenizer is used to tokenize the target style feature to obtain the target style token; The text tokenizer is used to determine the target text token of the target text; The text encoder is used to encode the target token to obtain a target feature vector; The diffusion model is used to generate the target image according to the target feature vector.

7. The method according to claim 1, wherein: The target text token and the target style token are in the same word vector space.

8. A method for training an image generation model, comprising: Obtaining a plurality of sample data pairs, wherein one sample data pair includes a sample style image and a sample text for generating a predicted image, wherein the sample text is used to indicate the content of the predicted image, and the sample style image is used to indicate the style of the predicted image; Determine a sample text token of the sample text and a sample style token of the sample style image, wherein the sample text token is used to characterize the content in the sample text, and the sample style token is used to characterize the style in the sample style image; generating the predicted image according to the sample text token and the sample style token; and The image generation model is obtained by minimizing the difference between the predicted image and the pre-labeled real image as a training goal.

9. The method according to claim 8, wherein: Determining a sample style token of the sample style image includes: Encoding the sample style image to obtain sample style features; and The style features of the sample are tokenized to obtain the style token of the sample.

10. The method according to claim 9, wherein: The generating the predicted image according to the sample text token and the sample style token includes: Concatenating the sample text token and the sample style token to obtain a sample token, wherein the sample text token and the sample style token are in the same word vector space; Encoding the sample token to obtain a sample feature vector; and The predicted image is generated according to the sample feature vector.

11. The method according to claim 10, wherein: The determining of the sample style token of the sample style image includes: encoding the sample style image based on a style encoder to obtain a sample style feature; The performing style lemma on the sample style feature to obtain the sample style token includes: performing style lemma on the sample style feature based on a style lemma to obtain the sample style token; The determining of the sample text token of the sample text includes: tokenizing the sample text based on a text tokenizer to obtain the sample text token; The encoding of the sample token to obtain a sample feature vector includes: encoding the sample token based on a text encoder to obtain the sample feature vector; The generating the predicted image according to the sample feature vector: generating the predicted image according to the sample feature vector based on a diffusion model.

12. The method according to claim 11, wherein: The image generation model is obtained by taking minimizing the difference between the predicted image and the pre-labeled real image as the training goal, including: Taking minimizing the difference between the predicted image and the real image as a training goal, the parameters of the style segmentor are adjusted to obtain the image generation model.

13. The method according to claim 11, wherein: The style encoder is obtained by training a plurality of training samples, wherein one training sample includes a training style image and annotation information corresponding to the training style image, and the annotation information represents actual style features of the training style image.

14. The method according to claim 13, wherein: The training process of the style encoder includes: Inputting the training style image into the style encoder to obtain predicted style features; and The parameters of the style encoder are updated with minimizing the difference between the actual style feature and the predicted style feature as a training goal.

15. An image generation system, comprising: at least one storage medium storing at least one set of instructions for generating an image; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method according to any one of claims 1 to 7 is implemented according to the instructions of the at least one instruction set.

16. A training system for an image generation model, comprising: at least one storage medium storing at least one instruction set for training an image generation model; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as claimed in any one of claims 8 to 14 is implemented according to the instructions of the at least one instruction set.

Citation Information

Cited By

  • Image generation method, image generation model training method, and related apparatus

    WO2026174811A1