Picture generation processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a picture generation processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a picture description text, wherein the picture description text comprises a name of a first character; encoding the picture description text to obtain a picture description text code; the picture description text code and the first picture feature of the noise picture are input into a bypass branch in a picture generation model, and a second picture feature output by the bypass branch is obtained; inputting the picture description text code, the second picture feature and the first picture feature into a de-noising branch in a picture generation model to obtain a third picture feature output by the de-noising branch; and performing decoding processing based on the third picture feature to obtain a picture corresponding to the picture description text.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for image generation and processing. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, AI drawing applications are becoming more and more widespread. For example, when a user enters a description of a specific person, the image generation model can automatically generate a picture of the specific person.

[0003] In the related art, the training process of a picture generation model for generating pictures of specific characters can be as follows: first, an initial picture generation model is trained through general character picture-description text pairs, and then, the initial picture generation model is retrained through specific character picture-description text pairs to obtain a picture generation model that can generate pictures of specific characters.

[0004] Since the number of samples of specific person pictures and description texts is limited, retraining the initial picture generation model through specific person picture-description text pairs will cause the picture generation model to overfit and can only generate specific person pictures similar to the training samples, thereby affecting the generalization of the model and the picture generation effect is poor. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, storage medium and program product for image generation and processing, which can improve the effect of generating images of specific people. The technical solution is as follows:

[0006] In one aspect, a method for image generation and processing is provided, the method being executed by a computer device, the method comprising:

[0007] Obtaining a picture description text, wherein the picture description text includes the name of the first person;

[0008] Encoding the picture description text to obtain the picture description text code;

[0009] The picture description text encoding and the first picture feature of the noise picture are input into a bypass branch in the picture generation model to obtain a second picture feature output by the bypass branch; the bypass branch is obtained by training a first picture-text pair; the first picture sample in the first picture-text pair includes the first person, and the first text sample in the first picture-text pair includes the name of the first person;

[0010] Input the picture description text encoding, the second picture feature, and the first picture feature into a denoising branch in the picture generation model to obtain a third picture feature output by the denoising branch;

[0011] Decoding is performed based on the third picture feature to obtain a picture corresponding to the picture description text.

[0012] In one aspect, a method for image generation and processing is provided, the method being executed by a computer device, the method comprising:

[0013] Encoding the first text sample to obtain a first text encoding sample; the first text sample includes the name of the first person;

[0014] Inputting the first text encoding sample and the first picture feature sample of the first noise picture sample into a bypass branch in the picture generation model, and obtaining a second picture feature sample output by the bypass branch;

[0015] Inputting the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model to obtain a third picture feature sample output by the denoising branch;

[0016] Performing decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample;

[0017] Obtaining a first loss function value based on a difference between the first predicted picture and the first picture sample;

[0018] Based on the first loss function value, updating the parameters of the bypass branch;

[0019] Among them, the picture generation model after the bypass branch is updated is used to generate a picture corresponding to the picture description text based on the input picture description text; the picture description text contains the name of the first person.

[0020] On the other hand, a picture generation processing device is provided, the device comprising:

[0021] A text acquisition module, used to acquire a picture description text, wherein the picture description text includes the name of the first person;

[0022] An encoding module, used to encode the picture description text to obtain the picture description text code;

[0023] A bypass processing module is used to input the picture description text encoding and the first picture feature of the noise picture into a bypass branch in the picture generation model to obtain a second picture feature output by the bypass branch; the bypass branch is obtained by training a first picture-text pair; the first picture sample in the first picture-text pair includes the first person, and the first text sample in the first picture-text pair includes the name of the first person;

[0024] a denoising processing module, configured to input the picture description text encoding, the second picture feature, and the first picture feature into a denoising branch in the picture generation model, and obtain a third picture feature output by the denoising branch;

[0025] A decoding module is used to perform decoding processing based on the third picture feature to obtain the picture corresponding to the picture description text.

[0026] On the other hand, a picture generation processing device is provided, the device comprising:

[0027] An encoding module, used for encoding a first text sample to obtain a first text encoding sample; the first text sample includes the name of a first person;

[0028] A bypass processing module, used for inputting the first text encoding sample and the first picture feature sample of the first noise picture sample into a bypass branch in the picture generation model, and obtaining a second picture feature sample output by the bypass branch;

[0029] a denoising processing module, configured to input the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model, and obtain a third picture feature sample output by the denoising branch;

[0030] A decoding module, configured to perform decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample;

[0031] a loss calculation module, configured to obtain a first loss function value based on a difference between the first predicted image and the first image sample;

[0032] A parameter updating module, used for updating the parameters of the bypass branch based on the first loss function value;

[0033] Among them, the picture generation model after the bypass branch is updated is used to generate a picture corresponding to the picture description text based on the input picture description text; the picture description text contains the name of the first person.

[0034] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned image generation processing method.

[0035] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement the above-mentioned image generation processing method.

[0036] On the other hand, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the image generation processing method provided in the above-mentioned various optional implementations.

[0037] The technical solution provided by this application may have the following beneficial effects:

[0038] After obtaining the picture description text containing the name of the first person, the picture description text is encoded to obtain the picture description text code, and then the picture description text code and the first picture feature of the noise picture are processed through a bypass branch to obtain a second picture feature, and then the second picture feature is input into the denoising branch to obtain a third picture feature, and finally decoded to obtain the picture corresponding to the picture description text; wherein, since the bypass branch is trained by the picture containing the first person and the text containing the name of the first person, and is independent of the network branch other than the denoising branch, therefore, the model in the above scheme can accurately generate pictures containing specific people while retaining the model generalization ability of the denoising branch, thereby improving the effect of generating pictures of specific people. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] Figure 1 is a schematic diagram of a system used in the image generation and processing method provided in this application;

[0041] Figure 2 is a flowchart of a method for image generation and processing shown in an exemplary embodiment of the present application;

[0042] Figure 3 is a schematic diagram of a picture-text pair involved in an embodiment of the present application;

[0043] Figure 4It is an architecture diagram of a diffusion model for generating images through texts involved in an embodiment of the present application;

[0044] Figure 5 It is a framework diagram of a picture generation model including a bypass branch involved in an embodiment of the present application;

[0045] Figure 6 is a schematic diagram of the structure of the QKV network unit involved in the embodiment of the present application;

[0046] Figure 7 is a schematic diagram of the structure of the residual block involved in this application;

[0047] Figure 8 is a schematic diagram of the structure of the space converter involved in this application;

[0048] Fig. 9 is a schematic diagram of the structure of the bypass branch and the denoising branch involved in the embodiment of the present application;

[0049] Fig.10 is a schematic diagram of the structure of the bypass branch and the denoising branch involved in the embodiment of the present application;

[0050] Fig.11 is a flowchart of a method for image generation and processing shown in an exemplary embodiment of the present application;

[0051] Fig.12 is a schematic diagram of a picture-text pair involved in an embodiment of the present application;

[0052] Fig.13 is a flowchart of a method for image generation and processing shown in an exemplary embodiment of the present application;

[0053] Fig.14 is a flowchart of a method for image generation and processing shown in an exemplary embodiment of the present application;

[0054] Fig.15 It is a schematic diagram of the interface of model training and application involved in the embodiment of the present application;

[0055] Fig.16 It is a framework flow chart of a method for generating and processing a scene picture of a film or TV series involved in an embodiment of the present application;

[0056] Fig.17 It is a framework diagram of the model training process involved in this application;

[0057] Fig.18 is a flow chart for generating a character representation library involved in this application;

[0058] Fig.19 It is a schematic diagram of the character representation replacement process in the text encoding involved in this application;

[0059] Fig. 20 is a flow chart of the model application process involved in this application;

[0060] Fig.21 It is a schematic diagram of the service process when the application is involved in this application;

[0061] Fig. 22 is a block diagram of a picture generation and processing device provided by an embodiment of the present application;

[0062] Fig.23 is a block diagram of a picture generation and processing device provided by an embodiment of the present application;

[0063] Fig.24 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0066] The embodiments of the present application provide a method, device, equipment, storage medium and program for image generation and processing, which relate to the field of artificial intelligence technology and can improve the accuracy of image generation. For ease of understanding, several terms involved in the present application are explained below.

[0067] 1) Artificial Intelligence (AI)

[0068] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0069] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The display device containing an image acquisition component shown in this application mainly involves computer vision technology, machine learning / deep learning, autonomous driving, smart transportation, and other directions.

[0070] 2) Machine Learning (ML)

[0071] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0072] 3) Diffusion model Vincent diagram

[0073] The generative model based on the diffusion process inputs the target text (text). The model performs a series of operations on a random noise image x and generates a predicted image Y related to the text under the cross-attention of the target text.

[0074] 4) Noise Label

[0075] When labeling images, some labels are noisy labels, which means that all labels of the image are not completely correct and some labels are wrong.

[0076] 5) Secondary creation

[0077] For original works (such as movies and TV series), other users re-create the characters in the original works to obtain new works including the same characters, such as new pictures.

[0078] Figure 1 A schematic diagram of a system used in a method for generating and processing an image provided by an exemplary embodiment of the present application is shown. Figure 1 As shown, the system includes: a server 110 and a terminal 120.

[0079] Among them, the above-mentioned server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0080] The server 110 may include a server that is deployed with an image generation and processing system and provides image generation and processing services to users through the image generation and processing system. Alternatively, the server 110 may also include a server that is deployed with an image generation and processing system and trains or updates the image generation and processing system.

[0081] The terminal 120 may be a terminal device having network connection function and data processing function. For example, the terminal 120 may be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, a smart TV, a laptop computer, a desktop computer, etc.

[0082] The terminal 120 may include a user terminal for receiving the image generation and processing service, or the terminal 120 may also include a development terminal used by developers of the image generation and processing system.

[0083] Optionally, a picture generation and processing system may also be deployed in the terminal 120 .

[0084] Optionally, the above system includes one or more servers 110 and multiple terminals 120. The embodiment of the present application does not limit the number of servers 110 and terminals 120.

[0085] The terminal and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0086] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the technology and / or format of Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent the data exchanged through the network. Conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology. This application is not limited here.

[0087] Figure 2 A flowchart of an exemplary embodiment of the present invention shows a method for image generation and processing, which is executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be Figure 1 The terminal or server shown, such as Figure 2 As shown, the image generation processing method includes the following steps:

[0088] Step 210: Obtain a picture description text, where the picture description text includes the name of the first person.

[0089] In an embodiment of the present application, the above-mentioned picture description text may be text input by a user, containing the name of a specific person (ie, the above-mentioned first person), and the picture description text may be used to describe the specific person and the environment in which the specific person is located.

[0090] For example, the above picture description text may be: xxx (character name) taking a walk by the lake; or, the above picture description text may also be: xxx (character name 1) and yyy (character name 2) drinking tea in a pavilion by the lake.

[0091] Step 220: Encode the picture description text to obtain the picture description text code.

[0092] In an embodiment of the present application, the image description text can be encoded using a pre-trained machine learning model.

[0093] Among them, the above-mentioned pre-trained machine learning model can be an independent text encoding model; or, the above-mentioned pre-trained machine learning model can also be a text encoding branch integrated in the image generation model.

[0094] Step 230: Encode the image description text and the first image feature of the noise image into a bypass branch in the image generation model to obtain a second image feature output by the bypass branch; the bypass branch is obtained by training the first image-text pair; the first image sample in the first image-text pair includes the first person, and the first text sample in the first image-text pair includes the name of the first person.

[0095] The first picture-text pair may be a picture-text pair consisting of a first picture sample and a first text sample, wherein the first picture sample is a picture containing a first person, and the first text sample may be a text describing the first picture sample.

[0096] For example, please refer to Figure 3 , which shows a schematic diagram of a picture-text pair involved in an embodiment of the present application. Figure 3 As shown, the first text sample 31 is a description text of the first sample picture 32 .

[0097] That is to say, in addition to the first person, the first picture sample also includes information such as the environment and behavior of the first person; correspondingly, in addition to the name of the first person, the first text sample also includes text describing the environment and behavior of the first person.

[0098] The noise picture may be a random noise picture (ie, the picture itself has no meaning and is just a picture full of snowflakes); or, the noise picture may be a picture obtained by superimposing noise on any picture.

[0099] In an embodiment of the present application, a bypass branch can be separately set in the image generation model, and the features of the specific person (i.e., the first person mentioned above) in the first text sample can be learned through the bypass branch, so that in the subsequent application process, the features of the specific person in the input image description text can be accurately extracted.

[0100] Step 240: Input the picture description text encoding, the second picture feature, and the first picture feature into a denoising branch in the picture generation model to obtain a third picture feature output by the denoising branch.

[0101] In an embodiment of the present application, after learning the second image feature related to the first person through the bypass branch, the second image feature is input into the denoising branch, and the denoising branch combines the image description text encoding and the first image feature to perform denoising processing to obtain the third image feature output by the denoising branch.

[0102] Step 250: Perform decoding processing based on the third picture feature to obtain the picture corresponding to the picture description text.

[0103] In an embodiment of the present application, after obtaining the third picture feature, the computer device may decode the third picture feature to reconstruct a picture, that is, a picture corresponding to the picture description text.

[0104] For example, the computer device may decode the third image feature through a decoding branch in the image generation model to obtain an image corresponding to the image description text.

[0105] Among them, the above-mentioned image generation model can be a machine learning model built and trained based on the diffusion model.

[0106] For example, please refer to Figure 4 , which shows an architecture diagram of a diffusion model for generating images from texts involved in an embodiment of the present application. Figure 4 As shown, the principle of generating pictures by the diffusion model is as follows:

[0107] The image is encoded by an encoder 41 (such as a variational auto encoder (VAE)) after adding noise to the image to obtain an image embedding. For example, VAE encoding can be used to map to a latent feature space), and then a diffusion model 42 is used to obtain a latent space representation at time T through a diffusion process. The denoising process is performed by a U-net (corresponding to the above denoising branch) that denoises T times. The features of the image (i.e., the features of the original image without adding noise) are restored through the above operations. The restored coded features are decoded by the encoder 41 (such as VAE) to obtain the image. For text, the text embedding is obtained through a text encoding branch 43, such as a contrastive language-image pre-training (CLIP) network, and the text embedding is input into the U-Net to be processed as the input features of the U-Net that need to be denoised. The diffusion branch is used to map the features of the VAE encoding of the noisy image to the latent space representation at time T. In the subsequent denoising branch learning of the image, the noise representation can be fitted to subtract the noise representation from the input image to obtain the actual image representation, and finally the actual image is obtained through the decoder D.

[0108] Above Figure 4 The architecture shown in FIG. 1 can be trained with a large number of samples of person picture-text pairs (e.g., hundreds of millions of samples) to obtain a model that can generate corresponding person pictures from the person description text; on this basis, if it is necessary to realize the generation of pictures of a specific person, it is necessary to collect the picture-text pairs corresponding to the specific person as samples. In one implementation method, the above Figure 4 Retrain the model shown, for example, Figure 4 The U-Net in the model is retrained to adjust the network parameters of the U-Net so that when the subsequent model receives a text containing the name of the specific person, it can generate a picture containing the specific person.

[0109] However, the above Figure 4 The model retraining scheme shown in the figure changes the parameters of the original U-Net. The method of fine-tuning the U-Net with a limited number of image-text sample pairs of specific characters (compared with the original model trained with hundreds of millions of image-text pairs, the number of samples of a single specific character is usually no more than tens of thousands, which is limited). This can easily make the model forget the originally trained parameters and directly remember the features in the limited training set, resulting in model overfitting - for example, the model can only generate images similar to the training set and cannot maintain the model generalization required for secondary creation.

[0110] In this regard, the solution shown in the embodiment of the present application, based on the original model, adds an additional bypass branch, through which the features in the training set of a specific person are learned, and then the features learned by the bypass branch are integrated into the original model. While maintaining the generalization ability of the original model, it is possible to additionally learn the features related to the specific person in the text and accurately generate pictures containing the specific person.

[0111] For example, please refer to Figure 5 , which shows a framework diagram of a picture generation model including a bypass branch involved in an embodiment of the present application. Figure 5 As shown, in Figure 4 Based on the model shown in FIG. 1 , an additional bypass branch 44 is added, and the bypass branch 44 is connected to the denoising branch (i.e. Figure 4 The U-Net part in the model is connected; during the model training process, in the above Figure 4 After the model training shown is completed, the bypass branch 44 is trained separately through the picture-text pair of the specific person (i.e., the first picture-text pair mentioned above), so that the bypass branch 44 has the ability to extract the features of the specific person from the input text; in the subsequent application process of picture generation, the bypass branch 44 processes the encoding of the input text, can extract the features of the specific person, and input the mentioned features into the denoising branch, so that the denoised picture features generated by the denoising branch can include the features of the specific person, so that after decoding, a picture containing the specific person is obtained.

[0112] To summarize, the scheme shown in the embodiment of the present application, after obtaining the picture description text containing the name of the first person, encodes the picture description text to obtain the picture description text code, then processes the picture description text code and the first picture feature of the noise picture through a bypass branch to obtain a second picture feature, and then inputs the second picture feature into the denoising branch to obtain a third picture feature, and finally decodes to obtain the picture corresponding to the picture description text; wherein, since the bypass branch is trained by the picture containing the first person and the text containing the name of the first person, and is independent of the network branch other than the denoising branch, the model in the above scheme can accurately generate pictures containing specific people while retaining the model generalization ability of the denoising branch, thereby improving the effect of generating pictures of specific people.

[0113] Based on the above Figure 2 Corresponding embodiments, in some embodiments, the bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2;

[0114] For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit, and the picture description text encoding;

[0115] For the i-th first processing unit among the n first processing units, the input of the i-th first processing unit is the output result of the i-1-th first processing unit and the picture description text encoding; 2≤i≤n, and i is an integer;

[0116] For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit and the picture description text encoding;

[0117] For the i-th second processing unit among the n second processing units, the input of the i-th second processing unit is the output result of the i-1-th second processing unit, the picture description text encoding, and the output result of the i-1-th first processing unit.

[0118] In the above scheme, the output result of each first processing unit in the bypass branch will be input into the second processing unit in the denoising branch, so as to learn and provide the features related to the specific person to the denoising branch through the bypass branch, avoiding the problem of updating the parameters of the denoising branch and causing the denoising branch to forget the original trained parameters and directly remember the existing limited training set, thereby maintaining the generalization of the model required for creation.

[0119] Among them, the image generation model is composed of several network layers, including an input layer, a convolution layer, a pooling layer, a normalization layer, an output layer, etc., and each network layer can perform a calculation function on the input features; the previous network layer of the above-mentioned first first processing unit can refer to the network layer in the image generation model that is located before the first first processing unit and is directly connected to the first first processing unit, except for the network layer that outputs the image description text encoding. For example, in the above-mentioned bypass branch, when there are no other network layers before the first first processing unit, the previous network layer of the above-mentioned first first processing unit can be the network layer that outputs the first image feature of the noise image.

[0120] Similarly, the previous network layer of the above-mentioned first second processing unit may refer to other network layers in the image generation model, which are located before the first second processing unit and directly connected to the first second processing unit, except for the network layer that outputs the image description text encoding.

[0121] In some embodiments, the first processing unit in the above-mentioned bypass branch is a feature processing network composed of one or more network layers. When n is greater than or equal to 3, the n-1 first processing units in the above-mentioned bypass branch are connected in sequence; the i-th first processing unit among the above-mentioned n-1 first processing units refers to the i-th first processing unit in the order of connection between the first processing units.

[0122] Similarly, the second processing unit in the above-mentioned denoising branch is a feature processing network composed of one or more network layers, and the n second processing units in the above-mentioned denoising branch are connected in sequence; the i-th second processing unit among the above-mentioned n second processing units refers to the i-th second processing unit in the order of connection between the second processing units.

[0123] In other embodiments, the first processing units in the above-mentioned bypass branches may also be connected through other structures. For example, each first processing unit in the above-mentioned bypass branch may be divided into multiple groups, each group containing two or more first processing units, and the first processing units in the same group are connected in sequence; in this case, the i-th first processing unit among the above-mentioned n-1 first processing units refers to the i-th first processing unit when the n-1 first processing units are arranged in a specified order.

[0124] Similarly, each second processing unit in the above-mentioned denoising branch can also be divided into multiple groups, each group contains two or more second processing units, and the second processing units in the same group are connected in sequence; at this time, the i-th second processing unit among the above-mentioned n second processing units refers to the i-th second processing unit under the n first processing units arranged in a specified order.

[0125] The first processing unit and the second processing unit may be processing units with the same structure; or the first processing unit and the second processing unit may be processing units with different structures.

[0126] For example, taking the denoising branch as U-Net as an example, the above-mentioned U-Net is a U-shaped network composed of a number of QKV network units. Accordingly, the above-mentioned first processing unit and the second processing unit can be QKV network units with the same structure.

[0127] Optionally, when the first processing unit and the second processing unit are processing units with the same structure, the initial parameters of the first processing unit can be set as the parameters of the second processing unit. For example, the initial parameters of the i-th first processing unit are the parameters of the i-th second processing unit.

[0128] For example, please refer to Figure 6 , which shows a schematic diagram of the structure of the QKV network unit involved in the embodiment of the present application. Figure 6As shown, the structure of a QKV network unit can include multiple stacked resBlock (residual blocks) and SpatialTransformer (spatial transformer) structures, where the structural diagram of the residual block is as follows Figure 7 As shown, in Figure 7 In the figure, the input of the residual block is processed by the padded 2D convolution layer, added to the result of the time step embedding after the hidden layer, and then added to the original input of the residual block, and finally processed by a padded 2D convolution layer before output; the schematic diagram of the spatial transformer is shown in Figure 8 As shown, in Figure 8 In the figure, the output of the previous layer is taken as Q, which is processed by a padded two-dimensional convolution layer and a density projection to obtain the Q matrix. The context embedding is taken as KV, which is processed by a density projection to obtain the K matrix and the V matrix respectively. The Q matrix and the K matrix are processed by matrix multiplication, normalization (SCALE) and normalized exponential function to obtain the attention weight matrix, which is then matrix multiplied with the V matrix and passed through a padded two-dimensional convolution layer to obtain the output result of height*width*number of channels.

[0129] exist Figure 6 In the figure, the QKV structure in U-Net contains two Spatial Transformers, each Spatial Transformer is a QKV process. In the first QKV process, KV is the same as the input Q (Q is the output of the previous network layer / network structure). In the second QKV process, KV is the text embedding constraint used to control the generation of the image, that is, the image description text encoding.

[0130] In a possible implementation, the denoising branch further includes an intermediate processing unit; the bypass branch further includes a bypass output unit; optionally, the bypass output unit may be connected after the nth first processing unit;

[0131] The input of the bypass output unit includes the output result of the nth first processing unit;

[0132] The input of the intermediate processing unit includes the output result of the nth first processing unit, the output result of the nth second processing unit, and the output result of the bypass output unit.

[0133] Optionally, the denoising branch further includes n third processing units, and the n third processing units are sequentially connected after the intermediate processing unit; optionally, the n second processing units, the intermediate processing unit and the n third processing units can constitute a U-Net.

[0134] In the above scheme, the final output result of the entire bypass branch can be input into the intermediate processing unit in the denoising branch through the bypass output unit in the bypass branch, so as to learn the information in the sample of a specific person through a bypass branch in a latent space, thereby avoiding the overfitting problem.

[0135] In the embodiment of the present application, in order to prevent overfitting, a bypass branch of a latent space can be used to learn the information in the sample of a specific person. Figure 5 Taking the model structure shown as an example, the module structure of the bypass branch is the same as the right half structure of denoisingU-Net, and the pre-trained model parameters (that is, initial parameters) of the bypass branch can be the right half structure parameters of U-Net.

[0136] Please refer to Fig. 9 , which shows a schematic diagram of the structure of the bypass branch and the denoising branch involved in the embodiment of the present application; Fig. 9 In the forward calculation, the input of the bypass branch 91 is the same as the U-Net input, which is the latent space representation at a certain moment. The structures of QKV7, 8, and 9 are exactly the same as those of QKV1, 2, and 3 in the denoising branch 92, and the pre-training parameters are also the same. The difference is that in this solution, the parameters of the bypass branch will be updated during retraining, while the parameters of U-Net may not be updated. The structure of the bypass branch can be fused with the structure of U-Net in the following way:

[0137] The output of a QKV network unit of the bypass branch (dashed arrows 93, 94, and 95 in the figure) is weighted and added to the output of the QKV network unit corresponding to U-Net. Fig. 9 In the example, the output of QKV7 is not only input into QKV8 for normal network forward calculation, but also generates the input of QKV2 according to the formula: out_QKV1+a*out_QKV7=input_QKV2; where out_QKV1 is the output of QKV1, out_QKV7 is the output of QKV7, a is the weight of out_QKV7, and input_QKV2 is the input of QKV2;

[0138] like Fig. 9 As shown in the figure, the output of the last Spatial transformer structure of the bypass branch is also weighted with the output of the U-Net intermediate layer as the input of the subsequent res Block. In the overall network, the parameters in the bypass branch are the parameters that need to be learned, and others such as VAE (the left side and D constitute the encoding and decoding structure of the visual auto encoder), U-Net structure, and the text encoding (text_encoder, such as the CLIP text representation branch) on the right can be left unchanged.

[0139] Based on the above Figure 2 Corresponding embodiments, in some embodiments, the bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2;

[0140] For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit, and the picture description text encoding;

[0141] For the jth first processing unit among the n first processing units, the input of the jth first processing unit is the output result of the j-1th first processing unit and the picture description text encoding; 2≤j≤n, and j is an integer;

[0142] For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit, the picture description text encoding, and the output result of the first first processing unit;

[0143] For the jth second processing unit among the n second processing units, the input of the jth second processing unit is the output result of the j-1th second processing unit, the picture description text encoding, and the output result of the jth first processing unit. In the above scheme, the output result of each first processing unit in the bypass branch will be input into the second processing unit in the denoising branch, so as to learn and provide the denoising branch with the features related to the specific person through the bypass branch, avoiding the problem of updating the parameters of the denoising branch so that the denoising branch forgets the original trained parameters and directly remembers the existing limited training set, and can maintain the generalization of the model required for creation.

[0144] Please refer to Fig.10 , which shows a schematic diagram of the structure of the bypass branch and the denoising branch involved in the embodiment of the present application; Fig.10 As shown, the structure of the bypass branch can be integrated with the structure of U-Net in the following way:

[0145] The output of a QKV network unit of the bypass branch (dashed arrows 1001, 1002, and 1003 in the figure) is weighted and added to the output of the QKV network unit corresponding to U-Net. Fig.10 In the example, the output of QKV8 is not only input into QKV9 for normal network forward calculation, but also generates the input of QKV2 according to the formula: out_QKV1+a*out_QKV8=input_QKV2; where out_QKV1 is the output of QKV1, out_QKV8 is the output of QKV8, and a is the weight of out_QKV8.

[0146] Based on the solution shown in any of the above embodiments of this application, please refer to Fig.11 , which shows a flowchart of a method for image generation processing shown in an exemplary embodiment of the present application, the method is executed by a computer device, the computer device can be implemented as a terminal or a server, the terminal or the server can be Figure 1 The terminal or server shown, such as Fig.11 As shown, the image generation processing method further includes steps 202a to 202f before step 210:

[0147] Step 202a: Encode the first text sample to obtain a first text encoding sample.

[0148] In the embodiment of the present application, during the training of the bypass branch, the computer device also performs encoding processing on the first text sample.

[0149] Step 202b: input the first text encoding sample and the first picture feature sample of the first noise picture sample into the bypass branch in the picture generation model to obtain the second picture feature sample output by the bypass branch.

[0150] In an embodiment of the present application, during the training of the bypass branch, the computer device inputs the first text encoding sample and the first image feature sample of the first noise image sample into the bypass branch in the image generation model.

[0151] When the bypass branch includes n first processing units, the second image feature sample includes the outputs of the n first processing units respectively.

[0152] For example, taking the bypass branch containing n QKV network units as an example, the computer device uses the first image feature sample as Q and the first KV, and the first image feature sample as the second KV, and inputs them into the first QKV network unit in the bypass branch; for other QKV network units in the bypass branch, the output of the previous QKV network unit is used as Q and the first KV, and the first image feature sample is used as the second KV, and inputs them into the current QKV network unit; wherein the outputs of the n QKV network units in the bypass branch constitute the above-mentioned second image feature sample.

[0153] Step 202c: input the first text encoding sample, the second image feature sample, and the first image feature sample into a denoising branch in the image generation model to obtain a third image feature sample output by the denoising branch.

[0154] In an embodiment of the present application, during the training of the bypass branch, the computer device inputs the first text encoding sample, the second image feature sample, and the first image feature sample into the denoising branch in the image generation model.

[0155] In which, when the above-mentioned denoising branch includes n second processing units, and the above-mentioned second image feature samples include the outputs of n first processing units, the computer device can input the outputs of the n first processing units into different second processing units in the denoising unit, for example, in a weighted manner, the outputs of the n first processing units can be integrated into different second processing units or intermediate processing units in the denoising unit.

[0156] The output of each of the n first processing units is used as the second image feature sample to be fused into the denoising branch. For details, see Fig. 9 or Fig.10 , I will not go into details here.

[0157] Step 202d: Perform decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample.

[0158] In an embodiment of the present application, the computer device may decode the third image feature sample to obtain a first predicted image obtained by predicting the first image sample during the training of the bypass branch.

[0159] Step 202e: Obtain a first loss function value based on the difference between the first predicted image and the first image sample.

[0160] In the embodiment of the present application, the computer device may use MSE loss to calculate the first loss function value. The MSE loss is calculated by outputting the predicted image and the mean square error of the image of the image-text pair, and the formula is as follows:

[0161]

[0162] Among them, y in the above formula i is the i-th pixel value in the first image sample, is the i-th pixel value in the first predicted picture.

[0163] Step 202f: Based on the first loss function value, update the parameters of the bypass branch.

[0164] Among them, the image generation model after the bypass branch is updated is used to generate an image corresponding to the image description text based on the input image description text; the image description text contains the name of the first person.

[0165] In an embodiment of the present application, after the computer device obtains the first loss function value, it can update the parameters of the bypass branch through algorithms such as gradient descent; during this process, the computer device can keep the parameters of other parts of the image generation model unchanged.

[0166] In the above scheme, by training the bypass branch with the first picture-text pair, the bypass branch can enhance the extraction of features of a specific person, thereby accurately generating a picture containing a specific person, thereby improving the effect of generating a picture of a specific person.

[0167] In another possible implementation, the computer device may also perform training through a second picture-text pair according to steps 202a to 202f above to update the parameters of the bypass branch; wherein the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person.

[0168] For example, please refer to Fig.12 , which shows a schematic diagram of a picture-text pair involved in an embodiment of the present application. Fig.12 As shown, text sample 1201 is a description text of sample image 1202, and text sample 1203 is a description text of sample image 1204. That is, the second image sample may include the first person (such as the face of the first person); correspondingly, the second text sample may include the name of the first person.

[0169] Based on the solution shown in any of the above embodiments of this application, please refer to Fig.13 , which shows a flowchart of a method for image generation processing shown in an exemplary embodiment of the present application, the method is executed by a computer device, the computer device can be implemented as a terminal or a server, the terminal or the server can be Figure 1 The terminal or server shown, such as Fig.13 As shown, the above step 220 can be implemented as step 220a and step 220b:

[0170] Step 220a: Encode the picture description text through the text encoding branch in the picture generation model to obtain an initial text encoding; the initial text encoding includes a first name representation corresponding to the name of the first person.

[0171] In an embodiment of the present application, a computer device may encode the picture description text through a text encoding branch to obtain an initial text encoding, wherein the first name representation in the initial text encoding is a representation obtained by encoding through the text encoding branch.

[0172] Step 220b: In response to the existence of a name representation corresponding to the name of the first person in the character representation library, the first name representation in the initial text code is replaced with the name representation corresponding to the name of the first person in the character representation library to obtain the picture description text code.

[0173] Among them, the text encoding branch is obtained through training of the second picture-text pair; the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person; the name representation corresponding to the name of the first person in the character representation library is the name representation obtained by encoding the name of the first person during the training process based on the text encoding branch, and is stored or updated in the character representation library.

[0174] In order to ensure that the same representation can be obtained for the name of the same person in subsequent applications, in an embodiment of the present application, the text encoding branch can be trained through the second image sample so that the text encoding branch can better learn the representation of the name of the specific person. At the same time, the representation of the name of the specific person during the training process is stored or updated in the character representation library. When the picture description text containing the name of the specific person is subsequently encoded, the representation of the name of the specific person in the character representation library can be used to replace the representation corresponding to the name of the specific person in the initial text encoding, so that the representation of the name of the specific person during the training process and the application process is consistent, thereby ensuring that when the picture is generated subsequently, the picture containing the specific person can be generated more accurately.

[0175] In a possible implementation, encoding the first text sample to obtain the first text encoding sample includes:

[0176] The first text sample is encoded by a text encoding branch in the image generation model to obtain an initial text encoding sample; the initial text encoding sample includes a second name representation sample corresponding to the name of the first person;

[0177] In response to the presence of a name representation corresponding to the name of the first person in the character representation library, the second name representation sample in the initial text encoding sample is replaced with the third name representation sample to obtain a first text encoding sample.

[0178] In an embodiment of the present application, during the process of training the bypass branch, when encoding the first text sample to obtain the first text encoding sample, the computer device also replaces the representation of the specific person in the initial text encoding sample with the representation of the name of the specific person in the character representation library, thereby ensuring the accuracy of the training of the bypass branch.

[0179] Based on the solution shown in any of the above embodiments of this application, please refer to Fig.14 , which shows a flowchart of a method for image generation processing shown in an exemplary embodiment of the present application, the method is executed by a computer device, the computer device can be implemented as a terminal or a server, the terminal or the server can be Figure 1 The terminal or server shown, such as Fig.14 As shown, the above method further includes steps 204a to 204g:

[0180] Step 204a: Encode the second text sample through the text encoding branch to obtain a first name representation sample.

[0181] In the embodiment of the present application, during the process of training the text encoding branch, the computer device also performs encoding processing on the second text sample.

[0182] In a possible implementation, the computer device encodes the second text sample through the text encoding branch. In the process of obtaining the first name representation sample, the representation output by the text encoding branch is used as the first name representation sample, without the need to query the character representation library.

[0183] Step 204b: In response to the existence of a name representation corresponding to the second text sample in the character representation library, based on the first name representation sample, the name representation corresponding to the second text sample in the character representation library is updated; in response to the absence of a name representation corresponding to the second text sample in the character representation library, the first name representation sample is stored in the character representation library as the name representation corresponding to the second text sample.

[0184] In the embodiment of the present application, in the process of training the text encoding branch, unlike the process of training the bypass branch, after obtaining the first name representation sample, the computer device updates or stores the first name representation sample to the character representation library.

[0185] Among them, the computer device can train the text encoding branch in batches, and each batch may encode the second text sample multiple times. For example, in one batch, the training sample contains multiple picture-text pairs corresponding to the second text sample. At this time, the text encoding branch may output multiple different first name representation samples obtained by encoding the second text sample; in this regard, the computer device can take the average of the multiple different first name representation samples and update or store them in the character representation library; or, the computer device can also update or store multiple different first name representation samples in the character representation library. In this case, in the subsequent training of the bypass branch, or in the model application process, when it is necessary to query the representation of a specific person from the character representation library, a representation can be randomly selected from the multiple representations of the specific person queried for replacement, thereby achieving the effect of diversification of the generated pictures.

[0186] Step 204c: input the first name representation sample and the fourth image feature sample of the second noise image sample into the bypass branch to obtain the fifth image feature sample output by the bypass branch.

[0187] Step 204d: input the first name representation sample, the fifth picture feature sample, and the fourth picture feature sample into the denoising branch to obtain a sixth picture feature sample output by the denoising branch.

[0188] Step 204e: Perform decoding processing based on the sixth picture feature sample to obtain a second predicted picture corresponding to the second text sample.

[0189] Step 204f: Obtain a second loss function value based on the difference between the second predicted picture and the second picture sample.

[0190] The execution process of the above steps 204c to 204f can refer to the execution process of steps 202b to 202e, which will not be repeated here.

[0191] Step 204g: Based on the second loss function value, update the parameters of the text encoding branch.

[0192] Different from step 202f, the above step 204g keeps other parts of the model (including the bypass branch) unchanged and updates the parameters of the text encoding branch.

[0193] In one possible implementation, the text encoding branch and the bypass branch are trained alternately.

[0194] Alternatively, the text encoding branch and the bypass branch can also be trained in sequence, for example, the text encoding branch is trained first, and then the bypass branch is trained. The above scheme designs an alternating training method, which avoids the problem that the slow-converging branch cannot learn good enough information and further slows down the convergence speed due to the different convergence speeds of the two branches by switching training between the two branches under random samples. At the same time, alternating training also avoids the problem that continuous training of a single branch for a single learning task easily causes the overall model to overfit the task.

[0195] The solution shown in the embodiment of this application can be used for secondary creation of film and television works. For example, after a TV series is played, users can perform secondary creation based on the characters and scenes of the original series to generate creative images. For example, please refer to Fig.15 , which shows a schematic diagram of the interface of model training and application involved in the embodiment of the present application. Fig.15 As shown:

[0196] 1) In the training part: users can train new TV series and film generation models; for example, users / developers input stills / screenshots of TV series and stills / screenshot description text 1501 as training samples, and the computer device trains and returns the trained model 1502.

[0197] 2) In the creation part: the user can perform secondary creation for all trained TV dramas. For example, the user enters the text description of the painting to be created 1503, selects the basic drama title 1504, clicks OK to create, and the model returns the creation result 1505. For example, for a TV drama, after the model is trained according to the scheme shown in the above embodiment of this application, the user can enter "Zhang XX and Li XX are looking at the willow trees on the opposite bank on the shore", and the model creates an image based on the text.

[0198] Taking the film and television drama scene application as an example, please refer to Fig.16 , which shows a framework flow chart of a method for generating and processing a scene picture of a film and television drama involved in an embodiment of the present application;

[0199] like Fig.16 As shown in the figure, the scheme includes three stages: sample collection, model training and model application.

[0200] S21, sample collection.

[0201] This scheme creates images based on text, and needs to collect the target film and television drama images and text pairs composed of text as supervision information (mainly used for training bypass branches), such as Figure 3 For the target TV series, it is necessary to collect the main scenes of the series and annotate the scenes with text descriptions, in which the texts need to include the names of the characters.

[0202] Since the solution requires specific training for characters to ensure that the corresponding face can be accurately generated when a character's name is input, a character name-face image-text pair is needed as supervision information for character representation. It is necessary to collect image-text pairs consisting of characters and texts of the target film and television dramas, such as Fig.12 For the target TV series, collect the faces and names of the characters.

[0203] In the alternating training, the bypass branch and the text encoding branch are trained separately. When training the bypass branch, film and television drama image-text pairs are required, and when training the text encoding branch, character image-text pairs are required. When training the text encoding branch, bs samples are randomly selected from the character image-text pairs for network learning; when training the bypass branch, bs samples are randomly selected from the film and television drama image-text pairs for learning.

[0204] For the names of the characters in the input text, this solution generates a new embedding representation for the characters (compared to the original text representation generated based on CLIP): Fig.17The framework diagram of the model training process involved in this application is shown as follows: Fig.17 As shown in the figure, the character representation is input into the denoising U-Net as a constraint, and the new character representation is input into the cross-attention QKV unit of the image generation model (i.e., the denoising branch and bypass branch mentioned above). During training, except for the text encoding branch and the character's name representation, other parameters are fixed and do not participate in the training. The forward calculation process is: the model projects the noise image x (the noise image is an image without any information, such as a snowflake image) generated by the random seed xi into the latent space through the diffusion branch to obtain the latent space vector (Z T ), the latent space vector is denoised and predicted T times by U-Net with QKV unit with new character representation constraints, and finally generates latent space prediction vector Z', which is restored to the prediction result of image i by decoder D. At this time, the text encoding branch is a structure that needs parameter fine-tuning. Other parts such as VAE (the left side and D constitute the encoding and decoding structure of visual autoencoder), U-Net, bypass branch, etc. do not need to be updated.

[0205] S22, model training.

[0206] This scheme adopts an alternating training method during training. First, the text encoding branch 231 is trained (that is, the character representation training is performed), and then the bypass branch 232 of the film and television drama scene is trained. In addition to the scene images, the training set of the bypass branch also includes character images. Therefore, the training process of the bypass branch also has the function of supervising the character generation, which can further enhance the consistency effect of the same character in different generated images.

[0207] The overall training process is: first, a complete character representation training is performed (i.e., the training of the text encoding branch 231), and then a complete film and television drama scene training is performed (i.e., the training of the bypass branch 232). Then, alternating training is performed - first training a batch of film and television drama scenes, and then training a batch of character representations.

[0208] In the embodiment of the present application, the loss 233 may adopt the MSE loss, which may be obtained by calculating the mean square error between the output image i and the image of the image-text pair.

[0209] Optionally, after the first batch is trained, an updated character / role representation can be obtained (i.e. Fig.16 Character 1 representation, character 2 representation, and so on), train the second batch on this person / role representation, update the representation after training, and continue training.

[0210] like Fig.17 As shown, the model structure involved in this scheme is in the stable diffusion model (such as Figure 4 On the basis of the QKV network (as shown in the figure), a bypass branch is added. The network structure of the bypass branch can be exactly the same as the QKV structure in the right half of U-Net. The model training uses film and television drama data to train the newly added bypass branch. At the same time, when training characters, the scheme trains the representation of specific characters through the text encoding branch (text-embedding), so that the character representation has a stable generation result in the image generation model.

[0211] in, Fig.17 The training process shown can set up two sets of data, one set is the picture-text pairs describing film and television dramas (corresponding to the first picture-text pairs mentioned above), which are used to train the bypass branch, and the other set is the picture-text pairs of film and television drama characters (corresponding to the second picture-text pairs mentioned above), which are used to train the representation of the names of specific characters.

[0212] Fig.17 The model structure and training method of this scheme are shown, which includes a bypass branch for learning film and television drama scenes and a text encoding branch for training film and television drama characters. During training, the text encoding branch and the bypass branch are alternately trained using different training image-text pairs, and when training a certain branch, only the parameters of the branch are learned, and the parameters of other parts / branches in the model are not learned.

[0213] The conventional deep learning training process is: a total of N rounds (such as 100) of iterations are performed on the entire data. In each round of iteration, the entire image data is trained once in the model, which is called an iteration. In each round of iteration, due to the limited video memory resources of the training machine, all samples cannot be input into the model for training at one time. Therefore, all samples need to be trained in batches. Each batch of samples is input into the model for forward calculation, backward calculation, and model parameter update.

[0214] This program trains Fig.17 As shown in the figure, the data and training model parameters used in the two training tasks are different. In order to ensure the stability of character embedding and film and television drama scene embedding, this scheme adopts an alternating training method during training. First, a text encoding branch training is performed, and then a bypass branch training of the film and television drama scene is performed.

[0215] Specific training related information is described as follows:

[0216] 1) Training 1 - Character Representation

[0217] In this scheme, during training 1 - character representation training, for a certain batch: first randomly extract bs image-text pairs from all character samples to obtain bs samples. During training:

[0218] S1, the first batch of the first round of pre-training parameter initialization: the VAE, text_encoder, and U-Net use the open source trained model parameters, and only the text encoder branch (text encoder) is updated in this training. The initialization can use a learning rate of 0.0004, and the learning rate becomes 0.1 times the original after each round of learning, and a total of 2 rounds of training;

[0219] S2, take a random seed xi to generate a noise image x, and the noise image x is represented by the latent space to generate Z T ;

[0220] S3, the text information is passed through the text encoder to obtain the text representation, which is input into the generation model (the text representation is used as the input of KV), and Z T Perform T denoising U-Net forward calculations under the KV constraint to obtain the Z0 prediction value, and then obtain the predicted image (i.e., output image i) through the decoder D (decoder);

[0221] S4, calculate batch loss: calculate the loss between the predicted image and the supervised image (i.e., the image in the image-text pair), and count the total loss of the batch of samples;

[0222] S5, using the SGD stochastic gradient descent method, backpropagates the loss to the model to obtain the gradient of the text encoding branch and update the parameters;

[0223] S6: Complete all N / bs batches of training and end one iteration.

[0224] 2) Generate a character representation library

[0225] Please refer to Fig.19 , which shows a flow chart for generating a character representation library involved in this application. After the previous training step, each character has a specific representation, and all character representations are saved in the character representation library: the text (name of the character) corresponding to the character sample (multiple images of character role 1, multiple images of character role 2, etc.) is processed through the text encoding branch to generate multiple representations τ of a specific character, and the mean (i.e., average vector) of these representations τ is taken to obtain the final representation (i.e., character 1 representation, character 2 representation, etc.), and each character's name and the final representation are stored in the character representation library.

[0226] 3) Training 2 - Bypass branch training

[0227] like Fig.16As shown in the figure, the bypass branch uses the film and television drama image and text pairs for training, and no parameter update is performed on other parts / branches in the model except the bypass branch. For a batch training, firstly, bs image and text pairs (image, text) are randomly selected from the full amount of image and text samples, where the text is used to generate constraints and the image is used for loss calculation. During training:

[0228] S1, the first batch of the first round of pre-training parameter initialization: the VAE, text_encoder, and U-Net use the open source trained model parameters and do not update them in this training, only training the bypass branch. The initialization can use a learning rate of 0.0004, and the learning rate will be reduced to 0.1 times the original after every 2 rounds of learning, and training will be carried out for 4 rounds;

[0229] S2, take a random seed xi to generate a noise image x, and the noise image x is represented by the latent space to generate Z T ;

[0230] S3, the text information is passed through the text encoding branch to obtain the text representation, the character representation in it is replaced with the corresponding character representation queried from the above character representation library, and input into the denoising branch and bypass branch in the image generation model, where the text representation is used as the input of KV, and Z T Perform T denoising U-Net forward calculations under the KV constraint to obtain the Z0 prediction value, and then obtain the predicted image through the decoder D;

[0231] S4, calculate batch loss: calculate the loss of the predicted image and the supervised image, and count the total loss of the batch of samples;

[0232] S5, using the SGD random gradient descent method, returns the loss to the model to obtain the gradient of the bypass branch parameters and update the parameters.

[0233] S6: Complete all N / bs batches of training and end one iteration.

[0234] 4) Text representation based on text encoding branch

[0235] In the solution shown in the present application, the corresponding character representation in the encoding of the text can be replaced by the representation in the character representation library.

[0236] 5) Character representation replacement

[0237] Please refer to Fig. 20 , which shows a schematic diagram of the character representation replacement process in the text encoding involved in this application. Fig. 20As shown: for a piece of text, the text encoding branch first maps it to the token space, that is, each word finds n corresponding tokens (since the word and token are not in one-to-one correspondence here, so different words may have tokens of unequal length corresponding to them. For example, in Fig. 20 "Zhang XX" may correspond to 5 tokens), and then generates the text encoding from the tokens. At this time, for the representation corresponding to the name of the person, it is replaced with a new representation from the person representation library, and the representations of other texts remain unchanged, so as to obtain a new text representation. At this time, the person in the representation is the representation of the person bound by the training of the specific person-image pair.

[0238] S23, model application.

[0239] As Fig.16 shown, the terminal of the user for secondary creation can upload the subsequent creative text of the movie and TV drama input by the user to the server and specify the number K of generated pictures (for example, the text is: Zhang XX and Li XX look at each other affectionately, and the value of k is 4). The server generates K random seeds and inputs the random seeds and the text into the trained picture generation model. The picture generation model finds the specific person representation according to the characters in the text, and uses the new text representation with the person representation replaced to generate K images in the generation model and returns to display K pictures to the user. As Fig.18 shown, for the user input text, on the one hand, the text feature encoding is obtained through the processing of the text encoding branch. On the other hand, the person is searched in the person representation library to confirm whether there is a matching person. If not, there is no need to replace. If there is, the person feature encoding is obtained from the person representation library and replaces the person feature encoding, that is, the person feature encoding in the output result of the text encoding branch is replaced with the newly obtained person feature encoding to obtain the updated text feature encoding, and then the updated text feature encoding is input back into the picture generation model, and the picture generation model generates an image and returns it.

[0240] The text representation is used as a constraint to drive the generation model to generate.

[0241] Fig.21 shows the flow chart of the model application process involved in this application. In Fig.21In the application process shown, after the model is trained, the character representations (embeddings) of all the trained characters in the play are extracted based on the newly trained text representations, and the character embeddings are saved in the character representation library. When the model is applied, for each text created by the user, the embedding is first extracted through the text representation, and then the character part in the text representation is replaced based on whether there is a character in the inventory in the text, and a new embedding of the text to be created is obtained. The embedding passes through the Denoising U-Net, bypass branch, and decoding part of the image generation model to output the image to obtain the generated result.

[0242] like Fig.21 As shown in the figure, the application process of this solution is shown. First, according to the trained model, a character representation library is built according to the characters in the film and television drama. When generating image inference, the system first converts the text into an embedding for the user input to be created, and at the same time searches for the characters in the text in the character representation library, replaces the character representation in the original text embedding with the found character representation, and then inputs the new text embedding into the model to generate an image and return it.

[0243] Please refer to Fig.21 , which shows a schematic diagram of the service process when applied, such as Fig.21 As shown, the image generation service deployed in the service collects user input, such as "Zhang XX and Li XX look at each other affectionately", and specifies the value of K as 4. The service generates K random seeds (for example, assuming that the user specifies the number of images to be generated as K, the image generation service will generate K random seeds), and inputs the seeds and text into the image generation model. The image generation model finds the character representation of the character according to the character name in the text, replaces the character representation in the original text representation, and uses the new text representation with the character representation replaced as the input of the image generation model, so as to generate K images through the image generation model and return them to the user, so as to display K images to the user.

[0244] The solution involved in this application includes at least the following effects:

[0245] 1) It can provide a model for generating images of film and television dramas, embedding different information of film and television drama scenes and characters into the same network, which can better meet the needs of creating consistent scenes and characters of film and television dramas;

[0246] 2) Providing a joint learning method for different functional network branches, by alternately training the embedding of characters and scenes, to achieve stable output of images jointly created by the two;

[0247] 3) Learn the complex scene information of film and television dramas through bypass branches and add it to U-Net as a generated feature, realizing the embedding of film and television drama information to resist overfitting;

[0248] 4) Independent extraction of character representations can be achieved through the character representation library, which can achieve a stronger consistency in character identity and avoid outputting different face images for the same name.

[0249] In addition, the hyperparameters in the training process in the above embodiments, such as the learning rate, the number of samples, etc., can be adjusted; in addition, in the above embodiments, the alternating training process of the bypass branch and the text encoding branch can also be replaced by a separate training process of the bypass branch and the text encoding branch.

[0250] As mentioned above Fig.16 and Fig.21 As shown in the figure, this scheme trains the film and television drama scenes through bypass branches to avoid the problem of overfitting of the original model caused by the U-Net branch in the direct micro-model, making it difficult to produce diversified generation results; at the same time, compared with the structure of fine-tuning denoising U-Net, the parameters of the bypass branch are reduced by half, which further helps to reduce the potential for overfitting.

[0251] On the other hand, in order to solve the problem of consistency between characters, this solution trains a stable character-text embedding mapping (i.e., the character representation library mentioned above) when embedding character text, so as to achieve the goal of stable character creation. In terms of training methods, in order to solve the problem that the bypass branch and the text encoding branch have different functions, compared with training the two branches at the same time, this solution designs an alternating training method, which can switch between the two branches under random samples to avoid the problem that the slow-converging branch cannot learn enough good information due to the different convergence speeds of the two branches, which further slows down the convergence speed; at the same time, alternating training also avoids the problem that continuous training of a single branch and a single learning task (there are two learning tasks in this solution: film and television drama picture-text pairs, character picture-text pairs) easily causes the overall model to overfit to the task.

[0252] Fig. 22 1 shows a block diagram of a picture generation processing device shown in an exemplary embodiment of the present application, and the device can be used to execute all or part of the steps in the methods shown in the above embodiments; Fig. 22 As shown, the device comprises:

[0253] The text acquisition module 2201 is used to acquire a picture description text, wherein the picture description text includes the name of the first person;

[0254] The encoding module 2202 is used to encode the picture description text to obtain the picture description text code;

[0255] The bypass processing module 2203 is used to input the picture description text encoding and the first picture feature of the noise picture into the bypass branch in the picture generation model to obtain the second picture feature output by the bypass branch; the bypass branch is obtained by training the first picture text pair; the first picture sample in the first picture text pair includes the first person, and the first text sample in the first picture text pair includes the name of the first person;

[0256] A denoising processing module 2204 is used to input the picture description text code, the second picture feature, and the first picture feature into a denoising branch in the picture generation model to obtain a third picture feature output by the denoising branch;

[0257] The decoding module 2205 is used to perform decoding processing based on the third picture feature to obtain the picture corresponding to the picture description text.

[0258] In a possible implementation, the bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2;

[0259] For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit and the picture description text encoding;

[0260] For the i-th first processing unit among the n first processing units, the input of the i-th first processing unit is the output result of the i-1-th first processing unit, and the picture description text encoding; 2≤i≤n, and i is an integer;

[0261] For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit and the picture description text encoding;

[0262] For the i-th second processing unit among the n second processing units, the input of the i-th second processing unit is the output result of the i-1-th second processing unit, the picture description text encoding, and the output result of the i-1-th first processing unit.

[0263] In a possible implementation, the denoising branch further includes an intermediate processing unit; the bypass branch further includes a bypass output unit;

[0264] The input of the bypass output unit includes the output result of the nth first processing unit;

[0265] The input of the intermediate processing unit includes the output result of the nth first processing unit, the output result of the nth second processing unit, and the output result of the bypass output unit.

[0266] In a possible implementation, the bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2;

[0267] For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit and the picture description text encoding;

[0268] For the jth first processing unit among the n first processing units, the input of the jth first processing unit is the output result of the j-1th first processing unit, and the picture description text encoding; 2≤j≤n, and j is an integer;

[0269] For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit, the picture description text encoding, and the output result of the first first processing unit;

[0270] For the jth second processing unit among the n second processing units, the input of the jth second processing unit is the output result of the j-1th second processing unit, the picture description text encoding, and the output result of the jth first processing unit.

[0271] In a possible implementation, the encoding module 2202 is further configured to perform encoding processing on the first text sample to obtain a first text encoding sample;

[0272] The bypass processing module 2203 is further used to input the first text encoding sample and the first picture feature sample of the first noise picture sample into the bypass branch in the picture generation model to obtain the second picture feature sample output by the bypass branch;

[0273] The denoising processing module 2204 is further configured to input the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model to obtain a third picture feature sample output by the denoising branch;

[0274] The decoding module 2205 is further configured to perform decoding processing based on the third image feature sample to obtain a first predicted image corresponding to the first text sample;

[0275] The device also includes:

[0276] A first loss calculation module, configured to obtain a first loss function value based on a difference between the first predicted image and the first image sample;

[0277] The first updating module is used to update the parameters of the bypass branch based on the first loss function value.

[0278] In a possible implementation, the encoding module 2202 is used to:

[0279] The picture description text is encoded by a text encoding branch in the picture generation model to obtain an initial text encoding; the initial text encoding includes a first name representation corresponding to the name of the first person;

[0280] In response to the presence of a name representation corresponding to the name of the first person in the character representation library, the first name representation in the initial text code is replaced with the name representation corresponding to the name of the first person in the character representation library to obtain the picture description text code;

[0281] Among them, the text encoding branch is obtained through training of a second picture-text pair; the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person; the name representation corresponding to the name of the first person in the character representation library is based on the name representation obtained by encoding the name of the first person during the training process by the text encoding branch, and is stored or updated in the character representation library.

[0282] In a possible implementation, the encoding module 2202 is further configured to encode the second text sample through the text encoding branch to obtain a first name representation sample;

[0283] The device also includes:

[0284] a representation library establishment module, configured to update the name representation corresponding to the second text sample in the character representation library based on the first name representation sample in response to the presence of the name representation corresponding to the second text sample in the character representation library; and to store the first name representation sample as the name representation corresponding to the second text sample in the character representation library in response to the absence of the name representation corresponding to the second text sample in the character representation library;

[0285] The bypass processing module 2203 is further configured to input the first name representation sample and the fourth image feature sample of the second noise image sample into the bypass branch to obtain a fifth image feature sample output by the bypass branch;

[0286] The denoising processing module 2204 is further configured to input the first name representation sample, the fifth image feature sample, and the fourth image feature sample into the denoising branch to obtain a sixth image feature sample output by the denoising branch;

[0287] The decoding module 2205 is further configured to perform decoding processing based on the sixth picture feature sample to obtain a second predicted picture corresponding to the second text sample;

[0288] The device also includes:

[0289] A second loss calculation module, configured to obtain a second loss function value based on a difference between the second predicted image and the second image sample;

[0290] The second updating module is used to update the parameters of the text encoding branch based on the second loss function value.

[0291] In a possible implementation, the text encoding branch and the bypass branch are trained alternately.

[0292] Fig.23 1 shows a block diagram of a picture generation processing device shown in an exemplary embodiment of the present application, and the device can be used to execute all or part of the steps in the methods shown in the above embodiments; Fig.23 As shown, the device comprises:

[0293] The encoding module 2301 is used to encode the first text sample to obtain a first text encoding sample; the first text sample includes the name of the first person;

[0294] The bypass processing module 2302 is used to input the first text encoding sample and the first picture feature sample of the first noise picture sample into the bypass branch in the picture generation model to obtain the second picture feature sample output by the bypass branch;

[0295] A denoising processing module 2303 is used to input the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model to obtain a third picture feature sample output by the denoising branch;

[0296] A decoding module 2304 is used to perform decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample;

[0297] A loss calculation module 2305, configured to obtain a first loss function value based on a difference between the first predicted image and the first image sample;

[0298] A parameter updating module 2306, configured to update the parameters of the bypass branch based on the first loss function value;

[0299] Among them, the picture generation model after the bypass branch is updated is used to generate a picture corresponding to the picture description text based on the input picture description text; the picture description text contains the name of the first person.

[0300] In a possible implementation, the encoding module 2301 is used to:

[0301] The first text sample is encoded by the text encoding branch in the picture generation model to obtain an initial text encoding sample; the initial text encoding sample includes a second name representation sample corresponding to the name of the first person;

[0302] In response to the presence of a name representation corresponding to the name of the first person in the character representation library, replacing the second name representation sample in the initial text encoding sample with the third name representation sample to obtain the first text encoding sample;

[0303] Among them, the text encoding branch is obtained through training of a second picture-text pair; the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person; the name representation corresponding to the name of the first person in the character representation library is based on the name representation obtained by encoding the name of the first person during the training process by the text encoding branch, and is stored or updated in the character representation library.

[0304] Before collecting relevant user data and during the process of collecting relevant user data, the present application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that the present application starts to execute the relevant steps of obtaining relevant user data only after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant user data are terminated, that is, the user's relevant data is not obtained.

[0305] In other words, all user data collected by this application is collected with the user's consent and authorization, and the collection, use and processing of relevant user data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0306] The above user-related data includes information (including but not limited to user accounts, user operation records), data (including but not limited to text data input by users, stored text / image data, displayed text / image data, etc.) and signal data. For example, the user data involved in this application is obtained with full authorization.

[0307] Fig.24 The structural block diagram of a computer device 2400 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as a server or terminal in the above-mentioned scheme of the present application. In this embodiment, the computer device is a server as an example for explanation. The computer device 2400 includes a central processing unit (CPU) 2401, a system memory 2404 including a random access memory (RAM) 2402 and a read-only memory (ROM) 2403, and a system bus 2405 connecting the system memory 2404 and the central processing unit 2401. The computer device 2400 also includes a mass storage device 2406 for storing an operating system 2409, an application program 2410 and other program modules 2411.

[0308] The mass storage device 2406 is connected to the central processing unit 2401 via a mass storage controller (not shown) connected to the system bus 2405. The mass storage device 2406 and its associated computer readable medium provide non-volatile storage for the computer device 2400. That is, the mass storage device 2406 may include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0309] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 2404 and mass storage device 2406 can be collectively referred to as memory.

[0310] According to various embodiments of the present disclosure, the computer device 2400 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 2400 can be connected to the network through a network interface unit 2407 connected to the system bus 2405, or the network interface unit 2407 can be used to connect to other types of networks or remote computer systems (not shown).

[0311] The memory also includes at least one computer program, which is stored in the memory. The central processor 2401 implements all or part of the steps in the methods shown in the above embodiments by executing the at least one computer program.

[0312] In an exemplary embodiment, a computer-readable storage medium is also provided, which is used to store at least one computer program, and the at least one computer program is loaded and executed by a processor to implement all or part of the steps in the methods shown in the above embodiments. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0313] In an exemplary embodiment, a computer program product is also provided, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes all or part of the steps in the methods shown in the above embodiments.

[0314] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0315] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for generating and processing an image, characterized in that: The method is performed by a computer device, and the method comprises: Obtaining a picture description text, wherein the picture description text includes the name of the first person; Encoding the picture description text to obtain the picture description text code; The picture description text encoding and the first picture feature of the noise picture are input into a bypass branch in the picture generation model to obtain a second picture feature output by the bypass branch; the bypass branch is obtained by training a first picture-text pair; the first picture sample in the first picture-text pair includes the first person, and the first text sample in the first picture-text pair includes the name of the first person; Input the picture description text encoding, the second picture feature, and the first picture feature into a denoising branch in the picture generation model to obtain a third picture feature output by the denoising branch; Decoding is performed based on the third picture feature to obtain a picture corresponding to the picture description text.

2. The method according to claim 1, characterized in that The bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2; For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit and the picture description text encoding; For the i-th first processing unit among the n first processing units, the input of the i-th first processing unit is the output result of the i-1-th first processing unit, and the picture description text encoding; 2≤i≤n, and i is an integer; For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit and the picture description text encoding; For the i-th second processing unit among the n second processing units, the input of the i-th second processing unit is the output result of the i-1-th second processing unit, the picture description text encoding, and the output result of the i-1-th first processing unit.

3. The method according to claim 2, characterized in that The denoising branch further comprises an intermediate processing unit; the bypass branch further comprises a bypass output unit; The input of the bypass output unit includes the output result of the nth first processing unit; The input of the intermediate processing unit includes the output result of the nth first processing unit, the output result of the nth second processing unit, and the output result of the bypass output unit.

4. The method according to claim 1, characterized in that: The bypass branch includes n first processing units; the denoising branch includes n second processing units; n is an integer greater than or equal to 2; For the first first processing unit among the n first processing units, the input of the first first processing unit is the output result of the previous network layer of the first first processing unit and the picture description text encoding; For the jth first processing unit among the n first processing units, the input of the jth first processing unit is the output result of the j-1th first processing unit, and the picture description text encoding; 2≤j≤n, and j is an integer; For the first second processing unit among the n second processing units, the input of the first second processing unit is the output result of the previous network layer of the first second processing unit, the picture description text encoding, and the output result of the first first processing unit; For the jth second processing unit among the n second processing units, the input of the jth second processing unit is the output result of the j-1th second processing unit, the picture description text encoding, and the output result of the jth first processing unit.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Performing encoding processing on the first text sample to obtain a first text encoding sample; Inputting the first text encoding sample and the first picture feature sample of the first noise picture sample into a bypass branch in the picture generation model to obtain a second picture feature sample output by the bypass branch; Inputting the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model to obtain a third picture feature sample output by the denoising branch; Performing decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample; Obtaining a first loss function value based on a difference between the first predicted picture and the first picture sample; Based on the first loss function value, the parameters of the bypass branch are updated.

6. The method according to claim 5, characterized in that The encoding process of the picture description text to obtain the picture description text encoding includes: The picture description text is encoded by a text encoding branch in the picture generation model to obtain an initial text encoding; the initial text encoding includes a first name representation corresponding to the name of the first person; In response to the presence of a name representation corresponding to the name of the first person in the character representation library, the first name representation in the initial text code is replaced with the name representation corresponding to the name of the first person in the character representation library to obtain the picture description text code; Among them, the text encoding branch is obtained through training of a second picture-text pair; the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person; the name representation corresponding to the name of the first person in the character representation library is based on the name representation obtained by encoding the name of the first person during the training process by the text encoding branch, and is stored or updated in the character representation library.

7. The method according to claim 6, characterized in that The method further comprises: Encoding the second text sample by the text encoding branch to obtain a first name representation sample; In response to the presence of a name representation corresponding to the second text sample in the character representation library, updating the name representation corresponding to the second text sample in the character representation library based on the first name representation sample; in response to the absence of a name representation corresponding to the second text sample in the character representation library, storing the first name representation sample as the name representation corresponding to the second text sample in the character representation library; Inputting the first name representation sample and the fourth picture feature sample of the second noise picture sample into the bypass branch, and obtaining a fifth picture feature sample output by the bypass branch; Inputting the first name representation sample, the fifth picture feature sample, and the fourth picture feature sample into the denoising branch to obtain a sixth picture feature sample output by the denoising branch; Performing decoding processing based on the sixth picture feature sample to obtain a second predicted picture corresponding to the second text sample; Obtaining a second loss function value based on a difference between the second predicted picture and the second picture sample; Based on the second loss function value, the parameters of the text encoding branch are updated.

8. The method according to claim 7, characterized in that The text encoding branch and the bypass branch are trained alternately.

9. A method for generating and processing an image, characterized in that: The method is performed by a computer device, and the method comprises: Encoding the first text sample to obtain a first text encoding sample; the first text sample includes the name of the first person; Inputting the first text encoding sample and the first picture feature sample of the first noise picture sample into a bypass branch in the picture generation model, and obtaining a second picture feature sample output by the bypass branch; Inputting the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model to obtain a third picture feature sample output by the denoising branch; Performing decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample; Obtaining a first loss function value based on a difference between the first predicted picture and the first picture sample; Based on the first loss function value, updating the parameters of the bypass branch; Among them, the picture generation model after the bypass branch is updated is used to generate a picture corresponding to the picture description text based on the input picture description text; the picture description text contains the name of the first person.

10. The method according to claim 9, characterized in that The encoding process of the first text sample to obtain the first text encoding sample includes: The first text sample is encoded by the text encoding branch in the picture generation model to obtain an initial text encoding sample; the initial text encoding sample includes a second name representation sample corresponding to the name of the first person; In response to the presence of a name representation corresponding to the name of the first person in the character representation library, replacing the second name representation sample in the initial text encoding sample with the third name representation sample to obtain the first text encoding sample; Among them, the text encoding branch is obtained through training of a second picture-text pair; the second picture sample in the second picture-text pair is a picture of the first person, and the second text sample in the second picture-text pair is the name of the first person; the name representation corresponding to the name of the first person in the character representation library is based on the name representation obtained by encoding the name of the first person during the training process by the text encoding branch, and is stored or updated in the character representation library.

11. A picture generation and processing device, characterized in that: The device comprises: A text acquisition module, used to acquire a picture description text, wherein the picture description text includes the name of the first person; An encoding module, used to encode the picture description text to obtain the picture description text code; A bypass processing module is used to input the picture description text encoding and the first picture feature of the noise picture into a bypass branch in the picture generation model to obtain a second picture feature output by the bypass branch; the bypass branch is obtained by training a first picture-text pair; the first picture sample in the first picture-text pair includes the first person, and the first text sample in the first picture-text pair includes the name of the first person; a denoising processing module, configured to input the picture description text encoding, the second picture feature, and the first picture feature into a denoising branch in the picture generation model, and obtain a third picture feature output by the denoising branch; A decoding module is used to perform decoding processing based on the third picture feature to obtain the picture corresponding to the picture description text.

12. A picture generation and processing device, characterized in that: The device comprises: An encoding module, used for encoding a first text sample to obtain a first text encoding sample; the first text sample includes the name of a first person; A bypass processing module, used for inputting the first text encoding sample and the first picture feature sample of the first noise picture sample into a bypass branch in the picture generation model, and obtaining a second picture feature sample output by the bypass branch; a denoising processing module, configured to input the first text encoding sample, the second picture feature sample, and the first picture feature sample into a denoising branch in the picture generation model, and obtain a third picture feature sample output by the denoising branch; A decoding module, configured to perform decoding processing based on the third picture feature sample to obtain a first predicted picture corresponding to the first text sample; a loss calculation module, configured to obtain a first loss function value based on a difference between the first predicted image and the first image sample; A parameter updating module, used for updating the parameters of the bypass branch based on the first loss function value; Among them, the picture generation model after the bypass branch is updated is used to generate a picture corresponding to the picture description text based on the input picture description text; the picture description text contains the name of the first person.

13. A computer device, characterized in that: The computer device includes a processor and a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image generation processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the image generation and processing method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium; the computer program is read and executed by a processor of a computer device to implement the image generation and processing method as described in any one of claims 1 to 10.

Images