Image generation model training, image generation method, device and computer equipment

Abstract

The present application relates to an image generation model training, image generation method, device, computer equipment, storage medium and computer program product. The method includes: obtaining training description text, training element mask information and training element image corresponding to the training label image, and the training description text includes the training element text corresponding to the training element image. The training description text and the training element image are input into the initial image generation model, and the training element text vector in the training text vector is enhanced based on the training element image vector to obtain the target training text vector, and the target training text vector is adjusted based on the training element mask information and image generation is performed to obtain the training generated image, and iterative training is performed based on the training generated image and the training label image to obtain the image generation model, and the image generation model generates the corresponding target element generated image according to the description text and the corresponding element image. The use of this method can improve the accuracy of image generation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation model training, an image generation method, an apparatus, a computer device, a storage medium and a computer program product. Background Art

[0002] With the development of image generation technology, text-generated image technology has emerged. Text-generated image technology refers to the use of artificial intelligence technology to generate corresponding images based on text descriptions. At present, image generation is performed by inputting text descriptions into a trained artificial intelligence model. When training an artificial intelligence model, detailed text is usually used to describe the elements that appear in the image to be generated, and then the detailed text is used to train the artificial intelligence model for image generation. However, when using detailed text and labeled images to train an artificial intelligence model, due to the problem of mutual influence between multiple descriptive words in the detailed text, each descriptive word cannot accurately correspond to the elements in the labeled image, thereby reducing the accuracy of the trained artificial intelligence model, and then reducing the accuracy of image generation. Summary of the invention

[0003] Based on this, it is necessary to provide an image generation model training, image generation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of image generation quasi-model training and thereby improve the accuracy of image generation in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides an image generation model training method. The method comprises:

[0005] Obtaining training description text and training element images corresponding to the training label images, wherein the training description text includes training element text corresponding to the training element images;

[0006] Input the training description text and the training element image into the initial image generation model, and perform semantic representation on the training description text and the training element image based on the initial image generation model to obtain the corresponding training text vector and training element image vector;

[0007] The training element text vector in the training text vector is enhanced based on the training element image vector by using the initial image generation model to obtain a target training text vector;

[0008] Obtain training element mask information corresponding to the training label image, and calculate the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain an image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain an image adjustment vector, and generate an image based on the image adjustment vector to obtain a training generated image;

[0009] The initial image generation model is updated based on the training generated image and the training label image until the training completion condition is met, and the image generation model is obtained. The image generation model is used to generate a corresponding target element generation image according to the input description text and the corresponding element image.

[0010] In a second aspect, the present application also provides an image generation model training device. The device comprises:

[0011] A training acquisition module, used to acquire training description text and training element images corresponding to the training label images, wherein the training description text includes training element text corresponding to the training element images;

[0012] A semantic representation module is used to input the training description text and the training element image into the initial image generation model, and perform semantic representation on the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors;

[0013] An enhancement module, used for enhancing the training element text vector in the training text vector based on the training element image vector through the initial image generation model to obtain a target training text vector;

[0014] A mask adjustment module is used to obtain the training element mask information corresponding to the training label image, and calculate the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain the image adjustment vector, and generate an image based on the image adjustment vector to obtain a training generated image;

[0015] The model iteration module is used to update the initial image generation model based on the training generated image and the training label image until the training completion condition is met, and the image generation model is obtained. The image generation model is used to generate the corresponding target element generation image according to the input description text and the corresponding element image.

[0016] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned image generation model training method are implemented.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned image generation model training method when executed by a processor.

[0018] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of the above-mentioned image generation model training method when executed by a processor.

[0019] The above-mentioned image generation model training method, device, computer equipment, storage medium and computer program product enhance the training element text vector in the training text vector through the training element image vector to obtain the target training text vector, which can integrate the image semantics of the training element image into the training element text vector, thereby improving the connection between the elements described by the text and the generated image elements. Then, the image-related vector is adjusted using the training element mask information, which can strengthen the connection between the elements described by the text and the shapes of the generated image elements, and make the training elements corresponding to the generated training element image more accurate without affecting the generation of other image elements, thereby improving the accuracy of the image generation quasi-model training, and then the image generation quasi-model is used to generate images, thereby improving the accuracy of image generation. In addition, by using the training element mask information to adjust the image-related vector during image generation, the convergence speed of the parameters during image generation can be enhanced, thereby improving the convergence speed of the overall model training.

[0020] In a first aspect, the present application provides an image generation method. The method comprises:

[0021] Get the current description text and the corresponding current element image, where the current description text includes the current element text corresponding to the current element image;

[0022] Semantically represent the current description text and the current element image respectively to obtain the corresponding current text vector and current element image vector;

[0023] Enhance the current element text vector in the current text vector based on the current element image vector to obtain a current target text vector;

[0024] The reference image vector is obtained, and the correlation between the reference image vector and the current target text vector is calculated to obtain the current image correlation vector corresponding to the current target text vector. The image is generated based on the current image correlation vector to obtain the target generated image.

[0025] In a second aspect, the present application also provides an image generation device. The device comprises:

[0026] A current acquisition module is used to acquire a current description text and a corresponding current element image, wherein the current description text includes a current element text corresponding to the current element image;

[0027] The current representation module is used to perform semantic representation on the current description text and the current element image respectively to obtain the corresponding current text vector and current element image vector;

[0028] A current enhancement module, used for enhancing a current element text vector in a current text vector based on a current element image vector to obtain a current target text vector;

[0029] The image generation module is used to obtain a reference image vector, calculate the correlation between the reference image vector and the current target text vector, obtain a current image correlation vector corresponding to the current target text vector, generate an image based on the current image correlation vector, and obtain a target generated image.

[0030] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned image generation method are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned image generation method are implemented.

[0032] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image generation method are implemented.

[0033] The above-mentioned image generation method, device, computer equipment, storage medium and computer program product obtain the corresponding current text vector and current element image vector by semantically representing the current description text and the current element image respectively, and then use the current element image vector to enhance the current element text vector in the current text vector to obtain the current target text vector, which can integrate the image semantics of the training element image into the training element text vector, thereby enhancing the connection between the elements described by the text and the generated image elements. Finally, the enhanced current target text vector is used for image generation, which can make the current element corresponding to the generated current element image more accurate without affecting the generation of other image elements, thereby improving the accuracy of the obtained target generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A diagram of an application environment of an image generation model training method in one embodiment;

[0035] Figure 2 A schematic diagram of a flow chart of an image generation model training method in one embodiment;

[0036] Figure 3 A schematic diagram of obtaining an enhanced element text vector in a specific embodiment;

[0037] Figure 4 is a schematic diagram of obtaining an image adjustment vector in a specific embodiment;

[0038] Figure 5 is a schematic diagram of image segmentation in a specific embodiment;

[0039] Figure 6 A schematic diagram of a framework for training an initial image generation model in a specific embodiment;

[0040] Figure 7 A schematic diagram of a process for obtaining an initial image generation model in one embodiment;

[0041] Figure 8 A schematic diagram of a pre-training framework in a specific embodiment;

[0042] Fig. 9 is a schematic flow chart of an image generating method in one embodiment;

[0043] Fig.10 A schematic diagram of a framework for image generation in a specific embodiment;

[0044] Fig.11 A schematic diagram of a process of training an image generation model in a specific embodiment;

[0045] Fig.12 It is a structural block diagram of an image generation model training device in one embodiment;

[0046] Fig.13 is a structural block diagram of an image generating device in one embodiment;

[0047] Fig.14 is an internal structure diagram of a computer device in one embodiment;

[0048] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, detect and measure targets, and further performs image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0051] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language used by people in daily life, which is closely related to linguistic research. It also involves computer science and mathematics. It is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. The important technology for model training in the field of artificial intelligence, the pre-training model, is developed from the large language model (LargeLanguage Model) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0052] The solution provided in the embodiments of the present application involves technologies such as image semantic understanding and text semantic understanding of artificial intelligence, which are specifically described by the following embodiments:

[0053] The image generation model training method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other devices. The server 104 can obtain the training description text and training element image corresponding to the training label image uploaded by the terminal 102, and the training description text includes the training element text corresponding to the training element image; the server 104 inputs the training description text and the training element image into the initial image generation model, and semantically represents the training description text and the training element image based on the initial image generation model, and obtains the corresponding training text vector and training element image vector; the server 104 enhances the training element text vector in the training text vector based on the training element image vector through the initial image generation model to obtain the target training text vector; the server 104 obtains the training element mask information corresponding to the training label image, and The model is used to calculate the correlation between the reference image vector and the target training text vector, obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain the image adjustment vector, and generate the image based on the image adjustment vector to obtain the training generated image; the server 104 updates the initial image generation model based on the training generated image and the training label image until the training completion condition is met, and the image generation model is obtained. The image generation model is used to generate the corresponding target element generated image according to the input description text and the corresponding element image. The server 104 can send the obtained image generation model to the terminal 102, and can also generate a model training completion prompt message to send to the terminal 102. Among them, the terminal 102 can be but not limited to various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal 102 and the server 104 may be connected directly or indirectly via wired or wireless communication, which is not limited in the present application.

[0054] In one embodiment, Figure 2 As shown, a method for training an image generation model is provided, and the method is applied to Figure 1 The server 104 in the example is used for explanation. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0055] S202, obtaining training description text and training element images corresponding to the training label image, wherein the training description text includes training element text corresponding to the training element image.

[0056] Among them, the training description text refers to the description text used in the training of the image generation model, and the text content in the training description text is used to describe the image content to be generated. The training label image refers to the label image used in the training of the image generation model, and the training label image is the image corresponding to the text content in the training description text. The training element text is the text in the training description text that describes the training element image. The training element image refers to the image used in the training of the image generation model that includes the image element described by the training element text. For example, the training description text can be "an apple on a table", and the training label image can be an image of an apple on a table. The training element text can be "apple", and the training element image can be an apple image. Image elements refer to the elements that make up the image screen, and can include the subject, ligand, foreground, and background, etc.

[0057] Specifically, the server can obtain the training description text and training element image corresponding to the training label image from the database, and the training description text includes the training element text and the content text other than the training element text. The training element text is the text used to describe the training element image. The server can also obtain the training description text and training element image corresponding to the training label image sent by the service provider providing data services. The server can also obtain the training description text and training element image corresponding to the training label image uploaded by the terminal.

[0058] S204, inputting the training description text and the training element image into the initial image generation model, and semantically representing the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors.

[0059] Among them, the initial image generation model refers to the image generation model of initial parameters, and the initial parameters can be initialized, for example, they can be randomly initialized, zero initialized, Gaussian distribution initialized, etc. The initial parameters can also be obtained through pre-training. The image generation model is established using a neural network algorithm and is used to generate images. The neural network algorithm can be a convolutional neural network algorithm, a recurrent neural network algorithm, a neural network algorithm based on self-attention, a fully connected neural network algorithm, etc. The training text vector refers to a feature vector that represents the semantics of the training description text. The training element image vector is a feature vector used to represent the semantics of the training element image. The feature vector can also be represented by a feature matrix, a feature map, etc.

[0060] Specifically, the server can use a neural network algorithm to establish an image generation model, and obtain the initial parameters in the image generation model to obtain the initial image generation model. Then, the initial image generation model is trained, that is, the training description text and the training element image are input into the initial image generation model. The initial image generation model can encode the training description text to extract the semantic representation to obtain the training text vector, wherein the training description text can be input into the text encoding network for text encoding. The initial image generation model can encode the training element image to extract the semantic representation to obtain the training element image vector, wherein the training element image can be divided into various image regions, and each image region can be encoded separately to obtain the training element image vector, or the training element image can be input into the image encoding network for image encoding to obtain the training element image vector.

[0061] S206, enhancing the training element text vector in the training text vector based on the training element image vector through the initial image generation model to obtain a target training text vector.

[0062] The training element text vector is a feature vector corresponding to the training element text, and the target training text vector refers to a feature vector obtained by enhancing the training element text vector in the training text vector.

[0063] Specifically, the initial image generation model in the server can use the training element image vector to enhance the training element text vector in the training text vector, wherein the training element image vector can be semantically fused with the training element text vector to obtain a target training text vector, and the training element image vector can also be spliced ​​with the training element text vector, and then the spliced ​​vector is semantically fused with the training element text vector to obtain a target training text vector.

[0064] S208, obtaining the training element mask information corresponding to the training label image, and calculating the correlation between the reference image vector and the target training text vector through the initial image generation model, obtaining the image correlation vector corresponding to the target training text vector, and adjusting the image correlation vector based on the training element mask information to obtain the image adjustment vector, and generating an image based on the image adjustment vector to obtain a training generated image.

[0065] The training element mask information is the information obtained by binarizing the training label image. The image correlation vector is used to represent the correlation between the target training text vector and the reference image vector, and the image correlation vector can be an attention map. The image adjustment vector refers to the feature vector obtained by re-weighting the image correlation vector using the training element mask information. The reference image vector refers to the semantic representation of the randomly generated noise image.

[0066] Specifically, the server can obtain the training element mask information corresponding to the training label image from the database, or binarize the training label image to obtain the training element mask information, which can be a black and white image, in which the white area is a high-weight area and the black area is a low-weight area. The high-weight area represents the training element area in the training label image, and the low-weight area represents the area other than the training element area in the training label image. Then the initial image generation model in the server calculates the correlation between the reference image vector and the target training text vector through the attention mechanism, obtains the image correlation vector corresponding to the target training text vector, and uses the training element mask information to re-weight the image correlation vector to obtain the image adjustment vector, and finally uses the image adjustment vector for decoding to obtain the training generated image.

[0067] S210, updating the initial image generation model based on the training generated image and the training label image until the training completion condition is met, and obtaining the image generation model, the image generation model is used to generate the corresponding target element generation image according to the input description text and the corresponding element image.

[0068] Among them, the training completion condition is the condition for stopping the training of the initial image generation model, including but not limited to the number of training iterations reaching the maximum number of iterations, the training model parameters no longer changing, and the training loss information being less than a preset loss threshold. The description text refers to the text that describes the content of the image to be generated. The element image refers to the image corresponding to the element text in the description text, and the element text is the text that describes the image element to be generated. The image element can be the subject in the image, the ligand in the image, the background in the image, or the foreground in the image. The target element generated image refers to an image including the elements corresponding to the element image generated using the description text and the corresponding element image.

[0069] Specifically, the server uses a pre-set loss function to calculate the error between the training generated image and the training label image to obtain the model training loss, and then uses the model training loss to reversely update the initial parameters in the initial image generation model through the gradient descent algorithm to obtain the updated image generation model. The server then performs a loop iteration, that is, the updated image generation model is used as the initial image generation model, and the step of returning to obtain the training description text and training element image corresponding to the training label image is iteratively executed until the training completion condition is met, and the initial image generation model that finally meets the training completion condition is used as the trained image generation model. The image generation model is used to generate the corresponding target element generation image according to the input description text and the corresponding element image.

[0070] The above-mentioned image generation model training method, device, computer equipment, storage medium and computer program product enhance the training element text vector in the training text vector through the training element image vector to obtain the target training text vector, which can integrate the image semantics of the training element image into the training element text vector, thereby improving the connection between the elements described by the text and the generated image elements. Then, the image-related vector is adjusted using the training element mask information, which can strengthen the connection between the elements described by the text and the shapes of the generated image elements, and make the training elements corresponding to the generated training element image more accurate without affecting the generation of other image elements, thereby improving the accuracy of the image generation quasi-model training, and then the image generation quasi-model is used to generate images, thereby improving the accuracy of image generation. In addition, by using the training element mask information to adjust the image-related vector during image generation, the convergence speed of the parameters during image generation can be enhanced, thereby improving the convergence speed of the overall model training.

[0071] In one embodiment, S206, enhancing the training element text vector in the training text vector based on the training element image vector to obtain a target training text vector includes the steps of:

[0072] The training element image vector and the training element text vector are fused to obtain a training fusion vector, and the training fusion vector is merged with the training element text vector to obtain an enhanced element text vector; the training element text vector in the training text vector is replaced with the enhanced element text vector to obtain a target training text vector.

[0073] The server fuses the training element image vector and the training element text vector, and the fusion may be performed by vector concatenation, vector operation, neural network operation, etc., to obtain a training fusion vector. The server then merges the training fusion vector with the training element text vector, and the fusion may be performed by weighting the training fusion vector and performing vector operation with the training element text vector to obtain an enhanced element text vector. Finally, the server replaces the training element text vector in the training text vector with the enhanced element text vector to obtain a target training text vector.

[0074] In one embodiment, when the dimensions of the training fusion vector and the training element text vector are inconsistent, the training fusion vector and the training element text vector can be converted into feature vectors of the same dimension and then fused, thereby improving the accuracy of the fusion.

[0075] In the above embodiment, the training element image vector and the training element text vector are fused to obtain a training fusion vector, and the training fusion vector is merged with the training element text vector to obtain an enhanced element text vector, thereby improving the accuracy of the obtained enhanced element text vector, and then improving the accuracy of the obtained target training text vector, that is, improving the association between the element text vector and the image element in the model, thereby improving the representation ability of the text vector in the image generation process, and then improving the accuracy of image generation.

[0076] In one embodiment, the training element image vector and the training element text vector are fused to obtain a training fused vector, and the training fused vector is merged with the training element text vector to obtain an enhanced element text vector, including the steps of:

[0077] The training element image vector and the training element text vector are spliced ​​to obtain a training spliced ​​vector, and feature interaction of attention is performed based on the training spliced ​​vector to obtain a training fusion vector; the enhancement weight is obtained, the training fusion vector is weighted based on the enhancement weight to obtain an enhanced fusion vector, and the sum of the enhanced fusion vector and the training element text vector is calculated to obtain an enhanced element text vector.

[0078] The training splicing vector refers to the splicing vector obtained by training. The training splicing vector can be a feature vector obtained by splicing the training element image vector and the training element text vector head to tail. The training splicing vector can also be a feature matrix obtained by splicing the training element image vector and the training element text vector up and down. The enhancement weight refers to a pre-set weight for weighting the training fusion vector. The enhancement weight is a pre-set hyperparameter.

[0079] Specifically, the server concatenates the training element image vector and the training element text vector up and down to obtain a training concatenation feature matrix, and then performs attention feature interaction on the training concatenation feature matrix through an attention mechanism to obtain a training fusion vector. For example, the training concatenation vector can be input into a cross-attention network for attention feature interaction, and the cross-attention network can be pre-trained. At this point, the server can obtain the enhancement weight from the database, and then calculate the product of the enhancement weight and the training fusion vector to obtain the enhancement fusion vector, and calculate the sum of the enhancement fusion vector and the training element text vector to obtain the enhancement element text vector.

[0080] In a specific embodiment, Figure 3As shown, it is a schematic diagram of obtaining an enhanced element text vector. Specifically, the server obtains a 768-dimensional training element text vector and a 2048-dimensional training element image vector, and then first converts the 768-dimensional training element text vector into a 2048-dimensional vector, that is, the 768-dimensional training element text vector can be converted through a fully connected network (MLP) and a 1-dimensional convolutional neural network (1Conv) to obtain a 2048-dimensional training element text vector. Then the 2048-dimensional training element image vector and the 2048-dimensional training element text vector are aligned, and then spliced ​​on the same dimension to obtain a training splicing feature matrix. Then the training splicing feature matrix is ​​input into the attention network for fusion to obtain a 768-dimensional training fusion vector. The attention network can be established using a fully connected network (MLP) and a 2-dimensional convolutional neural network (2Conv). Then, the 768-dimensional training element text vector is enhanced using the 768-dimensional training fusion vector, that is, the product of the enhancement weight and the training fusion vector is obtained to obtain the enhanced fusion vector, and the sum of the enhanced fusion vector and the training element text vector is calculated to obtain the enhanced element text vector.

[0081] In the above embodiment, the enhanced fusion vector is calculated and then used to enhance the training element text vector to obtain the enhanced element text vector, thereby improving the accuracy of the obtained enhanced element text vector.

[0082] In one embodiment, S208, obtaining training element mask information corresponding to the training label image, and calculating the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain the image correlation vector corresponding to the target training text vector, includes the steps of:

[0083] The training label image is binarized based on the training element image to obtain the training element mask information; the reference image vector is denoised through the initial image generation model to obtain the training denoised image vector, and the training denoised image vector is attention-weighted based on the target training text vector to obtain the image-related vector corresponding to the target training text vector.

[0084] The training denoised image vector refers to a feature vector obtained by denoising the reference image vector.

[0085] Specifically, the server converts the same element area in the training label image into a high-weight area according to the training element image, and converts other areas in the training label image into low-weight areas to obtain the training element mask information. For example, the pixel values ​​of the element area corresponding to the training element image in the training label image can be converted to 255, and the pixel values ​​of other areas can be converted to 0 to obtain a black and white image (mask image). At this time, the black and white image is the training element mask information. Then the denoising neural network in the initial image generation model denoises the reference image vector to obtain a training denoised image vector, and the target training text vector is weighted by attention to the training denoised image vector to obtain the image-related vector corresponding to the target training text vector. For example, the reference image vector is denoised by the initial deep residual network (ResNet) in the initial image generation model to obtain a training denoised image vector, and then the training denoised image vector and the target training text vector are input into the initial attention network for attention weighting to obtain an image-related vector (attention map).

[0086] In the above embodiment, by denoising the reference image vector to obtain a training denoised image vector, and performing attention weighting on the training denoised image vector based on the target training text vector, the connection between the text semantics and the shape of the image element to be generated can be strengthened, thereby improving the accuracy of the obtained image-related vector.

[0087] In one embodiment, S208, adjusting the image-related vector based on the training element mask information to obtain the image adjustment vector includes the steps of:

[0088] A training element related vector is obtained from the image related vector, and the training element related vector is weighted based on the training element mask information to obtain a training element adjustment vector; and the training element related vector in the image related vector is replaced with the training element adjustment vector to obtain an image adjustment vector.

[0089] The training element related vector refers to the feature vector corresponding to the training element text in the image related vector.

[0090] Specifically, the server can determine the training element related vector corresponding to the training element text from the image related vector according to the position of the training element text in the training description text. Then, the training element related vector is weighted using the training element mask information to obtain the training element adjustment vector. At this time, the server replaces the training element related vector in the image related vector with the training element adjustment vector, and keeps other related vectors in the image related vector unchanged to obtain the image adjustment vector.

[0091] In a specific embodiment, Figure 4As shown in the figure, it is a schematic diagram for obtaining an image adjustment vector. Specifically, an image-related vector is obtained, which includes a training element-related vector corresponding to the training element text in the training description text and a vector corresponding to other text content. For example, if the training description text is "a person holding a game prop", the image-related vector includes a feature vector corresponding to "a person", a feature vector corresponding to "holding", and a training element-related vector corresponding to "game prop". At this time, the server obtains the training element-related vector corresponding to "game prop" from the image-related vector, and at the same time obtains the training element mask information corresponding to "game prop" of the training label image, that is, the Mask information. Then, the training element mask information is used to re-weight the training element-related vector to obtain the training element adjustment vector, and the image adjustment vector is obtained according to the training element adjustment vector and other unchanged related vectors.

[0092] In the above embodiment, the training element related vectors are weighted by using the training element mask information to obtain the training element adjustment vector, so as to strengthen the connection between the text semantics and the shape of the image element to be generated, and ensure that other image contents to be generated are not affected, thereby improving the accuracy of the obtained image adjustment vector.

[0093] In one embodiment, weighting the training element related vector based on the training element mask information to obtain the training element adjustment vector includes the steps of:

[0094] The white pixel value is obtained, and a ratio calculation is performed based on the mask information of the training element and the white pixel value to obtain a weighted weight, and a product operation is performed based on the weighted weight and the training element related vector to obtain a weighted vector; the sum of the weighted vector and the training element related vector is calculated to obtain a weighted related vector, and an average is performed based on the weighted related vector to obtain a training element adjustment vector.

[0095] The white pixel value refers to the pixel value of a white pixel, which is 255.

[0096] Specifically, the server obtains the white pixel value, then calculates the ratio of each mask value to the white pixel value in the training element mask information to obtain the weighted weight, then calculates the product of the weighted weight and the training element related vector to obtain the weighted vector. Then the weighted vector is added to the training element related vector to obtain the weighted related vector, and finally the weighted related vector is evenly divided to obtain the training element adjustment vector.

[0097] In a specific embodiment, the training element adjustment vector may be calculated using the formula (1) shown below.

[0098] Formula (1)

[0099] Where M refers to the training element related vector, that is, the attention map corresponding to the training element. W refers to the training element mask information. Reweight refers to the training element adjustment vector obtained after re-weighting.

[0100] In the above embodiment, a weighted weight is obtained by performing a ratio calculation based on the training element mask information and the white pixel value, and a weighted vector is obtained by performing a multiplication operation based on the weighted weight and the training element related vector; the sum of the weighted vector and the training element related vector is calculated to obtain a weighted related vector, and the weighted related vector is averaged to obtain a training element adjustment vector, that is, the training element mask information is adjusted by the white pixel value and then the training element related vector is weighted, so that the obtained training element adjustment vector can contain the information of the training element shape, that is, the connection between the text guidance concept and the element to be generated is enhanced, thereby improving the accuracy of the obtained training element adjustment vector.

[0101] In one embodiment, S208, performing image generation based on the image adjustment vector to obtain a training generated image includes the following steps:

[0102] The image adjustment vector is used as the target training text vector, and the correlation between the reference image vector and the target training text vector is returned to obtain the image correlation vector corresponding to the target training text vector. The image correlation vector is adjusted based on the training element mask information to obtain the image adjustment vector. The steps are iteratively executed until the iteration completion condition is met to obtain the output target training image vector. Decoding is performed based on the target training image vector to obtain the training generated image.

[0103] The iteration completion condition refers to the condition that the adjustment using the training element mask information is completed, which may include but is not limited to the number of iterations reaching a preset number.

[0104] Specifically, the server can use the training element mask information to make multiple adjustments, that is, the image adjustment vector can be used as the target training text vector, and the correlation between the reference image vector and the target training text vector can be returned to obtain the image correlation vector corresponding to the target training text vector, and the image correlation vector is adjusted based on the training element mask information. The step of obtaining the image adjustment vector is iteratively executed until a preset number of iterations is reached, and the target training image vector outputted by the last iteration is obtained, and then the target training image vector is decoded, for example, upsampling can be performed to obtain a training generated image.

[0105] In a specific embodiment, the initial image generation model to be trained can be established using a Stable diffusion (a potential diffusion model) model. Multiple U-Net neural networks can be used in the Stable diffusion model to generate images, and each U-Net neural network can use the training element mask information to reweight the attention map calculated by the U-Net (a convolutional neural network) neural network to obtain an image adjustment vector, and the image adjustment vector is used as the input of the next U-Net neural network to continue iterating until the output of the last U-Net neural network is obtained, that is, the final image adjustment vector is obtained, and then the final image adjustment vector is used for decoding to obtain the training generated image.

[0106] In the above embodiment, the image adjustment vector is used as the target training text vector, and then the target training text vector is adjusted through multiple iterations using the training element mask information. When the iteration completion condition is reached, the output target training image vector is obtained, which further strengthens the connection between the text-guided concept and the element to be generated, thereby making the target training image vector more accurate. Finally, the target training image vector is used for decoding to obtain the training generated image, thereby improving the accuracy of the training generated image.

[0107] In one embodiment, S202, obtaining training description text and training element images corresponding to the training label image, includes the steps of:

[0108] A target image of a target style type is obtained, and the target image is used as a training label image; image content recognition is performed based on the target image to obtain a training description text, and target element region segmentation is performed based on the target image to obtain a training element image.

[0109] The target style type refers to a preset style type of an image to be generated, such as an animation style, a film and television style, etc. The style can be different image art styles.

[0110] Specifically, the server can obtain multiple target images of the target style type from the database, and then use each target image as a training label image. Perform content recognition on each target image to obtain the training description text corresponding to each target image, wherein a pre-trained image content recognition model can be used for content recognition, and the image content recognition model can be established using a neural network algorithm, and trained using training images and corresponding training label texts. At the same time, the server can perform target element area segmentation on each target image to obtain the training element image corresponding to each target image, wherein the pre-trained image segmentation model can be used for segmentation, and the image segmentation model can be established using a neural network algorithm, and trained using training images and corresponding training segmentation label results. At this point, the image generation training data of the target style type is obtained. Then, the image generation training data of the target style type is used to train the initial image generation model, so that the trained image generation model can improve the accuracy of image generation corresponding to the target style type.

[0111] In a specific embodiment, the server can input all image data into the caption taggermodel (image content recognition model), detect and identify the content in the image, generate corresponding image description text, and then use the image description text and the corresponding image to train the image generation model. Among them, the image content recognition model can be a BLIP (a unified visual language understanding and generation model) model. At the same time, the server can input the target image into the image segmentation model to perform target element area segmentation to obtain a training element image. Among them, the image segmentation model can be a BiSeNet (bilateral segmentation network for real-time semantic segmentation) model. The network of the BiSeNet model mainly includes two parts, namely the spatial path and the context path. Among them, there is an attention refinement network (aattention refinment module) in the context path. Finally, the two branches of the spatial path and the context path are fused through the feature fusion network (feature fusion module) to obtain the fused feature for further upsampling calculation and generation. The attention refinement network uses global average pooling to guide the generation of features. The spatial information captured by the spatial path encodes rich detail information. The output features of the context path mainly encode semantic information. Then, the fusion is performed through feature fusion to generate fusion features. Finally, the fused features are upsampled according to the size of the input original image, and the size of the output features is kept consistent with the input image. After upsampling, semantic judgment is performed on each pixel point. After the pixels of the same category are aggregated, the distribution of different semantic elements in the image can be calculated, so as to locate the dividing line between different elements, and then extract the required image elements. That is, the target image can be segmented and the training element image can be extracted, such as Figure 5 As shown, it is a schematic diagram of image segmentation, in which the game character image on the left side of the game style type is segmented into prop elements to obtain the prop element image on the right side.

[0112] In the above embodiment, by using the training label image to perform image content recognition to obtain the training description text, and performing target element region segmentation on the training label image to obtain the training element image, and then using the training element image and the training description text model training data to perform model training, the accuracy of the training label image can be guaranteed, thereby improving the accuracy of the model training.

[0113] In a specific embodiment, Figure 6As shown in FIG. 1 , it is a schematic diagram of the framework of the initial image generation model training. Specifically, the server obtains the game character image, inputs the game character image into the image content generation model for image content recognition, obtains the output description text of the element in the image "1 person holding a prop", and inputs the game character image into the image segmentation model for image segmentation to obtain the output prop element image. Then the description text and the prop element image are used as training data for training. That is, the server inputs the description text into the text encoding network for semantic representation to obtain the training text vector, wherein the text encoding network divides the description text into pre-set basic units to obtain each basic unit text, which can be a unit for text segmentation such as a character or a word. Then, each basic unit text is vectorized to obtain each basic unit feature vector, and each basic unit vector includes the feature vector of the prop text. At the same time, the server inputs the prop element image into the image encoding network for semantic representation to obtain the prop element image vector. For example, the image encoding network can be an image encoding model built based on resnet50 (residual network). At this time, the server fuses the feature vector of the prop element image and the feature vector of the prop text to obtain a fusion vector, and merges the fusion vector with the feature vector of the prop text to obtain an enhanced prop text vector, and then obtains a target training text vector. At this time, the target training text vector is used to generate an image through the U-Net network, that is, the cross-attention network in the U-Net network uses the target training text vector to calculate the image-related vector, and then the prop area mask image of the game character image is obtained, and the prop area mask image is used to perform attention weighting on the image-related vector to obtain an image adjustment vector, and then the image adjustment vector is input into the next cross-attention network for iteration until the target training image vector is obtained, and then the target training image vector is input into the decoding network for decoding to obtain a training generated image, and at this time, the training generated image and the game character image are used for loss calculation, and the initial image generation model is updated using the gradient descent algorithm, and the loop iteration is continuously performed until the training completion condition is met, and the trained game image generation model is obtained, and then the game image generation model is used to generate the game prop image, thereby improving the accuracy of the prop element generation in the game image.

[0114] In one embodiment, Figure 7 As shown, before S202, that is, before obtaining the training description text and the training element image corresponding to the training label image, the following step is also included:

[0115] S702, obtaining pre-trained description text corresponding to the pre-trained label image.

[0116] Specifically, the server may obtain the pre-trained description text corresponding to the pre-trained label image from the database. The style type of the pre-trained label image is the same as the label image used in the training process. The pre-trained description text may be obtained by performing image content recognition on the pre-trained label image. The style type of the pre-trained label image is the same as or similar to the style type of the training label image used in the training process.

[0117] S704, input the pre-trained description text into the pre-trained image generation model, perform semantic representation on the pre-trained description text based on the pre-trained image generation model, obtain a pre-trained text vector corresponding to the pre-trained description text, calculate the correlation between the reference image vector and the pre-trained text vector, obtain a pre-trained image correlation vector, and generate an image based on the pre-trained image correlation vector to obtain a pre-trained generated image.

[0118] The pre-trained image generation model is an image generation model that needs to be pre-trained. The pre-trained image generation model may use a stable diffusion model.

[0119] Specifically, the server inputs the pre-trained description text into the pre-trained image generation model, and the pre-trained image generation model performs semantic representation on the pre-trained description text to obtain a pre-trained text vector corresponding to the pre-trained description text, which can be semantically represented by a text encoder network, and the text encoder network (text encoder) can be established using a neural network. Then the server inputs the pre-trained text vector into the image generation network in the pre-trained image generation model to calculate the correlation between the reference image vector and the pre-trained text vector, obtain a pre-trained image correlation vector, and generate an image based on the pre-trained image correlation vector to obtain a pre-trained generated image.

[0120] S706, updating the pre-trained image generation model based on the pre-trained generated image and the pre-trained label image until the pre-training completion condition is met, thereby obtaining an initial image generation model.

[0121] Specifically, the server uses a loss function to calculate the loss between the pre-trained generated image and the pre-trained labeled image, and then uses a descent algorithm to reversely update the pre-trained image generation model according to the loss, and when the pre-training completion condition is met, the pre-trained image generation model that has reached the pre-training completion is used as the initial image generation model. The loss function may use the loss function used when the image is generated, for example, a mean square error loss function.

[0122] In the above embodiment, the pre-trained image generation model is pre-trained using the pre-trained label image and the corresponding pre-trained description text to obtain the initial image generation model, thereby improving the accuracy of image generation by the obtained initial image generation model and improving the efficiency of subsequent model training.

[0123] In one embodiment, S702, obtaining pre-trained description text corresponding to the pre-trained label image includes the steps of:

[0124] Obtain a target image corresponding to the target style type; use the target image as a pre-trained label image, and perform image content recognition based on the target image to obtain a pre-trained description text.

[0125] Specifically, the server can obtain a large number of target images corresponding to the target style type used in the training process for pre-training. That is, the server can use the target image as a pre-training label image, and perform content recognition on the target image to obtain the training description text corresponding to each target image, wherein a pre-trained image content recognition model can be used for content recognition. The image content recognition model can be established using a neural network algorithm and trained using training images and corresponding training label texts. For example, the image content recognition model can be a BLIP model. That is, the pre-training description text and the training description text are the same text.

[0126] In one embodiment, the server can obtain images of various style types, and then use the images of various style types for pre-training to obtain a pre-trained image generation model that can generate images of various style types, and then use the pre-trained image generation model of each style type image as the initial image generation model, and obtain images of a specific style type and specific element images of the specific style type to train the initial image generation model, thereby obtaining an image generation model that can generate images of a specific style type containing specific element images, that is, using a large number of images of different style types for pre-training, and then using images of a specific style type for fine-tuning training, thereby improving the training efficiency of the image generation model.

[0127] In the above embodiment, by using a large number of target images of the target style type for pre-training, the model can deeply feel the representation of the target style type, so that it can learn the style type that needs to be represented, while maintaining sufficient generalization ability for the generation of image elements of the target style type, thereby ensuring the normal generation of other image elements during model training.

[0128] In a specific embodiment, Figure 8As shown, it is a schematic diagram of the pre-training framework. Specifically, the server obtains a large number of game-style images and saves them in the database for pre-training. During pre-training, the server obtains the game-style images from the database, and then uses the image content generation model to perform content recognition on the game-style images to obtain pre-trained description text. Then the server inputs the pre-trained description text into the text encoding network in the pre-trained image generation model for semantic representation to obtain a text feature vector. The pre-trained image generation model can be a stable diffusion model. The text encoding network can use a trained neural network model for semantic representation of text. Then the text feature vector is input into the image generation network in the pre-trained image generation model. The parameters in the image generation network need to be pre-trained and obtained through training. The cross-attention network in the image generation network performs attention feature interaction to obtain the output attention map, and then the attention map is input into the decoding network for decoding to obtain the pre-trained generated image. Then the mean square error loss function calculates the error between the pre-trained generated image and the pre-trained label image to obtain the loss information. The loss information is used to reversely update the parameters in the pre-trained image generation model through the gradient descent algorithm, and the cycle iteration is continued until the pre-training completion condition is reached, and the pre-trained image generation model obtained in the last iteration is used as the initial image generation model. That is, by using a large amount of game style image data to pre-train the pre-trained image generation model, the pre-trained image generation model can have a deep perception of the style of the game during the pre-training stage, and then when training for a specific prop element, the pre-trained image generation model can directly learn the specific prop element. The learning perception of the game style in the pre-training can accelerate the learning speed of the prop element, and at the same time can also improve the restoration of the game prop element.

[0129] In one embodiment, Fig. 9 As shown, a method for generating an image is provided, and the method is applied to Figure 1 The server 104 in the example is used for explanation. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0130] S902, obtaining a current description text and a corresponding current element image, wherein the current description text includes a current element text corresponding to the current element image;

[0131] S904, semantically represent the current description text and the current element image respectively to obtain a corresponding current text vector and current element image vector.

[0132] Among them, the text content in the current description text is used to describe the content of the image to be generated. The current element text refers to the text in the current description text that describes the current element image. The current element image is an image that includes the image element described by the current element text. That is, the current element image is a specific element image, and the image to be generated includes the specific element. The current element text refers to the text that describes the specific element. For example, the current description text can be "a person holding a mobile phone", and the specific element can be a mobile phone. In this case, the current element image is the mobile phone image, and the current element text is "mobile phone".

[0133] Specifically, the server can obtain the current description text and the corresponding current element image from the database, and the current description text includes the current element text corresponding to the current element image. The server can also obtain the current description text and the corresponding current element image uploaded by the terminal. The server can also obtain the current description text and the corresponding current element image sent by the business service provider providing image generation services. The server then semantically represents the current description text and the current element image respectively to obtain the corresponding current text vector and current element image vector, wherein the text can be semantically represented using an existing text representation model, and the image can be semantically represented using an existing image representation model, the existing text representation model can be a neural network model, and the existing image representation model can also be a neural network model.

[0134] S906, enhancing the current element text vector in the current text vector based on the current element image vector to obtain a current target text vector.

[0135] The current target text vector refers to a text vector obtained by enhancing the current element text vector using the current element image vector.

[0136] Specifically, the server may semantically fuse the training element image vector with the training element text vector to obtain the target training text vector. The server may also splice the training element image vector with the training element text vector, and then semantically fuse the spliced ​​vector with the training element text vector to obtain the target training text vector.

[0137] S908, obtaining a reference image vector, and calculating the correlation between the reference image vector and the current target text vector, obtaining a current image correlation vector corresponding to the current target text vector, performing image generation based on the current image correlation vector, and obtaining a target generated image.

[0138] The current image correlation vector refers to the image correlation vector calculated when the image is generated, which is used to characterize the correlation between the text and the reference image. The reference image can be a randomly generated noise image, and when the image is generated, an image that matches the input text is generated by continuously removing noise.

[0139] Specifically, the server can randomly generate a reference image, and then semantically represent the reference image to obtain a reference image vector. Then, the correlation between the reference image vector and the current target text vector is calculated. The reference image vector and the current target text vector can be cross-attentionally interacted with each other to obtain the current image-related vector corresponding to the current target text vector. Then, the current image-related vector is decoded, where upsampling and skip-layer connections can be used to decode the target generated image.

[0140] The above-mentioned image generation method, device, computer equipment, storage medium and computer program product obtain the corresponding current text vector and current element image vector by semantically representing the current description text and the current element image respectively, and then use the current element image vector to enhance the current element text vector in the current text vector to obtain the current target text vector, which can integrate the image semantics of the training element image into the training element text vector, thereby enhancing the connection between the elements described by the text and the generated image elements. Finally, the enhanced current target text vector is used for image generation, which can make the current element corresponding to the generated current element image more accurate without affecting the generation of other image elements, thereby improving the accuracy of the obtained target generated image.

[0141] In one embodiment, the image generation method further includes the steps of:

[0142] The current description text and the current element image are input into the image generation model. The image generation model is obtained by updating the initial image generation model based on the training generated image and the training label image. The training generated image is obtained by adjusting the image correlation vector using the training element mask information corresponding to the training label image to obtain the image adjustment vector, and the image is generated based on the image adjustment vector. The image correlation vector is obtained by calculating the correlation between the reference image vector and the target training text vector. The target training text vector is obtained by enhancing the training element text vector in the training text vector using the training element image vector. The current description text and the current element image are semantically represented respectively by the image generation model to obtain the corresponding current text vector and current element image vector. The current element text vector in the current text vector is enhanced based on the current element image vector by the image generation model to obtain the current target text vector. The correlation between the reference image vector and the current target text vector is calculated by the image generation model to obtain the current image correlation vector corresponding to the current target text vector. Image generation is performed based on the current image correlation vector to obtain the target generated image.

[0143] The image generation model may be obtained by training using any one of the methods in the above-mentioned image generation model training method.

[0144] Specifically, the server can input the current description text and the current element image into the image generation model, and then use the image generation model to generate the image. The image generation model can be obtained by using the training method of any one of the above-mentioned image generation model training methods. The server obtains the image generation model trained by using the method of any one of the above-mentioned image generation model training methods, and then deploys the image generation model. When the image generation model needs to be used, the image generation model is called for use to obtain the output target generated image.

[0145] In the above embodiment, by using the image generation model trained by the image generation model training method to perform image generation, the accuracy of generating specific image elements in the image is improved, thereby improving the accuracy of image generation.

[0146] In one embodiment, S906, enhancing the current element text vector in the current text vector based on the current element image vector to obtain the current target text vector includes the steps of:

[0147] The current element image vector and the current element text vector are fused to obtain a current fused vector, and the current fused vector is merged with the current element text vector to obtain a current enhanced element text vector; the current element text vector in the current text vector is replaced with the current enhanced element text vector to obtain a current target text vector.

[0148] In this embodiment, the server may perform vector splicing of the current element image vector and the current element text vector, or may perform fusion through vector operation, or may perform fusion through neural network operation, etc., to obtain the current fusion vector. The server then merges the current fusion vector with the current element text vector, and the fusion may be performed by weighting the current fusion vector and performing vector operation with the current element text vector to obtain the current enhanced element text vector. Finally, the server replaces the training current element text vector in the current text vector with the current enhanced element text vector to obtain the current target text vector. In one embodiment, if the dimensions of the current fusion vector and the current element text vector are inconsistent, the current fusion vector and the current element text vector may be converted into feature vectors of the same dimension and then fused, thereby improving the accuracy of the fusion.

[0149] Furthermore, the server can splice the current element image vector and the current element text vector to obtain the current splicing vector, and perform self-attention feature interaction based on the current splicing vector to obtain the current fusion vector; obtain the enhancement weight, weight the current fusion vector based on the enhancement weight to obtain the current enhanced fusion vector, and calculate the sum of the current enhanced fusion vector and the current element text vector to obtain the current enhanced element text vector.

[0150] Specifically, the server concatenates the current element image vector and the current element text vector head to tail to obtain the current concatenation vector, or concatenates them top to bottom to obtain the current concatenation feature matrix. The current concatenation feature matrix is ​​then subjected to attention feature interaction through an attention mechanism to obtain the current fusion vector. At this point, the product of the enhancement weight and the current fusion vector is calculated to obtain the current enhanced fusion vector, and the current enhanced fusion vector is added to the current element text vector to obtain the current enhanced element text vector. In a specific embodiment, the server may use the following method: Figure 3 The enhancement method shown enhances the current element text vector through the current element image vector to obtain the current enhanced element text vector.

[0151] In the above embodiment, the current enhanced element text vector is obtained by calculating the current enhanced fusion vector and then using the current enhanced fusion vector to enhance the training element text vector, thereby improving the accuracy of the current enhanced element text vector and further improving the accuracy of the current target text vector.

[0152] In one embodiment, the current element image includes at least two, and the current element image vector includes current element image vectors corresponding to at least two current element images respectively; the current element image vector and the current element text vector are fused to obtain a current fused vector, and the current fused vector is merged with the current element text vector to obtain a current enhanced element text vector, including the steps of:

[0153] The current element text vector is merged with the current element image vectors corresponding to at least two current element images to obtain current fused vectors corresponding to at least two current element images; an average calculation is performed based on the current fused vectors corresponding to at least two current element images to obtain a current average vector, and the current average vector is merged with the current element text vector to obtain a current enhanced element text vector corresponding to at least two current element images.

[0154] Specifically, the server can obtain multiple current element images corresponding to the image element to be generated, and the multiple can be at least two. Then the server splices the current element text vector with the image vector corresponding to each current element image, and then performs cross-attention feature interaction to obtain the current fusion vector corresponding to each current element image, and then averages all current fusion vectors to obtain the current average vector, and finally merges the current average vector with the current element text vector to obtain the enhancement weight, calculate the product of the enhancement weight and the current average vector, obtain the enhanced average vector, and calculate the sum of the enhanced average vector and the current element text vector to obtain the current enhanced element text vector corresponding to at least two current element images.

[0155] In the above embodiment, a current enhanced element text vector is obtained by fusing multiple current element images with the current element text vector, thereby improving the generalization of the generation of a specific element corresponding to the current element image during the generation process.

[0156] In a specific embodiment, Fig.10As shown, it is a schematic diagram of the framework of image generation. Specifically, the server obtains the game description text and three prop element images, and the game description text includes the prop text corresponding to the prop element. Then the server inputs the game description text and the three prop element images into the image generation model. The image generation model characterizes the game character description text through the text network to obtain a text vector, which includes the feature vector corresponding to the prop text and the feature vector corresponding to other basic unit texts. At the same time, the three prop element images are encoded through the image coding network to obtain three image vectors. Then the server fuses the three image vectors with the feature vectors corresponding to the prop text respectively to obtain the current fusion vector 1, the current fusion vector 2 and the current fusion vector 3. Then the current fusion vector 1, the current fusion vector 2 and the current fusion vector 3 are averaged to obtain the current average vector, and then the current average vector is merged with the feature vector corresponding to the prop text to obtain an enhanced text vector, which includes the enhanced vector corresponding to the prop text and the feature vectors corresponding to other basic unit texts. Then the image generation model uses the enhanced text vector for image generation, that is, image generation is performed through a cross-attention network and a decoding network to obtain an output game character image including prop elements. The prop text is enhanced by using multiple prop element images, thereby improving the accuracy of prop element generation in the image generation process, and improving the restoration and generalization of prop element generation.

[0157] In a specific embodiment, Fig.11 As shown, it is a flowchart of the image generation model training method, which is executed by a computer device, which can be a terminal or a server, and specifically includes the following steps:

[0158] S1102, obtaining a target image of a target style type, using the target image as a training label image, performing image content recognition based on the target image to obtain training description text, and performing target element region segmentation based on the target image to obtain a training element image. The training description text includes training element text corresponding to the training element image.

[0159] S1104, input the training description text and the training element image into the initial image generation model, and perform semantic representation on the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors.

[0160] S1106, splicing the training element image vector and the training element text vector through the initial image generation model to obtain a training splicing vector, and performing feature interaction of attention based on the training splicing vector to obtain a training fusion vector, obtaining an enhancement weight, weighting the training fusion vector based on the enhancement weight to obtain an enhanced fusion vector, and calculating the sum of the enhanced fusion vector and the training element text vector to obtain an enhanced element text vector.

[0161] S1108, binarize the training label image based on the training element image to obtain the training element mask information, and denoise the reference image vector through the initial image generation model to obtain the training denoised image vector, and perform attention weighting on the training denoised image vector based on the target training text vector to obtain the image-related vector corresponding to the target training text vector.

[0162] S1110, determine the training element related vector from the image related vector through the initial image generation model, obtain the white pixel value, and perform ratio calculation based on the training element mask information and the white pixel value to obtain the weighted weight, and perform product operation based on the weighted weight and the training element related vector to obtain the weighted vector; calculate the sum of the weighted vector and the training element related vector to obtain the weighted related vector, and perform averaging based on the weighted related vector to obtain the training element adjustment vector; replace the training element related vector in the image related vector with the training element adjustment vector to obtain the image adjustment vector.

[0163] S1112, using the initial image generation model to take the image adjustment vector as the target training text vector, and returning to calculate the correlation between the reference image vector and the target training text vector, obtaining the image correlation vector corresponding to the target training text vector, and adjusting the image correlation vector based on the training element mask information, and the step of obtaining the image adjustment vector is iteratively executed until the iteration completion condition is met, and the output target training image vector is obtained.

[0164] S1114, decoding the target training image vector through the initial image generation model to obtain a training generated image. The initial image generation model is updated based on the training generated image and the training label image until the training completion condition is met, and an image generation model is obtained, which is used to generate a corresponding target element generation image according to the input description text and the corresponding element image.

[0165] In the above embodiment, the semantic information of the training element text is enhanced by using the semantic information of the training element image, thereby improving the connection between the target element text and the generated object during the image training and generation process, and strengthening the representation ability of the target element text concept, thereby improving the generation restoration degree of the target element in the final generated image. Then, the image correlation vector is adjusted by the training element mask information, thereby strengthening the weight convergence speed of the text semantic vector when calculating the correlation degree, thereby improving the overall convergence speed of the model, and strengthening the connection between the target element text semantic concept and the shape of the target element, so that when the image is finally generated, the generation of the target element in the image is more vivid and restored.

[0166] In a specific embodiment, the image generation method is applied to a content generation platform. Specifically, a user can generate content on the content generation platform, wherein images can be generated, for example, game images, character images, animal images, landscape images, animation images, etc. When the user generates an animation image, the description text of the generated animation image and the target animation element image included in the animation image to be generated are first input into the content generation platform. The content generation platform receives the description text of the animation image to be generated and the target animation element image uploaded by the user, and inputs the description text and the target animation element image into a trained animation style image generation model for image generation, that is, semantic representation is performed on the description text and the target animation element image respectively to obtain the corresponding text vector and the target animation element image vector; based on the target animation element image vector, the target animation element text vector in the text vector is enhanced to obtain the target animation text vector. Then, a reference image vector is obtained, and the correlation between the reference image vector and the target animation text vector is calculated to obtain the animation image correlation vector corresponding to the target animation text vector, and image generation is performed based on the animation image correlation vector to obtain the output generated image including the target animation element. The generated image is then returned to the user's terminal for display, which can improve the accuracy of the target animation element generation in the generated image, that is, the restoration degree. The image generation method can also be applied to game scenes, and a game object image containing prop elements with high restoration degree can be generated for the prop elements of the game object, and the game object image can be used as the game original painting, thereby improving the efficiency of game production, and the target prop element image can also be generated, and prop element pictures and original paintings of different sizes can be generated to facilitate subsequent use, etc.

[0167] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0168] Based on the same inventive concept, the embodiment of the present application also provides an image generation model training device for implementing the above-mentioned image generation model training method and an image generation device of the image generation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image generation device embodiments provided below can refer to the above-mentioned limitations on the image generation model training method and the image generation method, and will not be repeated here.

[0169] In one embodiment, Fig.12 As shown, an image generation model training device 1200 is provided, comprising: a training acquisition module 1202, a semantic representation module 1204, an enhancement module 1206, a mask adjustment module 1208 and a model iteration module 1210, wherein:

[0170] A training acquisition module 1202 is used to acquire training description text and training element images corresponding to the training label images, wherein the training description text includes training element text corresponding to the training element images;

[0171] The semantic representation module 1204 is used to input the training description text and the training element image into the initial image generation model, and perform semantic representation on the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors;

[0172] An enhancement module 1206 is used to enhance the training element text vector in the training text vector based on the training element image vector through the initial image generation model to obtain a target training text vector;

[0173] The mask adjustment module 1208 is used to obtain the training element mask information corresponding to the training label image, and calculate the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain the image adjustment vector, and generate the image based on the image adjustment vector to obtain the training generated image;

[0174] The model iteration module 1210 is used to update the initial image generation model based on the training generated image and the training label image until the training completion condition is met, and the image generation model is obtained. The image generation model is used to generate the corresponding target element generation image according to the input description text and the corresponding element image.

[0175] In one embodiment, the enhancement module is also used to fuse the training element image vector and the training element text vector to obtain a training fusion vector, and merge the training fusion vector with the training element text vector to obtain an enhanced element text vector; replace the training element text vector in the training text vector with the enhanced element text vector to obtain a target training text vector.

[0176] In one embodiment, the enhancement module is also used to splice the training element image vector and the training element text vector to obtain a training splicing vector, and perform feature interaction of attention based on the training splicing vector to obtain a training fusion vector; obtain an enhancement weight, weight the training fusion vector based on the enhancement weight to obtain an enhanced fusion vector, and calculate the sum of the enhanced fusion vector and the training element text vector to obtain an enhanced element text vector.

[0177] In one embodiment, the mask adjustment module is also used to binarize the training label image based on the training element image to obtain the training element mask information; denoise the reference image vector through the initial image generation model to obtain the training denoised image vector, and perform attention weighting on the training denoised image vector based on the target training text vector to obtain the image-related vector corresponding to the target training text vector.

[0178] In one embodiment, the mask adjustment module obtains a training element related vector from the image related vector, and weights the training element related vector based on the training element mask information to obtain a training element adjustment vector; and replaces the training element related vector in the image related vector with the training element adjustment vector to obtain an image adjustment vector.

[0179] In one embodiment, the mask adjustment module obtains the white pixel value, and performs a ratio calculation based on the training element mask information and the white pixel value to obtain a weighted weight, and performs a product operation based on the weighted weight and the training element related vector to obtain a weighted vector; calculates the sum of the weighted vector and the training element related vector to obtain a weighted related vector, and performs an average based on the weighted related vector to obtain a training element adjustment vector.

[0180] In one embodiment, the mask adjustment module is also used to use the image adjustment vector as the target training text vector, and return to calculate the correlation between the reference image vector and the target training text vector, obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information, and the step of obtaining the image adjustment vector is iteratively executed until the iteration completion condition is met, and the output target training image vector is obtained; decoding is performed based on the target training image vector to obtain the training generated image.

[0181] In one embodiment, the training acquisition module is also used to acquire a target image of a target style type and use the target image as a training label image; perform image content recognition based on the target image to obtain training description text, and perform target element region segmentation based on the target image to obtain a training element image.

[0182] In one embodiment, the image generation model training device 1200 further includes:

[0183] The pre-training module is used to obtain the pre-trained description text corresponding to the pre-trained label image; input the pre-trained description text into the pre-trained image generation model, perform semantic representation on the pre-trained description text based on the pre-trained image generation model, obtain the pre-trained text vector corresponding to the pre-trained description text, calculate the correlation between the reference image vector and the pre-trained text vector, obtain the pre-trained image correlation vector, and generate an image based on the pre-trained image correlation vector to obtain a pre-trained generated image; update the pre-trained image generation model based on the pre-trained generated image and the pre-trained label image until the pre-training completion condition is met, and obtain the initial image generation model.

[0184] In one embodiment, the pre-training module is further used to obtain a target image corresponding to a target style type; use the target image as a pre-training label image, and perform image content recognition based on the target image to obtain a pre-training description text.

[0185] In one embodiment, Fig.13 As shown, an image generation device 1300 is provided, comprising: a current acquisition module 1302, a current characterization module 1304, a current enhancement module 1306 and an image generation module 1308, wherein:

[0186] The current acquisition module 1302 is used to acquire the current description text and the corresponding current element image, wherein the current description text includes the current element text corresponding to the current element image;

[0187] The current representation module 1304 is used to perform semantic representation on the current description text and the current element image respectively to obtain the corresponding current text vector and current element image vector;

[0188] A current enhancement module 1306, configured to enhance the current element text vector in the current text vector based on the current element image vector to obtain a current target text vector;

[0189] The image generation module 1308 is used to obtain a reference image vector, calculate the correlation between the reference image vector and the current target text vector, obtain a current image correlation vector corresponding to the current target text vector, and generate an image based on the current image correlation vector to obtain a target generated image.

[0190] In one embodiment, the image generating device further includes:

[0191] The model generation module is used to input the current description text and the current element image into the image generation model. The image generation model is obtained by updating the initial image generation model based on the training generated image and the training label image. The training generated image is obtained by adjusting the image correlation vector by the training element mask information corresponding to the training label image to obtain the image adjustment vector, and the image is generated based on the image adjustment vector. The image correlation vector is obtained by calculating the correlation between the reference image vector and the target training text vector. The target training text vector is obtained by enhancing the training element text vector in the training text vector using the training element image vector; the current description text and the current element image are semantically represented respectively by the image generation model to obtain the corresponding current text vector and current element image vector; the current element text vector in the current text vector is enhanced based on the current element image vector by the image generation model to obtain the current target text vector; the correlation between the reference image vector and the current target text vector is calculated by the image generation model to obtain the current image correlation vector corresponding to the current target text vector, and the image is generated based on the current image correlation vector to obtain the target generated image.

[0192] In one embodiment, the current enhancement module is also used to fuse the current element image vector and the current element text vector to obtain a current fused vector, and merge the current fused vector with the current element text vector to obtain a current enhanced element text vector; replace the current element text vector in the current text vector with the current enhanced element text vector to obtain a current target text vector.

[0193] In one embodiment, the current enhancement module is also used to splice the current element image vector and the current element text vector to obtain a current splicing vector, and perform self-attention feature interaction based on the current splicing vector to obtain a current fusion vector; obtain an enhancement weight, weight the current fusion vector based on the enhancement weight to obtain a current enhanced fusion vector, and calculate the sum of the current enhanced fusion vector and the current element text vector to obtain a current enhanced element text vector.

[0194] In one embodiment, the current element image includes at least two, and the current element image vector includes current element image vectors corresponding to at least two current element images respectively;

[0195] The current enhancement module is also used to fuse the current element text vector with the current element image vectors corresponding to at least two current element images to obtain current fused vectors corresponding to at least two current element images; perform average calculation based on the current fused vectors corresponding to at least two current element images to obtain a current average vector, and merge the current average vector with the current element text vector to obtain a current enhanced element text vector corresponding to at least two current element images.

[0196] The above-mentioned image generation model training device and each module in the image generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0197] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training label images, corresponding training description texts, training element images, and target images of various styles and types, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation model training method or an image generation method is implemented.

[0198] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image generation model training method or an image generation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0199] Those skilled in the art will understand that Fig.14 as well as Fig.15The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0201] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0202] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training an image generation model, characterized in that: The method comprises: Acquire a training description text and a training element image corresponding to the training label image, wherein the training description text includes a training element text corresponding to the training element image; Inputting the training description text and the training element image into an initial image generation model, and performing semantic representation on the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors; Enhance the training element text vector in the training text vector based on the training element image vector by the initial image generation model to obtain a target training text vector; Acquire training element mask information corresponding to the training label image, and calculate the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain an image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain an image adjustment vector, and generate an image based on the image adjustment vector to obtain a training generated image; The initial image generation model is updated based on the training generated image and the training label image until the training completion condition is met, thereby obtaining an image generation model, wherein the image generation model is used to generate a corresponding target element generation image according to the input description text and the corresponding element image.

2. The method according to claim 1, characterized in that The step of enhancing the training element text vector in the training text vector based on the training element image vector to obtain a target training text vector includes: Fusing the training element image vector and the training element text vector to obtain a training fusion vector, and merging the training fusion vector with the training element text vector to obtain an enhanced element text vector; The training element text vector in the training text vector is replaced with the enhanced element text vector to obtain the target training text vector.

3. The method according to claim 2, characterized in that The step of fusing the training element image vector and the training element text vector to obtain a training fusion vector, and merging the training fusion vector with the training element text vector to obtain an enhanced element text vector includes: Splicing the training element image vector and the training element text vector to obtain a training splicing vector, and performing attention feature interaction based on the training splicing vector to obtain the training fusion vector; An enhancement weight is obtained, and the training fusion vector is weighted based on the enhancement weight to obtain an enhanced fusion vector, and a sum of the enhanced fusion vector and the training element text vector is calculated to obtain the enhanced element text vector.

4. The method according to claim 1, characterized in that: The step of acquiring the training element mask information corresponding to the training label image, and calculating the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain the image correlation vector corresponding to the target training text vector includes: Binarizing the training label image based on the training element image to obtain the training element mask information; The reference image vector is denoised by the initial image generation model to obtain a training denoised image vector, and the training denoised image vector is attention-weighted based on the target training text vector to obtain an image-related vector corresponding to the target training text vector.

5. The method according to claim 1, characterized in that The step of adjusting the image-related vector based on the training element mask information to obtain an image adjustment vector includes: Acquire a training element related vector from the image related vector, and weight the training element related vector based on the training element mask information to obtain a training element adjustment vector; The training element related vector in the image related vector is replaced with the training element adjustment vector to obtain the image adjustment vector.

6. The method according to claim 5, characterized in that The step of weighting the training element related vector based on the training element mask information to obtain the training element adjustment vector includes: Acquire a white pixel value, perform a ratio calculation based on the mask information of the training element and the white pixel value to obtain a weighted weight, and perform a product operation based on the weighted weight and the training element related vector to obtain a weighted vector; The sum of the weighted vector and the training element related vector is calculated to obtain a weighted related vector, and average division is performed based on the weighted related vector to obtain the training element adjustment vector.

7. The method according to claim 1, characterized in that The step of generating an image based on the image adjustment vector to obtain a training generated image includes: The image adjustment vector is used as the target training text vector, and the correlation between the reference image vector and the target training text vector is calculated to obtain the image correlation vector corresponding to the target training text vector, and the image correlation vector is adjusted based on the training element mask information. The step of obtaining the image adjustment vector is iteratively executed until the iteration completion condition is met, and the output target training image vector is obtained; Decoding is performed based on the target training image vector to obtain the training generated image.

8. The method according to claim 1, characterized in that The step of obtaining the training description text and the training element image corresponding to the training label image includes: Obtain a target image of a target style type, and use the target image as the training label image; Image content recognition is performed based on the target image to obtain the training description text, and target element region segmentation is performed based on the target image to obtain the training element image.

9. The method according to claim 1, characterized in that: Before obtaining the training description text and the training element image corresponding to the training label image, the method further includes: Get the pre-trained description text corresponding to the pre-trained label image; The pre-trained description text is input into a pre-trained image generation model, and semantic representation is performed on the pre-trained description text based on the pre-trained image generation model to obtain a pre-trained text vector corresponding to the pre-trained description text, and the correlation between the reference image vector and the pre-trained text vector is calculated to obtain a pre-trained image correlation vector, and image generation is performed based on the pre-trained image correlation vector to obtain a pre-trained generated image; The pre-trained image generation model is updated based on the pre-trained generated image and the pre-trained label image until the pre-training completion condition is met, thereby obtaining the initial image generation model.

10. The method according to claim 9, characterized in that The obtaining of the pre-trained description text corresponding to the pre-trained label image includes: Get the target image corresponding to the target style type; The target image is used as the pre-trained label image, and image content recognition is performed based on the target image to obtain the pre-trained description text.

11. An image generation method, characterized in that: The method comprises: Obtain a current description text and a corresponding current element image, wherein the current description text includes a current element text corresponding to the current element image, and input the current description text and the current element image into an image generation model, wherein the image generation model is obtained by updating an initial image generation model based on a training generation image and a training label image, wherein the training generation image is obtained by adjusting an image correlation vector through training element mask information corresponding to the training label image to obtain an image adjustment vector, and performing image generation based on the image adjustment vector, wherein the image correlation vector is obtained by calculating a correlation degree between a reference image vector and a target training text vector, and wherein the target training text vector is obtained by enhancing a training element text vector in the training text vector using a training element image vector; Semantically representing the current description text and the current element image respectively through the image generation model to obtain a corresponding current text vector and a current element image vector; Enhance the current element text vector in the current text vector based on the current element image vector by using the image generation model to obtain a current target text vector; A reference image vector is obtained through the image generation model, and the correlation between the reference image vector and the current target text vector is calculated to obtain a current image correlation vector corresponding to the current target text vector, and image generation is performed based on the current image correlation vector to obtain a target generated image.

12. The method according to claim 11, characterized in that The step of enhancing the current element text vector in the current text vector based on the current element image vector to obtain the current target text vector includes: Fusing the current element image vector and the current element text vector to obtain a current fused vector, and merging the current fused vector with the current element text vector to obtain a current enhanced element text vector; The current element text vector in the current text vector is replaced with the current enhanced element text vector to obtain the current target text vector.

13. The method according to claim 12, characterized in that The step of fusing the current element image vector and the current element text vector to obtain a current fused vector, and merging the current fused vector with the current element text vector to obtain a current enhanced element text vector includes: Splicing the current element image vector and the current element text vector to obtain a current splicing vector, and performing self-attention feature interaction based on the current splicing vector to obtain the current fusion vector; An enhancement weight is obtained, and the current fusion vector is weighted based on the enhancement weight to obtain a current enhanced fusion vector, and a sum of the current enhanced fusion vector and the current element text vector is calculated to obtain the current enhanced element text vector.

14. The method according to claim 12, characterized in that The current element images include at least two, and the current element image vector includes current element image vectors corresponding to the at least two current element images respectively; The step of fusing the current element image vector and the current element text vector to obtain a current fused vector, and merging the current fused vector with the current element text vector to obtain a current enhanced element text vector includes: Fusing the current element text vector with the current element image vectors respectively corresponding to the at least two current element images to obtain current fused vectors respectively corresponding to the at least two current element images; An average calculation is performed based on the current fusion vectors respectively corresponding to the at least two current element images to obtain a current average vector, and the current average vector is merged with the current element text vector to obtain a current enhanced element text vector corresponding to the at least two current element images.

15. An image generation model training device, characterized in that: The device comprises: A training acquisition module, used to acquire training description text and training element images corresponding to the training label images, wherein the training description text includes training element text corresponding to the training element images; A semantic representation module, used for inputting the training description text and the training element image into an initial image generation model, and performing semantic representation on the training description text and the training element image based on the initial image generation model to obtain corresponding training text vectors and training element image vectors; an enhancement module, configured to enhance the training element text vector in the training text vector based on the training element image vector by using the initial image generation model to obtain a target training text vector; A mask adjustment module, used to obtain the training element mask information corresponding to the training label image, and calculate the correlation between the reference image vector and the target training text vector through the initial image generation model to obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information to obtain the image adjustment vector, and generate an image based on the image adjustment vector to obtain a training generated image; A model iteration module is used to update the initial image generation model based on the training generated image and the training label image until the training completion condition is met, thereby obtaining an image generation model, wherein the image generation model is used to generate a corresponding target element generation image based on the input description text and the corresponding element image.

16. The device according to claim 15, characterized in that The enhancement module is further used to fuse the training element image vector and the training element text vector to obtain a training fusion vector, and merge the training fusion vector with the training element text vector to obtain an enhanced element text vector; The training element text vector in the training text vector is replaced with the enhanced element text vector to obtain the target training text vector.

17. The device according to claim 16, characterized in that The enhancement module is further used to splice the training element image vector and the training element text vector to obtain a training splicing vector, and perform attention feature interaction based on the training splicing vector to obtain the training fusion vector; An enhancement weight is obtained, and the training fusion vector is weighted based on the enhancement weight to obtain an enhanced fusion vector, and a sum of the enhanced fusion vector and the training element text vector is calculated to obtain the enhanced element text vector.

18. The device according to claim 15, characterized in that The mask adjustment module is further used to binarize the training label image based on the training element image to obtain the training element mask information; The reference image vector is denoised by the initial image generation model to obtain a training denoised image vector, and the training denoised image vector is attention-weighted based on the target training text vector to obtain an image-related vector corresponding to the target training text vector.

19. The device according to claim 15, characterized in that The mask adjustment module is also used to obtain a training element related vector from the image related vector, and weight the training element related vector based on the training element mask information to obtain a training element adjustment vector; and replace the training element related vector in the image related vector with the training element adjustment vector to obtain the image adjustment vector.

20. The device according to claim 19, characterized in that The mask adjustment module is also used to obtain a white pixel value, and perform a ratio calculation based on the training element mask information and the white pixel value to obtain a weighted weight, and perform a product operation based on the weighted weight and the training element related vector to obtain a weighted vector; calculate the sum of the weighted vector and the training element related vector to obtain a weighted related vector, and perform an average based on the weighted related vector to obtain the training element adjustment vector.

21. The device according to claim 15, characterized in that The mask adjustment module is also used to use the image adjustment vector as the target training text vector, and return to calculate the correlation between the reference image vector and the target training text vector to obtain the image correlation vector corresponding to the target training text vector, and adjust the image correlation vector based on the training element mask information. The step of obtaining the image adjustment vector is iteratively executed until the iteration completion condition is met to obtain the output target training image vector; decoding is performed based on the target training image vector to obtain the training generated image.

22. The device according to claim 15, characterized in that The training acquisition module is also used to acquire a target image of a target style type and use the target image as the training label image; perform image content recognition based on the target image to obtain the training description text; and perform target element region segmentation based on the target image to obtain the training element image.

23. The device according to claim 15, characterized in that The device further comprises: A pre-training module is used to obtain a pre-trained description text corresponding to a pre-trained label image; input the pre-trained description text into a pre-trained image generation model, perform semantic representation on the pre-trained description text based on the pre-trained image generation model, obtain a pre-trained text vector corresponding to the pre-trained description text, calculate the correlation between the reference image vector and the pre-trained text vector, obtain a pre-trained image correlation vector, and perform image generation based on the pre-trained image correlation vector to obtain a pre-trained generated image; update the pre-trained image generation model based on the pre-trained generated image and the pre-trained label image until the pre-training completion condition is met, thereby obtaining the initial image generation model.

24. The device according to claim 23, characterized in that The pre-training module is also used to obtain a target image corresponding to a target style type; use the target image as the pre-training label image, and perform image content recognition based on the target image to obtain the pre-training description text.

25. An image generating device, characterized in that: The device comprises: A current acquisition module is used to acquire a current description text and a corresponding current element image, wherein the current description text includes a current element text corresponding to the current element image, and the current description text and the current element image are input into an image generation model, wherein the image generation model is obtained by updating an initial image generation model based on a training generation image and a training label image, wherein the training generation image is obtained by adjusting an image correlation vector through training element mask information corresponding to the training label image to obtain an image adjustment vector, and performing image generation based on the image adjustment vector, wherein the image correlation vector is obtained by calculating a correlation degree between a reference image vector and a target training text vector, and wherein the target training text vector is obtained by enhancing a training element text vector in the training text vector using a training element image vector; A current representation module, used to perform semantic representation on the current description text and the current element image respectively through the image generation model to obtain a corresponding current text vector and a current element image vector; A current enhancement module, used to enhance the current element text vector in the current text vector based on the current element image vector through the image generation model to obtain a current target text vector; An image generation module is used to obtain a reference image vector through the image generation model, and calculate the correlation between the reference image vector and the current target text vector, to obtain a current image correlation vector corresponding to the current target text vector, and to generate an image based on the current image correlation vector to obtain a target generated image.

26. The device according to claim 25, characterized in that The current enhancement module is further used to fuse the current element image vector and the current element text vector to obtain a current fused vector, and merge the current fused vector with the current element text vector to obtain a current enhanced element text vector; The current element text vector in the current text vector is replaced with the current enhanced element text vector to obtain the current target text vector.

27. The device according to claim 26, characterized in that The current enhancement module is further used to splice the current element image vector and the current element text vector to obtain a current splicing vector, and perform self-attention feature interaction based on the current splicing vector to obtain the current fusion vector; An enhancement weight is obtained, and the current fusion vector is weighted based on the enhancement weight to obtain a current enhanced fusion vector, and a sum of the current enhanced fusion vector and the current element text vector is calculated to obtain the current enhanced element text vector.

28. The device according to claim 26, characterized in that The current element images include at least two, and the current element image vector includes current element image vectors corresponding to the at least two current element images respectively; The current enhancement module is further used to fuse the current element text vector with the current element image vectors respectively corresponding to the at least two current element images to obtain current fused vectors respectively corresponding to the at least two current element images; An average calculation is performed based on the current fusion vectors respectively corresponding to the at least two current element images to obtain a current average vector, and the current average vector is merged with the current element text vector to obtain a current enhanced element text vector corresponding to the at least two current element images.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images