Image generation model training method and related apparatus

Abstract

The application relates to the computer technical field and provides an image generation model training method and related devices to improve image generation quality, wherein the method comprises the following steps: performing label prediction of two kinds of languages for each sample image, determining the accuracy evaluation value of each reference word based on the prediction probability of each sample image for each reference word and the prediction threshold of each reference word, obtaining at least one group of keywords corresponding to each sample image based on the prediction probability, the prediction threshold and the accuracy evaluation value, then, constructing a set of image-text pairs based on each sample image and each group of keywords corresponding to the sample image, and training an image generation model for the first language to obtain a pre-trained image generation model. In this way, the accuracy of the generated image in the second language scene is improved by describing the sample image in two languages.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and provides an image generation model training method and related apparatus. BACKGROUND

[0002] With the continuous development of computer technology, text-to-image technology can be used to quickly generate images. Text-to-image technology refers to a process of converting text into images using artificial intelligence technology, which can generate images that meet the description of the given text.

[0003] In related technologies, an image generation model for text-to-image is usually trained based on English corpus. Therefore, in the model application process, for a given Chinese text, the Chinese text is translated into English text, and then the trained image generation model is used to obtain the corresponding image based on the translated English text.

[0004] However, there are great semantic differences between texts of different languages. Therefore, the image generation model trained based on English corpus cannot meet the image generation requirements of Chinese text, and the image generation effect is not good. For example, for a straw house, a Mongolian yurt style barn on a large grassland in Europe and America will be generated instead of a Chinese grass shed. SUMMARY

[0005] The embodiments of the present application provide an image generation model training method and related apparatus to improve the accuracy of generated images in the text-to-image scenario.

[0006] In a first aspect, the embodiments of the present application provide an image generation model training method, comprising:

[0007] Based on the obtained sample images, the following operations are performed for the first language and the second language respectively:

[0008] Based on the reference words set for the current language, the prediction probability of each sample image for each reference word is obtained, and the accuracy evaluation value of each reference word is determined in combination with the prediction threshold of each reference word;

[0009] Based on the obtained prediction probability, the at least one group of keywords corresponding to each sample image is obtained in combination with the prediction threshold and the accuracy evaluation value of each reference word;

[0010] Based on the sample images and the groups of keywords corresponding to each sample image, a set of image-text pairs is constructed, each image-text pair containing a sample image and a group of keywords corresponding to the sample image, and in each image-text pair, the keywords obtained for the second language are keywords obtained by language conversion from the first language;

[0011] The image generation model pre-trained in the first language is iteratively trained based on the set of image-text pairs to obtain a target image generation model.

[0012] In a second aspect, an embodiment of the present application provides an image generation model training device, comprising:

[0013] The label prediction unit is configured to, based on the obtained sample images, perform the following operations for the first language and the second language respectively: obtaining a prediction probability of each sample image for each reference word based on the reference words set for the current language, and determining an accuracy evaluation value of each reference word in combination with a prediction threshold of each reference word; and obtaining at least one group of keywords corresponding to each sample image in combination with the prediction threshold and the accuracy evaluation value of each reference word based on the obtained prediction probability.

[0014] The image-text pair construction unit is configured to construct a set of image-text pairs based on the sample images and the groups of keywords corresponding to the sample images respectively, each image-text pair comprising: a sample image and a group of keywords corresponding to the sample image, and the keywords obtained for the second language in each image-text pair are keywords obtained after language conversion from the first language.

[0015] The training unit is configured to iteratively train an image generation model pre-trained in the first language based on the set of image-text pairs to obtain a target image generation model.

[0016] As a possible implementation manner, the label prediction unit is further configured to:

[0017] Obtain each candidate image in a target application scenario, and obtain a scene category corresponding to each candidate image by using a scene category classification model.

[0018] When at least one candidate image meeting a set erasing condition is filtered out from the candidate images based on the scene categories corresponding to the candidate images respectively, perform object erasing on the at least one candidate image respectively to obtain corresponding erasing images.

[0019] Use the obtained at least one erasing image and other candidate images except the at least one candidate image in the candidate images as sample images.

[0020] As a possible implementation manner, when the at least one candidate image meeting the set erasing condition is filtered out from the candidate images based on the scene categories corresponding to the candidate images respectively, the label prediction unit is specifically configured to:

[0021] perform object detection on each of the candidate images that are filtered out based on the respective scene category corresponding to each of the candidate images.

[0022] based on the detection result, at least one of the candidate images that contains the target object is determined as at least one of the candidate images that meets the set erasing condition.

[0023] As a possible implementation, when the accuracy evaluation value of each reference word is determined based on the prediction probability of each sample image for each reference word and in combination with the prediction threshold of each reference word, the label prediction unit is specifically configured to:

[0024] For each of the reference words, the following operations are performed respectively:

[0025] based on the prediction probability of each sample image for one reference word, sample images with prediction probability greater than the prediction threshold of the one reference word are filtered out from the sample images;

[0026] According to the set sample extraction quantity, each labeled sample is extracted from the filtered sample images, and the accuracy evaluation value of the one reference word is determined based on the corresponding label result of each labeled sample, wherein each label result is used to represent whether the corresponding labeled sample has the one reference word.

[0027] As a possible implementation, the label prediction unit is further configured to:

[0028] obtain each label sample labeled for one reference word, and perform reference word prediction on the label samples to obtain the prediction probability of each label sample for the one reference word;

[0029] based on the prediction probability of each label sample for the one reference word and in combination with the corresponding label result of each labeled sample, the accuracy evaluation value of the one reference word under each candidate threshold is determined;

[0030] based on the accuracy evaluation value under each candidate threshold, the prediction threshold of the one reference word is determined from the candidate thresholds.

[0031] As a possible implementation, when the at least one group of keywords corresponding to each sample image is obtained based on the obtained prediction probability, in combination with the prediction threshold and the accuracy evaluation value of each reference word, the label prediction unit is specifically configured to:

[0032] For each of the sample images, the following operations are performed respectively:

[0033] when there is at least one reference word in the reference words for which the corresponding predicted probability is not less than the corresponding predicted threshold, taking the at least one reference word as at least one non-noise word of the current sample image;

[0034] when there is at least one other reference word in the other reference words other than the at least one reference word for which the corresponding accuracy evaluation value is less than the evaluation value threshold, taking the at least one other reference word as at least one noise word of the current sample image;

[0035] combining the at least one non-noise word and the at least one noise word to obtain at least one set of keywords corresponding to the current sample image.

[0036] As a possible implementation, when the at least one target word and the at least one noise word are combined to obtain at least one set of keywords corresponding to the current sample image, the label prediction unit is specifically configured to:

[0037] combining the at least one non-noise word and the at least one noise word to obtain an initial word set, and taking the initial word set as a set of keywords;

[0038] deleting the at least one noise word from the initial word set respectively to obtain at least one set of enhanced words, and taking the at least one set of enhanced words as at least one set of keywords.

[0039] As a possible implementation, the training unit is further configured to, in each iteration process, perform the following operations on each set of keywords in the selected sample image-text pairs respectively:

[0040] if a set of keywords contains noise words, directly obtaining a model loss corresponding to the set of keywords based on a predicted image and a sample image corresponding to the set of keywords;

[0041] if a set of keywords does not contain noise words, performing image erasing processing on a corresponding sample image based on a deleted noise word in the set of keywords to obtain a processed sample image, and obtaining a model loss corresponding to the set of keywords based on a predicted image corresponding to the set of keywords and the processed sample image.

[0042] As a possible implementation, when the image erasing processing on the corresponding sample image based on the deleted noise word in the set of keywords is performed to obtain a processed sample image, the training unit is specifically configured to:

[0043] determining an image erasing position of the sample image based on a heat map position of the deleted noise word in the set of keywords in the corresponding sample image;

[0044] based on the image erasing position, performing image erasing processing on the sample image to obtain a processed sample image.

[0045] As a possible implementation manner, the label prediction unit is further configured to:

[0046] when it is determined that there is at least one similar word group meeting the similarity condition based on the similarity between each reference word set for the first language and each reference word set for the second language, merging two reference words respectively included in the at least one similar word group to obtain a corresponding merged word;

[0047] based on the obtained at least one merged word, obtaining at least one group of keywords corresponding to each sample image.

[0048] As a possible implementation manner, the image generation model comprises: a noise adding model, a noise reducing model, a text encoding sub-model, an image encoding sub-model, and an image decoding sub-model; and the training unit is configured to, in each iteration process, perform the following operations for each group of keywords in each selected sample image-text pair:

[0049] inputting a random image into the image encoding sub-model to obtain image encoding, and inputting the image encoding into the noise adding model to obtain noise-added image encoding;

[0050] inputting a group of keywords into the text encoding sub-model to obtain text encoding, and inputting the text encoding and the noise-added image encoding into the noise reducing model to obtain image encoding of a predicted image corresponding to the group of keywords;

[0051] inputting the image encoding of the predicted image into the image decoding sub-model to obtain the predicted image corresponding to the group of keywords.

[0052] As a possible implementation manner, the training unit is further configured to:

[0053] based on the model loss, adjusting parameters of each model parameter included in the noise reducing model.

[0054] As a possible implementation manner, the training unit is further configured to:

[0055] obtaining at least one keyword included in the text based on the text input by the target object;

[0056] inputting the at least one keyword into the target image generation model to obtain at least one candidate generated image;

[0057] Determine the target generated image based on the aesthetic evaluation information corresponding to each of the at least one candidate generated image.

[0058] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the computer program causes the processor to perform the steps of the above method.

[0059] In a fourth aspect, a computer-readable storage medium is provided, which includes a computer program. When the computer program is run on an electronic device, the computer program is configured to cause the electronic device to perform the steps of the above method.

[0060] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of the above method.

[0061] In the embodiments of the present application, based on the obtained sample images, the following operations are performed for the first language and the second language respectively: based on the reference words set for the current language, the prediction probabilities of each sample image for each reference word are obtained, and combined with the prediction threshold of each reference word, the accuracy evaluation value of each reference word is determined; based on the obtained prediction probabilities, combined with the prediction threshold and the accuracy evaluation value of each reference word, at least one group of keywords corresponding to each sample image is obtained.

[0062] Based on each sample image and its corresponding each group of keywords, a set of image-text pairs is constructed, each image-text pair contains one sample image and its corresponding one group of keywords, and in each image-text pair, the keywords obtained for the second language are the keywords obtained by converting the first language.

[0063] Based on the set of image-text pairs, the image generation model pre-trained in the first language is iteratively trained to obtain a target image generation model.

[0064] In this way, by using reference words in two languages (such as Chinese and English) to jointly describe sample images, the accuracy of image-text data in another language is improved for the image generation model pre-trained in one language, and the problem of poor model performance due to the lack of description in another language is avoided. In addition, since there may be a domain difference between the training samples (such as short videos) used for pre-training and the training samples (such as films and television dramas) used in the training process, determining the labels possessed by the training images based on the prediction threshold and the accuracy evaluation value of each reference word can further improve the image generation quality of the model.

[0065] Other features and advantages of the present application will be set forth in the following specification, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0066] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0067] Figure 1 An application scenario provided in an embodiment of the present application;

[0068] Figure 2 A structure of an image generation model provided in an embodiment of the present application;

[0069] Figure 3 A structure of another image generation model provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of an image enhancement process provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a scene category provided in an embodiment of the present application;

[0072] Figure 6A A schematic diagram of a candidate image provided in an embodiment of the present application;

[0073] Figure 6B A logic schematic diagram of object erasing using an object erasing model provided in an embodiment of the present application;

[0074] Figure 7 A logic schematic diagram of an object erasing process provided in an embodiment of the present application;

[0075] Figure 8 A logic schematic diagram of a prediction threshold selection process provided in an embodiment of the present application;

[0076] Figure 9A A logic schematic diagram of a non-noise word determination process provided in an embodiment of the present application;

[0077] Figure 9B A logic schematic diagram of a noise word determination process provided in an embodiment of the present application;

[0078] Figure 10A schematic diagram of a plurality of groups of keywords provided in an embodiment of the present application;

[0079] Figure 11 A flowchart of an image generation model training method provided in an embodiment of the present application;

[0080] Figure 12 A logic diagram of a loss calculation manner provided in an embodiment of the present application;

[0081] Figure 13 A schematic diagram of a heat map provided in an embodiment of the present application;

[0082] Figure 14 A logic diagram of another loss calculation manner provided in an embodiment of the present application;

[0083] Figure 15 A schematic diagram of a model loss calculation process provided in an embodiment of the present application;

[0084] Figure 16 An interaction schematic diagram of a server and a client provided in an embodiment of the present application;

[0085] Figure 17 A logic diagram of an image generation model training method provided in an embodiment of the present application;

[0086] Figure 18 A structural schematic diagram of an image generation model training apparatus provided in an embodiment of the present application;

[0087] Figure 19 A structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments described in the present application document, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the technical solutions of the present application.

[0089] The embodiments of the present application relate to artificial intelligence and machine learning technology, and are mainly designed based on machine learning in artificial intelligence.

[0090] Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computer or machine controlled by digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0091] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation and several other major directions.

[0092] Computer Vision (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, locate and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, Optical Character Recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0093] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Letting computers hear, see, speak and feel is the development direction of future human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0094] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, i.e., the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0095] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0096] Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, etc. Artificial neural network (ANN) is an abstraction of human brain neuron network from the perspective of information processing, which establishes a certain simple model and forms different networks according to different connection modes. Neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. Each node represents a specific output function, called activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called weight, which is equivalent to the memory of artificial neural network. The output of the network depends on the connection mode, weight value and activation function, while the network itself is usually an approximation of a certain algorithm or function in nature, or a representation of a logical strategy.

[0097] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, smart customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0098] The scheme provided in the embodiments of the present application mainly relates to machine learning technology of artificial intelligence, and particularly relates to two parts of model training and model application. In the model training stage, label prediction of two languages (such as English and Chinese) is performed on each sample image obtained, and a label group corresponding to each sample image is obtained, and then a set of image-text pairs is constructed based on each sample image and the label group corresponding to each sample image, and then the image generation model pre-trained in the first language is iteratively trained based on the set of image-text pairs, and a target image generation model is obtained. In the model application stage, the target image generation model trained is used to perform text-to-image processing on the input text, and a target generated image is obtained. The image generation model in the embodiments of the present application can be online training or offline training, which is not specifically limited here, and offline training is taken as an example for illustration in this paper.

[0099] The design idea of the embodiments of the present application is briefly introduced as follows.

[0100] With the continuous development of computer technology, images can be quickly generated by using text-to-image technology. The text-to-image technology refers to a process of converting text into images by using artificial intelligence technology, which can generate images conforming to the description of the given text.

[0101] In the related art, the image generation model for text-to-image is usually trained based on English corpus, and therefore, in the model application process, the Chinese text is translated into English text based on the given Chinese text, and then the image generation model trained is used to obtain the corresponding image based on the translated English text.

[0102] However, there is a big semantic difference between texts of different languages, and therefore, the image generation model trained based on English corpus is difficult to meet the image generation requirement of Chinese text, and the image generation effect is not good. For example, a yurt style barn will be generated on a large grassland in Europe and America for a straw house, instead of a Chinese grass shed.

[0103] In the embodiments of the present application, based on each sample image obtained, the following operations are performed for the first language and the second language: based on each reference word set for the current language, the prediction probability of each sample image for each reference word is obtained, and combined with the prediction threshold of each reference word, the accuracy evaluation value of each reference word is determined; based on the obtained prediction probability, combined with the prediction threshold and the accuracy evaluation value of each reference word, at least one group of keywords corresponding to each sample image is obtained.

[0104] Based on each sample image and each corresponding group of keywords, a set of image-text pairs is constructed, each image-text pair including a sample image and a corresponding group of keywords, and in each image-text pair, the keywords obtained in the second language are keywords obtained after language conversion from the first language;

[0105] Based on the set of image-text pairs, the image generation model pre-trained in the first language is iteratively trained to obtain a target image generation model.

[0106] In this way, by jointly describing sample images using reference words in two languages (such as Chinese and English), the accuracy of image-text data in one language is improved by pre-training an image generation model in the other language, and the problem of poor model performance due to the lack of description in the other language is avoided. In addition, since there may be a domain difference between the training samples (such as short videos) used for pre-training and the training samples (such as movies and TV series) used in the training process, determining the labels possessed by the training images based on the respective prediction thresholds and accuracy evaluation values of each reference word can further improve the image generation quality of the model.

[0107] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios described below are only used to illustrate the present application and are not limiting. In the specific implementation process, the technical solutions provided by the present application can be flexibly applied according to actual needs.

[0108] Referring to Figure 1 As shown in the figure, it is an application scenario diagram provided by an embodiment of the present application, and in this scenario, a terminal device 101 and a server 102 can be included.

[0109] The terminal device 101 can be a device owned by the text-to-image object itself, such as a mobile phone, a tablet computer (PAD), a notebook computer, a desktop computer, a smart television, a smart vehicle device, and a smart wearable device, etc. The terminal device 101 can be installed with a text-to-image application. The application involved in the present application can be a software client, a web page, a small program, etc. The server is a background server corresponding to the software or web page, small program, etc. The specific type of the client is not limited.

[0110] The server 102 can be a background server corresponding to the installed text-to-image application on the terminal device 101, which can provide a background service function of the electronic resource transfer system, for example, implement the steps of the image generation model training method and the image generation method provided in the embodiments of the present application. The server 102 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services, but not limited to.

[0111] The terminal device 101 and the server 102 can be directly or indirectly connected through one or more networks. The network can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network, or a wireless fidelity (WIFI) network, of course, it can also be other possible networks, which are not limited.

[0112] It should be noted that in the embodiments of the present application, the number of terminal devices 101 can be one or more, and similarly, the number of servers 102 can also be one or more, that is, the number of terminal devices 101 or servers 102 is not limited.

[0113] In a possible application scenario, the related data (such as feature vectors, etc.) and model parameters involved in the embodiments of the present application can be stored using cloud storage technology. Cloud storage is a new concept extending and developing from the concept of cloud computing. Distributed cloud storage system refers to a storage system that cooperatively works together by collecting networked various types of storage devices (or storage nodes) through cluster application, grid technology, and distributed storage file system, etc. Application software or application interface, and providing data storage and business access functions to the outside.

[0114] In a possible application scenario, in order to reduce the communication delay of retrieval, the server 102 can be deployed in each region, or for load balancing, different servers 102 can be used to serve terminal devices 101 in different regions, for example, the terminal device 101 is located in place a, and establishes a communication connection with the server 102 serving place a, the terminal device 101 is located in place b, and establishes a communication connection with the server 102 serving place b, a plurality of servers 102 form a data sharing system, and data sharing is realized through a block chain.

[0115] For each server 102 in the data sharing system, the server 102 has a node identifier corresponding to the server 102, and each server 102 in the data sharing system can store node identifiers of other servers 102 in the data sharing system, so as to subsequently broadcast the generated block to other servers 102 in the data sharing system according to the node identifiers of the other servers 102. Each server 102 can maintain a node identifier list, and store the server 102 name and the node identifier in the node identifier list. The node identifier can be an Internet Protocol (IP) address or any other information that can be used to identify the node.

[0116] Of course, the method provided in the embodiments of the present application is not limited to the application scenarios shown in the Figure 1 application scenarios shown in the application scenarios, and the embodiments of the present application do not limit the application scenarios. For Figure 1 the functions that can be achieved by the devices in the application scenarios shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0117] The method flow provided in the embodiments of the present application (such as the training process of the image generation model, the image generation method, etc.) can be executed by the server 102 or the terminal device 101 in Figure 1 , or can be executed by the server 102 and the terminal device 101 together. In the following, the server 102 will be mainly taken as an example for introduction.

[0118] In the embodiments of the present application, the training process of the image generation model is a process of multiple loop iteration training using training samples, which mainly includes a model design stage, a data preparation stage and an iteration training stage, which will be introduced respectively in the following.

[0119] I. Model design stage

[0120] Referring to Figure 2As shown, it is a possible structure diagram of an image generation model provided in the embodiment of the present application. The image generation model comprises a noise adding model, a noise reducing model, a text encoding sub-model, an image encoding sub-model, and an image decoding sub-model. The image encoding sub-model is used to perform image encoding on the obtained random image (i.e., random seed xi), to obtain corresponding image encoding. The noise adding model is used to diffuse and add noise to the image encoding, to obtain corresponding noise-added image encoding. The text encoding sub-model is used to perform text encoding on the obtained group of keywords (also referred to as multi-label information i), to obtain corresponding text encoding. Then, the noise reducing model is used to reduce noise, according to the obtained text encoding, on the obtained noise-added image encoding, to obtain noise-reduced image encoding. After that, the image decoding sub-model is used to decode the obtained noise-reduced image encoding, to obtain a predicted image corresponding to the group of keywords, i.e., output image i. It should be noted that in the embodiment of the present application, the random image can also be referred to as a random seed.

[0121] For example, as shown in the following formula (1), the processing process of the group of keywords (multi-label information i) is as follows. Figure 3 As shown, for the random image x (also referred to as noise image x) generated by the random seed xi, the image encoding sub-model (ε) is used to obtain image encoding (Z). Then, the noise adding model is used to diffuse and add noise to Z, to project it into the latent space to obtain the latent space vector Z T (i.e., noise-added image encoding). At the same time, the group of keywords (i.e., Text) is input into the text encoding sub-model (τ) to obtain text encoding. After that, the text encoding and Z T are input into the noise reducing model. Z T is subjected to T times of noise reduction prediction under the constraint of the text encoding, to finally generate the latent space prediction vector Z', i.e., to generate the image encoding of the predicted image. Finally, the image decoding sub-model (D) is used to decode Z' to output the image i, which is the predicted image.

[0122] In some embodiments, the image encoding sub-model can use, but is not limited to, a variational autoencoder (VAE) to map the noise image x to the latent feature space, to obtain corresponding image encoding.

[0123] In some embodiments, in the noise adding model, Z T is generated by T times of diffusion process for Z T , where Z T represents the latent space value at time T. Correspondingly, in the noise reducing model, Z T is subjected to T times of noise reduction prediction by the noise reduction process, to obtain the image encoding Z' of the predicted image, which can also be referred to as Z0. Z0 represents the original image feature without noise.

[0124] In some embodiments, the noise reduction sub-model can employ, but is not limited to, a U-Net network, which is an encoding-decoding structure. In the embodiments of the present application, the U-Net network can employ, but is not limited to, an attention mechanism. Referring to Figure 3 , the U-Net network contains a cross-attention (QKV) module. Z T is reduced T times under the constraint of text encoding. Taking the first noise reduction process as an example, the text encoding is used as KV in the QKV module, and Z T is used as Q in the QKV module. The text encoding is used to constrain the noise reduction of Z T , so that the QKV module outputs Z' related to a set of key words after T times of noise reduction. The U-Net network can also contain a skip connection structure. Each down-sampling can have a skip connection cascaded with the corresponding up-sampling, so that the U-Net network fuses the features at the corresponding positions of the encoder in the channel during each up-sampling, thereby improving the detection accuracy. It should be noted that Figure 3 only one possible hierarchical relationship is shown. In actual application, the number and connection relationship of the QKV module can be designed based on actual conditions.

[0125] In some embodiments, the text encoding sub-model can employ, but is not limited to, a contrastive text-image pre-training model (CLIP).

[0126] In the embodiments of the present application, the diffusion sub-model maps the features of the noise map encoded by the VAE to the latent space representation at time T. Subsequently, the noise reduction sub-model learns to fit the noise representation, thereby reducing the noise representation based on the original image to obtain the real image representation, and then obtaining the predicted image through the decoder.

[0127] The data processing process performed by the above model will be described in detail in the subsequent process, and therefore will not be described in detail here.

[0128] II. Data preparation phase

[0129] Data collection is the most important in machine learning, and can be said to be the most important link. The data preparation phase of the embodiments of the present application mainly includes: a sample image preparation process and a label annotation process for sample images.

[0130] 1. Sample image preparation process

[0131] In some embodiments, the target application scenario can be generating a movie poster corresponding to the specified text, or generating a short video cover corresponding to the specified text, but not limited to this.

[0132] In the movie poster generation scenario, for example, video frames are extracted from each movie according to a set extraction frequency as candidate images. The set extraction frequency can be one frame per second, and the specific frame extraction method is not limited, which will not be described here.

[0133] In the following, only the generation of a movie poster corresponding to the specified text is taken as an example for description.

[0134] In some embodiments, considering that all sample images in the movie scene need to be collected to ensure the generation effect, but in actual application process, there are many long tail scenes that cannot be covered, which leads to the difficulty of the image generation model trained to be applicable to various application scenarios, and the model generalization ability is poor. Therefore, in the embodiment of the present application, referring to Figure 4 As shown in the figure, before the text-image pair set is constructed, the sample images can also be enhanced in the following way:

[0135] First, for each candidate image in the target application scenario, a scene category classification model is used to obtain the scene category corresponding to each candidate image.

[0136] Second, when at least one candidate image that meets the set erasing condition is selected from each candidate image based on the scene category corresponding to each candidate image, the object of the at least one candidate image is erased to obtain a corresponding erased image.

[0137] Finally, the at least one erased image obtained and other candidate images in the candidate images except the at least one candidate image are used as sample images.

[0138] Among them, considering that a video is composed of multiple different shots, for example, when a movie or a TV series is shot, according to different scenes, the shooter will use different shots to express, such as long shot, full shot, medium shot, close-up and close-up. Therefore, in the embodiment of the present application, the scene category includes but is not limited to long shot, full shot, medium shot, close-up and close-up. Referring to Figure 5 As shown in the figure, it is a possible schematic diagram of long shot, full shot, medium shot, close-up and close-up provided in the embodiment of the present application.

[0139] The scene category classification model can adopt a ResNet model such as ResNet101, but is not limited thereto. ResNet is a series of residual blocks. The ResNet structure is very easy to modify and expand. By adjusting the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expression capabilities without worrying too much about the "degradation" problem of the network. As an example, a ResNet101 model pre-trained based on an ImageNet data set can be used as a pre-trained model. During the training process, the deep convolution parameters of the ResNet101 model are fixed. Then, the collected scene category data in each movie and television series is used to train the classification layer of the scene category classification model to identify the above five scene categories, obtain the trained scene category classification model, and then use the trained scene category classification model to identify the scene categories of each candidate image. The model parameters of the scene category classification model can be updated by, but are not limited to, a stochastic gradient descent (SGD) gradient update method. Since the training process of the scene category classification model is consistent with that of a conventional classification model, it will not be described here.

[0140] Specifically, when at least one candidate image meeting the set erasing condition is selected from each candidate image based on the respective scene category corresponding to each candidate image, the following methods can be used, but are not limited thereto:

[0141] When a candidate image corresponding to the scene category of the target scene category is selected from each candidate image based on the respective scene category corresponding to each candidate image, the object detection model is used to detect objects in each selected candidate image. Then, based on the detection result, at least one candidate image containing a target object in each selected candidate image is selected as at least one candidate image meeting the set erasing condition.

[0142] The target scene category includes long shot, full shot, and medium shot. When the scene category of a candidate image is long shot, full shot, or medium shot, the candidate image is selected as a candidate image corresponding to the target scene category.

[0143] The object detection model can be, but is not limited to, a yolo-v5 detection model. The yolo-v5 is a deep learning-based object detection model that can quickly and efficiently identify objects in an image. For example, the yolo-v5 detection model can be trained based on a coco detection dataset. Each detection data in the coco detection dataset includes a corresponding object label, which indicates whether the corresponding detection data contains a target object.

[0144] In the embodiments of the present application, considering that the generation of a scene often requires the generation of a scene without people, and the close-up of a character in a film and television scene is often required, the part of the candidate image containing the character is taken as the target object, so as to obtain more scene data without people and solve the problem of poor model performance in the long-tail scene.

[0145] In some embodiments, referring to Figure 6A As shown, the detection result can also include the position of the target object in the at least one filtered candidate image. The position can be, but is not limited to, an object box mask (Mask). In the object erasing process, referring to Figure 6B As shown, for each of the at least one filtered candidate image, the position of the target object in the candidate image (such as a human body box) included in the detection result and the candidate image can be input into an object erasing model to obtain an erasing image corresponding to the candidate image. The object erasing model can be, but is not limited to, a stable-diffusion model. Specifically, the inpainting function in stable-diffusion is used for implementation. Since the object erasing method is not limited in the present application, it will not be described here. In the embodiments of the present application, a detection result corresponding to each candidate image can be generated, or a total detection result can be generated for each candidate image. This is not limited, and the above is only an example of generating a total detection result.

[0146] It should be noted that in the embodiments of the present application, the object erasing process can not be performed on the candidate image corresponding to the target scene category.

[0147] For example, referring to Figure 7As shown, first, the image 1, the image 2, the image 3 and the like are obtained, and the scene category classification model is used to obtain the scene category corresponding to each candidate image, wherein the scene types of the image 1, the image 2 and the image 3 are long shot, medium shot and close-up, at this time, based on the scene category corresponding to each candidate image, the candidate images corresponding to the scene category of long shot, full shot or medium shot are selected from the candidate images, and the selected candidate images include the image 1 and the image 2. Again, the selected image 1, image 2 and the like are input into the object detection model, and the object detection is performed on the selected candidate images to obtain the detection result. Assuming that the detection result indicates that the image 1 contains a human body and the image 2 does not contain a human body, the detection result also contains the object frame Mask of the human body in the image 1. Then, based on the detection result, the image 1 and the like containing the human body are selected as the candidate images meeting the set erasing condition. Then, for each candidate image containing the human body, the object frame Mask of the image 1 and the image 1 are input into the object erasing model to obtain the erasing image corresponding to the image 1. Similarly, the object erasing processing can be performed on the candidate images meeting the set erasing condition to obtain the corresponding erasing image. Finally, the image 1, the image 2, the image 3 and the like and the erasing images corresponding to the image 1 and the like meeting the set erasing condition are all used as sample images.

[0148] Obviously, through the above implementation, a batch of film and television drama frame extraction and unmanned scene images can be obtained. In this way, through the image enhancement of removing common characters to restore pure scenes, the problem that only training certain labels cannot cover more film and television drama scenes can be avoided.

[0149] 2. Label annotation process for sample images

[0150] In the implementation of the present application, the first language label prediction and the second language label prediction are performed on each obtained sample image. It should be noted that the label in this paper can also be referred to as a keyword or a reference word. The keyword and the reference word are only used to represent different states of the label. The reference word represents the candidate label, and the keyword represents the label possessed by the sample image. The first language and the second language are two different types of languages. In the following, the first language is taken as English and the second language is taken as Chinese as an example for description.

[0151] Specifically, in the embodiments of the present application, the following operations are performed for English and Chinese respectively: based on the reference words set for the current language, the prediction probabilities of each sample image for each reference word are obtained, and based on the prediction threshold of each reference word, the keyword set corresponding to each sample image is determined. Further, based on the keyword set corresponding to each sample image, at least one group of keywords corresponding to each sample is obtained, and subsequently, based on the at least one group of keywords corresponding to each sample image, a set of image-text pairs for training the image generation model is constructed.

[0152] Since the pre-trained image generation model is trained based on English corpus, it is more relevant to the English label system, therefore, it needs to be transferred to Chinese information, so the Chinese label system is also introduced. Since the Chinese label prediction process is the same as the English label prediction process, only the Chinese label prediction process is described below.

[0153] For example, the reference words set for the Chinese language include but are not limited to environment description words such as high-burning, romantic, small lake, small environment, etc. Through the Chinese label system, the subsequent image generation model can establish a connection between such environment description and related images.

[0154] As a possible implementation manner, based on the reference words set for the Chinese language, for each sample image in the sample images, the sample image is input into the Chinese label model to obtain the prediction probability of the sample image for each reference word. The Chinese label model can use a multi-label model trained based on a ResNet101 model using corresponding business data, and the business data includes an image and one or more Chinese labels corresponding to the image. Further, the obtained prediction probabilities can be normalized to obtain normalized prediction probabilities, and then based on the normalized prediction probabilities and in combination with the prediction threshold of each reference word, the accuracy evaluation value of each reference word is determined, for example, by using a softmax function to normalize the obtained prediction probabilities and map the prediction probabilities to between 0 and 1.

[0155] In the embodiments of the present application, for each reference word in the reference words, a corresponding prediction threshold is set in advance, and based on the prediction threshold of each reference word, the keyword set corresponding to each sample image can be determined using but not limited to the following two ways, for ease of description, only sample image a and reference word b are taken as examples for description, and sample image a is any one of the obtained sample images, and reference word b is any one of the reference words:

[0156] The first mode is: when the prediction probability of the sample image a for the reference word b is greater than the prediction threshold value corresponding to the reference word b, the reference word b is taken as the reference word possessed by the sample image a, that is, a key word of the sample image a.

[0157] The prediction threshold value of the reference word b can be between 0 and 1, for example, 0.5. The determination process of the prediction threshold value is described below.

[0158] For example, the reference words set for Chinese include: high-burning, romantic, small lake, and small environment. The prediction threshold values corresponding to high-burning, romantic, small lake, and small environment are 0.5, 0.5, 0.6, and 0.7 respectively. Assuming that the prediction probabilities of image 1 for high-burning, romantic, small lake, and small environment are 0.2, 0.6, 0.7, and 0.8 respectively, at this time, the prediction probability of image 1 for the reference word romantic is greater than the prediction threshold value corresponding to the reference word romantic, the prediction probability of image 1 for the reference word small lake is greater than the prediction threshold value corresponding to the reference word small lake, and the prediction probability of image 1 for the reference word small environment is greater than the prediction threshold value corresponding to the reference word small environment. Therefore, it is considered that romantic, small lake, and small environment are all reference words possessed by image 1, that is, romantic, small lake, and small environment are all key words of image 1.

[0159] The second mode is: based on the prediction probability of each sample image for each reference word, and in combination with the prediction threshold value of each reference word, the accuracy evaluation value of each reference word is determined, and then based on the obtained prediction probability, in combination with the prediction threshold value and the accuracy evaluation value of each reference word, the key word set corresponding to each sample image is obtained.

[0160] As a possible implementation, the accuracy evaluation value of each reference word can be determined in the following manner: still taking the reference word b as an example, based on the prediction probability of each sample image for the reference word b, the sample images corresponding to the prediction probability greater than the prediction threshold value of the reference word b are screened out from the sample images, then according to the set sample extraction quantity (K), the labeled samples are extracted from the screened sample images, and based on the labeled result corresponding to each labeled sample, the accuracy evaluation value of the reference word b is determined, wherein each labeled result is used to represent whether the corresponding labeled sample possesses the reference word b.

[0161] That is, in the embodiment of the present application, for the reference word b, the sample images possessing the reference word b are screened out from the obtained sample images, and K sample images (such as 1000) possessing the reference word b are extracted from the sample images possessing the reference word b for labeling, so as to determine whether the K sample images extracted possess the reference word b accurately. It should be noted that in order to ensure the labeling accuracy, the K sample images (such as 1000) extracted can be but are not limited to manually labeled.

[0162] The sampling of the K sample images can be performed in a random sampling manner, but is not limited thereto. The accuracy evaluation value can be represented by an accuracy (ACC), but is not limited thereto.

[0163] For example, the accuracy evaluation value (ACCb) of the reference word b can be calculated according to the following formula: ACCb=x / K, where x represents the number of the annotation results indicating that the corresponding annotation sample has the reference word b.

[0164] For example, for the reference word high-burning, the sample images having the reference word high-burning are filtered from the obtained sample images, 1000 sample images are sampled from the sample images having the reference word high-burning for annotation, and the ACC of the reference word high-burning is determined based on the annotation results corresponding to the 1000 sample images. Assuming that the annotation results indicate that 600 sample images of the 1000 sample images have the reference word high-burning, 600 / 1000=0.6 is taken as the ACC of the reference word high-burning.

[0165] Assuming that the number of the reference words is M, the accuracy evaluation value of each reference word can be obtained by the above implementation manner, and all label accuracies ACC={ACC_1, ACC_2, … ACC_M} are obtained.

[0166] In the above implementation manner, considering that the ACC of the label model is different for different labels, the ACC of different labels is obtained, and then the training samples are processed based on the ACC of different labels, so as to improve the model accuracy.

[0167] As a possible implementation manner, the prediction threshold of the reference word b can be determined in the following manner in the embodiment of the present application:

[0168] The label samples annotated for the reference word b are obtained, and the reference word prediction is performed on the label samples to obtain the prediction probability of each label sample for the reference word b;

[0169] Based on the prediction probability of each label sample for the reference word b, the accuracy evaluation value of the reference word b under each candidate threshold is determined in combination with the annotation result corresponding to each annotation sample;

[0170] Based on the accuracy evaluation value under each candidate threshold, the prediction threshold of the reference word b is determined from the candidate thresholds.

[0171] The label samples include one or more of the historical generated images and the images collected through the webpage, but are not limited thereto. The label samples can be manually annotated, but are not limited thereto.

[0172] One possible approach is to input each label sample into a Chinese labeling model to obtain the prediction probability of each label sample for the reference word b.

[0173] As one possible implementation, for each candidate threshold, based on the prediction probability of each labeled sample for the reference word b, and combined with the annotation results of each labeled sample, the accuracy evaluation value of the reference word b under that candidate threshold is determined. For example, when the prediction probability of one labeled sample is greater than the candidate threshold, the labeled sample is predicted to possess the reference word b. Combined with the annotation results of that labeled sample, it is determined whether the labeled sample is correctly predicted using the candidate threshold. Similarly, it is determined whether each labeled sample is correctly predicted using the candidate threshold. Therefore, the accuracy evaluation value of the candidate threshold can be expressed as: the number of correctly predicted labeled samples / the total number of labeled samples.

[0174] The prediction threshold for reference word b can be the candidate threshold with the highest corresponding accuracy evaluation value among all candidate thresholds, but it is not limited to this.

[0175] Taking the term "high-energy" as an example, please refer to... Figure 8 As shown, for each tag sample annotated with the reference word "highly trending", reference word prediction is performed on each tag sample to obtain the prediction probability of each tag sample for the reference word "highly trending". It is assumed that a threshold search is performed from 0.05 to 0.95, with a value of 0.05, meaning the candidate threshold values ​​include: 0.05, 0.10, 0.15, ..., 0.95. For a candidate threshold of 0.05, when the prediction probability of one of the annotated samples is greater than the candidate threshold of 0.05, the annotated sample is predicted to have the reference word "highly trending". Based on the annotation result of that annotated sample, it is determined whether the annotated sample is correctly predicted when using the candidate threshold of 0.05. Similarly, the proportion of correctly predicted tag samples in each annotated sample is determined when using the candidate threshold of 0.05, obtaining the ACC of the candidate threshold of 0.05. Similarly, we obtain the ACC corresponding to each of the candidate thresholds 0.05, 0.10, 0.15, ..., 0.95. Assuming that among the ACCs corresponding to the candidate thresholds 0.05, 0.10, 0.15, ..., 0.95, the ACC corresponding to the candidate threshold 0.60 is the highest, we use the candidate threshold 0.60 as the prediction threshold for the reference word "high-energy".

[0176] Obviously, by using the above implementation method, assuming there are M reference words, for the M reference words, we can obtain M prediction thresholds: T = {T1, T2, ... TM}.

[0177] Since there is a domain difference between the training samples and the training data used when the label model is modeled (for example, the label model is trained using short videos uploaded by users, while the label samples in the embodiments of the present application are images of movies and TV series), by labeling samples, the corresponding prediction threshold is determined for each reference word in advance, which can effectively avoid the problem of poor classification accuracy caused by domain difference.

[0178] Considering that different labels generate different results, and noise labels are unstable, the object corresponding to the label may not appear on the graph, may appear, and the degree of appearance may also be different (for example, a human body label is included, but only the fingers of the person appear), in the embodiments of the present application, a label enhancement method is designed to handle the problem of different degrees of noise labels, specifically, based on the obtained prediction probabilities, in combination with the prediction threshold and the accuracy evaluation value of each reference word, when obtaining at least one group of keywords corresponding to each sample image, the following methods can be used:

[0179] For each sample image in each sample image, taking sample image a as an example, when there is at least one reference word in each reference word whose corresponding prediction probability is not less than the corresponding prediction threshold, the at least one reference word is taken as at least one non-noise word of the sample image a;

[0180] When there is at least one other reference word in each other reference word whose corresponding accuracy evaluation value is less than the evaluation value threshold, the at least one other reference word is taken as at least one noise word of the sample image a;

[0181] Combining the at least one non-noise word and the at least one noise word, at least one group of keywords corresponding to the sample image a is obtained.

[0182] That is, in the embodiments of the present application, for the prediction probability of each reference word of each image, when the prediction probability of a reference word is greater than the prediction threshold of the reference word, it indicates that the sample image has a higher possibility of possessing the reference word, and at this time, the label does not need to be processed, that is, the sample image retains the label. When the prediction probability of a reference word is not greater than the prediction threshold of the reference word, it indicates that the sample image has a lower possibility of possessing the reference word, and the reference word is more likely to be a noise word.

[0183] The evaluation value threshold can be a random number or a pre-set fixed value, in the embodiments of the present application, if a fixed value is used, it means that label enhancement is performed for each reference word, if a random number is used, randomness can bring more generalization, which can improve the generalization ability of the model.

[0184] As a possible implementation, when the predicted probability of a reference word is not greater than the prediction threshold of the reference word, a random number is first generated. When the random number is greater than the ACC of the reference word, the reference word is regarded as a noise word, and subsequent noise enhancement is performed on the noise word. When enhancing the label with the ACC threshold (thr) as the random probability, for example, when the ACC is 0.7, it means that the accuracy is not so high. At this time, if a random number between 0.7 and 1 is generated, label enhancement will be performed; when the ACC is 0.9, it means that the accuracy is high, and a random number between 0.9 and 1 needs to be generated to enhance. That is, the higher the accuracy, the less enhancement is required.

[0185] For example, refer to Figure 9A As shown, the reference words set for Chinese includes: high-energy, romantic, small lake, small environment. Among them, the respective prediction thresholds corresponding to high-energy, romantic, small lake, and small environment are 0.5, 0.5, 0.6, and 0.7. Assume that the predicted probabilities of image 1 for high-energy, romantic, small lake, and small environment are 0.2, 0.4, 0.7, and 0.8 respectively. Obviously, the predicted probability of image 1 for the reference word "small lake" is greater than the prediction threshold corresponding to the reference word "small lake", and the predicted probability of image 1 for the reference word "small environment" is greater than the prediction threshold corresponding to the reference word "small environment". Therefore, "small lake" and "small environment" are regarded as non-noise words of image 1. Assume that the random number is 0.75. Refer to Figure 9B As shown, at this time, the ACC of the reference word "high-energy" is greater than 0.75, while the ACC of the reference word "romantic" is less than 0.75. Therefore, the reference word "romantic" is regarded as a noise word of image 1.

[0186] After obtaining the keyword sets corresponding to each sample image, the following methods can be used, but are not limited to, to obtain at least one keyword group corresponding to each sample image:

[0187] Method 1: Still taking sample image a as an example, combine each keyword in the keyword set corresponding to sample image a as a group of keywords corresponding to sample image a.

[0188] For example, the keywords included in the keyword set of image 1 are: romantic, small lake, small environment. Then, (romantic, small lake, small environment) is used as a group of keyword groups of image 1, and then based on image 1 and (romantic, small lake, small environment), a text-image pair is obtained.

[0189] Method 2: For each keyword in the keyword set corresponding to this image a, combine at least one non-noise word and at least one noise word in each keyword to obtain an initial phrase group, and use the initial phrase group as a group of keywords; delete at least one noise word from the initial phrase group respectively to obtain at least one enhanced phrase group, and use the at least one enhanced phrase group as at least one group of keywords.

[0190] For example, assuming that the non-noise words include 11, 14, 15, the noise words include 12, 13, combining the non-noise words and the noise words obtains an initial word group (11, 12, 13, 14, 15), then deleting the noise word 12 from the initial word group (11, 12, 13, 14, 15) obtains a set of enhanced words (11, 13, 14, 15), deleting the noise word 13 from the initial word group (11, 12, 13, 14, 15) obtains a set of enhanced words (11, 12, 14, 15), and deleting the noise word 12 and the noise word 13 from the initial word group (11, 12, 13, 14, 15) obtains a set of enhanced words (11, 14, 15), thus, 4 sets of keywords are obtained, (11, 12, 13, 14, 15), (11, 13, 14, 15), (11, 12, 14, 15), (11, 14, 15), wherein (11, 13, 14, 15), (11, 12, 14, 15), (11, 14, 15) are each set of expanded enhanced words.

[0191] For another example, the keywords included in the keyword set of the image 1 are: romantic, small lake, small environment, the non-noise words are small lake and small environment, and the noise word is romantic, then the keyword sets corresponding to the image 1 include (romantic, small lake, small environment) and (small lake, small environment).

[0192] In some embodiments, there are Chinese labels and English labels for each sample image, the Chinese label is translated into the English label and directly assigned to the corresponding sample image, at this time, the two sets of labels of the sample image are both English description labels. In application, it is also necessary to remove repeated labels, for example, for Chinese and English labels, the same or similar labels are merged to remove the Chinese and English labels with the same or similar semantics. For example, when it is determined that there is at least one similar word group meeting the similarity condition based on the similarity between each reference word set for English and each reference word set for Chinese, the two reference words included in the at least one similar word group are respectively merged to obtain a corresponding merged word.

[0193] Each similar word group includes one English reference word and one Chinese reference word meeting the similarity condition. The similarity condition can refer to that the similarity exceeds a similarity threshold, but is not limited thereto. The similarity between the English reference word and the Chinese reference word is used to represent the semantic similarity between the English reference word and the Chinese reference word. The calculation method of the similarity is not limited herein and will not be described here.

[0194] When the English reference word (a1) and the Chinese reference word (a2) are merged, the merged word (a3) can adopt a1 or a2, or other words determined based on a1 and a2, which are not limited. For example, the reference word face and the reference word face close-up are merged to obtain the merged word face.

[0195] In some implementations, the similarity between reference words set for Chinese characters can be used to merge multiple reference words that meet the similarity criteria, thereby obtaining a merged word. For example, the reference words "beard" and "whiskers" can be merged to obtain the merged word "beard".

[0196] In some implementations, English reference words with the same or similar meanings (i.e., reference words set for English) can be merged to obtain corresponding merged words. Specifically, when multiple reference words that meet the similarity conditions are determined based on the similarity between the reference words set for English, the multiple reference words are merged to obtain corresponding merged words.

[0197] By cleaning and fusing Chinese and English labels, a large number of usable labels can be obtained. Then, through label noise evaluation, noisy processing can be carried out during training, which can reduce the problem of generating noisy results to a certain extent.

[0198] Furthermore, based on the obtained at least one merged word, at least one set of merged words corresponding to each sample image is obtained, and the at least one set of merged words corresponding to each sample image is used as at least one set of keywords corresponding to each sample image.

[0199] Specifically, when obtaining at least one set of keywords corresponding to each sample image based on at least one merged word, one possible implementation is to construct reference words for the merged label system based on at least one merged word, other Chinese reference words besides the merged word, and other English reference words. Then, based on the reference words for the merged label system, the prediction probability of each sample image for each reference word is obtained, and the prediction probability of each sample image for each reference word is used to obtain at least one set of keywords corresponding to each sample image. Since the method of obtaining each set of keywords for each reference word of the merged label system is the same as the processing process for Chinese labels above, it will not be repeated here.

[0200] For example, the accuracy of the merged labels is the mean of the ACC of the original labels. For example, Acc_a3 = (Acc_a1 + Acc_a2) / 2, where Acc_a1 and Acc_a2 are the ACC of two semantically similar keywords.

[0201] Clearly, for a single sample image, there are three sets of labels: Chinese labels that have not been merged, English labels that have not been merged, and merged labels. All three sets of labels are translated into English. That is, based on a single sample image, three types of image-text pairs can be generated: the first type of image-text pair is generated based on the sample image and its corresponding Chinese labels, the second type of image-text pair is generated based on the sample image and its corresponding English labels, and the third type of image-text pair is generated based on the sample image and its corresponding merged labels.

[0202] The accuracy of the merged labels is the mean of the original labels' ACC, i.e., Acc_a3 = (Acc_a1 + Acc_a2) / 2.

[0203] After obtaining each sample image and its corresponding set of keywords, a set of image-text pairs is constructed based on each sample image and its corresponding set of keywords. Each image-text pair contains: a sample image and its corresponding set of keywords. In each image-text pair, the keywords obtained for the second language are obtained after language conversion with reference to the first language.

[0204] For example, see Figure 10 As shown, the Chinese tags for Image 1 include: 11, 12, 13, 14, 15, where 12 and 13 are noise words. Each Chinese tag is translated into English. The English tags for Image 1 include: 21, 22, 23, 24, 25, where 22 and 23 are noise words. The merged tags for Image 1 include: 1, 2, 3, 14, 15, 24, 25, where 3 and 14 are noise words. Based on the Chinese tags for Image 1, the following four keyword groups can be generated: (11, 12, 13, 14, 15), (11, 12, 14, 15), (11, 13, 14, 15), (11, 14, 15). The English tags for image 1 can generate the following four keyword groups: (21, 22, 23, 24, 25), (21, 22, 24, 25), (21, 23, 24, 25), (21, 24, 25). The merged tags for image 1 can generate the following four keyword groups: (1, 2, 3, 14, 15, 24, 25), (1, 2, 3, 15, 24, 25), (1, 2, 14, 15, 24, 25), (1, 2, 15, 24, 25). The remaining keyword groups are enhanced word groups. Obviously, for image 1, 12 image-text pairs can be generated, each containing image 1 and a set of keywords.

[0205] Further, in the embodiments of the present application, a batch of clean label data can also be obtained, M images (such as 10,000 images) are randomly sampled from all sample images, and Chinese and English labels are inferred again for the M images. At this time, the prediction threshold of all labels is set to 0.2 times the original value (such as from 0.5 to 0.1). By reducing the prediction threshold, each sample image can obtain more possible labels, thereby avoiding missing some labels. Then, each sample image and the newly predicted Chinese and English labels are manually checked. During the manual checking process, incorrect labels of the sample images are deleted, thereby obtaining a batch of more accurate image-text pairs. After that, the prediction probability of the checked sample images is rewritten as 1, indicating that the label is correct and does not need to be enhanced subsequently. For model training, the more accurate data, the better the model training effect, so as much as possible, the image-text pairs are manually annotated under the budget.

[0206] In the embodiments of the present application, after the image-text pairs for training are prepared, the image-text pairs can be used to train the constructed model.

[0207] (3) Iterative training phase

[0208] In the embodiments of the present application, the target image generation model is obtained by iteratively training the image generation model. In the model training process, a total of N rounds (such as 100) of iterations are performed on the full sample pairs. One round of iteration is that the full sample pairs are trained once in the image generation model. In each round of iteration, due to the limited memory resources of the training machine, the full sample pairs cannot be input into the model at one time for training, so all sample pairs need to be trained in batches. Each batch of samples is generated by, for example, random division, and each batch of samples is input into the model for forward calculation, backward calculation, model parameter update, and the like.

[0209] In some embodiments, first, bs image-text pairs are extracted from the full image-text pairs, and then for each image-text pair, k1 (such as 2) image-text pair variants are randomly generated according to the noise degree. These image-text pair variants are referred to as enhanced samples for noise, and are different from the original image-text pair only in text, and the images are the same. According to the noise degree, k1 (such as 2) image-text pair variants are randomly generated. Referring to the above-mentioned based on the obtained prediction probability, combined with the prediction threshold and the accuracy evaluation value of each reference word, at least one group of keywords corresponding to each sample image is obtained. Each group of keywords in the at least one group of keywords corresponding to each sample image, which is composed of each keyword in the keyword set corresponding to the sample image, can be referred to as the original image-text pair. Each group of keywords obtained by deleting noise words from each keyword in the keyword set can be referred to as an image-text pair variant.

[0210] That is, in the embodiments of the present application, the original image-text pair can be constructed based on the corresponding keyword set, and then the original image-text pair is enhanced to obtain the enhanced sample, or the original image-text pair and the enhanced sample can be constructed based on the corresponding keyword set at the same time, and this is not limited.

[0211] Before the first round of training, the parameter initialization of the image generation model is performed. Specifically, for the model parameters that do not need to be adjusted (such as the model parameters contained in the noise adding model, the image encoding sub-model, the text encoding sub-model, and the image decoding sub-model), the model parameters of the corresponding pre-trained model are adopted. Further, after the batch, the number of iterations, and the learning rate and other hyperparameters are set, the training is started, and finally the target image generation model is obtained. The initial value of the learning rate is 0.0001, and after learning for 5 rounds, the learning rate becomes 0.1 times the original value.

[0212] Referring to Figure 11 As shown in FIG. 1, a flowchart of an image generation model training method provided by an embodiment of the present application is shown. The training method can be applied to a server or a terminal device, and the specific process is as follows:

[0213] S1101, based on the obtained sample images, for the first language and the second language, the following operations are performed: based on the reference words set for the current language, the prediction probability of each sample image for each reference word is obtained, and combined with the prediction threshold of each reference word, the accuracy evaluation value of each reference word is determined; and based on the obtained prediction probability, combined with the prediction threshold and the accuracy evaluation value of each reference word, at least one group of keywords corresponding to each sample image is obtained.

[0214] S1102, based on each sample image and its corresponding each group of keywords, a set of image-text pairs is constructed, each image-text pair contains: a sample image and a group of keywords corresponding thereto, and in each image-text pair, the keywords obtained for the second language are the keywords obtained by converting the first language.

[0215] S1103, based on the set of image-text pairs, the image generation model pre-trained in the first language is iteratively trained to obtain a target image generation model.

[0216] Specifically, the execution process of S1101, S1102 is described above, which will not be repeated here. In the following, S1103 is described.

[0217] In the embodiment of the present application, in the iterative training process, all training samples are divided into specified batches, and training is performed based on the training samples of each batch. Since the steps performed when training each batch in each iteration process are similar, the training of one batch is taken as an example for description.

[0218] When S1103 is performed, in each iteration process, the keywords in each sample image-text pair are input into the image generation model respectively to obtain the corresponding predicted image. Then, the model loss of each group of keywords is obtained based on the predicted image and the sample image corresponding to each group of keywords, and the parameter adjustment is performed based on the model loss.

[0219] Specifically, the image generation model includes a noise adding model, a noise reducing model, a text encoding sub-model, an image encoding sub-model, and an image decoding sub-model. In each iteration process, the predicted image can be obtained in the following manner:

[0220] For each group of keywords in the selected sample image-text pair, first, a random image is input into the image encoding sub-model to obtain image encoding, and the image encoding is input into the noise adding model to obtain noise-added image encoding.

[0221] Secondly, a group of keywords is input into the text encoding sub-model to obtain text encoding, and the text encoding and the noise-added image encoding are input into the noise reducing model to obtain the image encoding of the predicted image corresponding to the group of keywords.

[0222] Then, the image encoding of the predicted image is input into the image decoding sub-model to obtain the predicted image corresponding to the group of keywords.

[0223] After obtaining the predicted image, the model loss of the group of keywords can be obtained based on the predicted image and the sample image corresponding to the group of keywords in the following two loss calculation methods:

[0224] Loss calculation method 1: the mean square error (MSE) loss is used to calculate the pixel-by-pixel loss of the predicted image and the sample image corresponding to the group of keywords.

[0225] For example, the calculation formula of MSE is as follows:

[0226]

[0227] Wherein, y i represents the sample image of the i-th group of keywords, represents the predicted image of the i-th group of keywords, and n represents the number of image-text pairs included in the current batch.

[0228] For example, refer to Figure 12As shown, first, the image encoding (Z) of the random image x generated for the random seed xi is obtained through a diffusion process to obtain the latent space vector (Z T ), then for a text-image pair, a set of keywords in the text-image pair is taken as multi-label information i, and the noise feature prediction (Z') at time 0 is obtained through T times of denoising with the aid of the denoising U-Net network, and the predicted image (image i) is obtained after passing through the D module. Based on the predicted image and the sample image in the text-image pair, the corresponding model loss is obtained according to the calculation formula of MSE. For specific model processing process, please refer to Figure 2 .

[0229] Loss calculation method 2: When calculating the enhanced label text-image pair, since the enhanced text reduces some possible noise labels compared to the original, it is necessary to ensure that the image used for supervision does not have these labels. The method of this scheme is to erase the loss at the position where the image has a response, that is, not to calculate the loss at this position to achieve it.

[0230] Specifically, for each set of keywords in the selected sample text-image pair, the following operations are performed respectively:

[0231] If a set of keywords contains noise words, the model loss is obtained based on the predicted image and the sample image corresponding to the set of keywords;

[0232] If a set of keywords does not contain noise words, the sample image corresponding to the set of keywords is processed by image erasing based on the deleted noise words in the set of keywords, to obtain the processed sample image, and the model loss is obtained based on the predicted image and the processed sample image corresponding to the set of keywords.

[0233] Wherein, the case of containing noise words refers to loss calculation method 1. When a set of keywords contains part of the noise words of the sample image or does not contain noise words, it is considered that a set of keywords does not contain noise words. When a set of keywords contains all the noise words of the sample image, it is considered that a set of keywords contains noise words.

[0234] As a possible implementation, in the case of containing noise words, the following methods can be used for image erasing, but are not limited to:

[0235] Based on the heat map position of the deleted noise words in a set of keywords in the corresponding sample image, the image erasing position of the corresponding sample image is determined, and the image erasing processing is performed on the corresponding sample image based on the image erasing position to obtain the processed sample image.

[0236] Wherein, the image erasing processing can refer to adjusting the pixel value of the image erasing position in the sample image to a set value, but is not limited thereto.

[0237] In some embodiments, for each of the reference words, taking the sample image a and the reference word b as an example, the heat map position of the reference word b in the sample image a can be obtained by, but not limited to, the following ways:

[0238] 1) After the class activation mapping (CAM) for the reference word b and the sample image a, the heat map of the reference word b in the sample image a is obtained.

[0239] 2) Based on the maximum value max and the minimum value min in the heat map, the heat map threshold is obtained, and the heat map value of the pixel in the heat map whose heat map value is greater than the heat map threshold is adjusted to a first value, and the heat map value of the pixel in the heat map whose heat map value is not greater than the heat map threshold is adjusted to a second value.

[0240] For example, the heat map threshold thre=(max-min)*0.7+min, when the heat map value of a pixel is greater than thre, the heat map value of the pixel is set to 1, and when the heat map value of the pixel is less than thre, the heat map value of the pixel is set to 0.

[0241] 3) Based on the adjusted heat map value, each connected region contained in the heat map is determined, and the final connected region is determined from each connected region.

[0242] As a possible implementation, the largest connected region composed of the first value of the heat map value can be taken as a connected region. For example, starting from any pixel with a heat map value of 1 in the heat map, when a pixel with a heat map value of 1 appears around the pixel, it is merged into the connected region of the pixel, and when there is no pixel with a heat map value of 1 around the position, it indicates that the calculation of one connected region has been completed, and the connected region and the number of pixels with a heat map value of 1 (representing the area of the connected region) are saved. Then, starting from the next pixel with a heat map value of 1 in the heat map that has not been merged, the next connected region is calculated. When all pixels with a heat map value of 1 are merged, the largest connected region is taken as the final connected region.

[0243] 4) Based on the final connected region, the heat map position of the reference word b is determined.

[0244] As a possible implementation, based on the coordinates of the pixels with a heat map value of 1 in the final connected region, the coordinates of the leftmost top pixel are taken as x1 y1, and the coordinates of the rightmost bottom pixel are taken as x2 y2, so that the heat map position with the maximum response to the reference word b can be framed, and recorded as label1(x1, y1, x2, y2).

[0245] As another possible implementation, the heatmap position of reference word b can also be obtained by taking the coordinates of the leftmost, rightmost, topmost, and bottommost pixels of the pixels with a heatmap value of 1 in the final connected region.

[0246] For example, see Figure 13 As shown, it is a schematic diagram of the heat map positions in two sample images provided in the embodiments of this application. Figure 13 In the image, the two images on the left are sample images, and the two images on the right are heatmaps of the reference word "toothbrush" in the two sample images. The area enclosed in the heatmap is the heatmap area corresponding to the toothbrush, and the diagonal area can be red. Red indicates the part of the heatmap with a large heatmap value and a response.

[0247] See Figure 14 As shown, in the case of noise words, assuming that the noise word is a toothbrush, and based on the fact that a set of keywords in the image-text pair does not contain a toothbrush, the sample objects in the image-text pair are processed by image erasure to obtain the processed sample image. The heat map position of the toothbrush in the processed sample image is erased. Then, based on the predicted image and the processed sample image, the model loss corresponding to the set of keywords is obtained.

[0248] See Figure 15 As shown, for each set of keywords in each image pair, after obtaining the corresponding predicted samples through the image generation model, based on the corresponding predicted images (such as predicted image 1, predicted image 2, predicted image 3, etc.) and sample images (such as image 1), the model loss corresponding to each set of keywords (such as loss 1, loss 2, loss 3) can be obtained. Then, the model losses corresponding to each set of keywords are summarized to obtain the total model loss.

[0249] Incorporating noisy labels during training increases model robustness, avoids loss of diversity in generated results, and suppresses noise labels, thereby improving overall generation quality. Furthermore, by erasing noisy labels absent in the augmented samples and calculating loss based on the processed images, even with incorrect original labels, more accurate training information can be obtained with the aid of different augmented samples, achieving high-quality training with noisy labels. In new domains, training text-image pairs with noise uses a label image erasure method to introduce new labeled text-image pairs as augmented data. This introduces text augmentation information while avoiding erroneous information augmentation, providing more accurate information for model training and improving training effectiveness.

[0250] In some embodiments, to improve the model training efficiency, the model can be fine-tuned, specifically, based on the model loss, the parameter adjustment is performed on each model parameter contained in the denoising sub-model, that is, the model parameters of other sub-models except the denoising sub-model are fixed.

[0251] In some embodiments, the model loss can be back-propagated into the model by using, but not limited to, Stochastic Gradient Descent (SGD) and the like, to obtain the gradient of the model parameters (the model parameters of the U-Net) and update the model parameters.

[0252] In some embodiments, after completing the training of one batch, the iteration process is ended.

[0253] In some embodiments, before performing the model parameter adjustment, it can also be determined whether the model convergence condition is met. Exemplarily, the model convergence condition can include at least one of the following conditions: the model loss is not greater than a preset loss threshold; the number of iterations reaches a preset upper limit value.

[0254] In some embodiments, the target image generation model obtained through the model training can also be used to generate a target image corresponding to a specified text. Specifically, first, the text input by the target object is obtained, and based on the text, at least one keyword contained in the text is obtained, second, the at least one keyword is input into the target image generation model to obtain at least one candidate generated image, and finally, based on the aesthetic evaluation information corresponding to each of the at least one candidate generated image, the target generated image is determined.

[0255] In the embodiments of the present application, the text input mode includes, but is not limited to, voice, text, etc. If the target object inputs the text through voice, the voice technology can be used to convert the input voice information into text. The text can contain one or more keywords. In the embodiments of the present application, the extraction method of the keyword is not limited, and will not be described here. The aesthetic evaluation information can also be referred to as the aesthetic evaluation value. The number of target generated images can be one or more, which is not limited.

[0256] After obtaining the candidate generated image, the aesthetic evaluation model can be used to determine the aesthetic evaluation information corresponding to each of the at least one candidate generated image, and based on the aesthetic evaluation information corresponding to each of the at least one candidate generated image, the topK candidate generated images in the order of value from high to low are selected as the target generated images from the at least one candidate generated image.

[0257] Referring to Figure 16As shown, after the server obtains the text containing at least one keyword input by the target object through the client, the server generates z random seeds, then inputs the z random seeds and the at least one keyword contained in the text into the target image generation model respectively to obtain z candidate generated images, then determines K target generated images based on the aesthetic degree evaluation information corresponding to each of the z candidate generated images, and returns the K target generated images to the client to enable the client to present the K target generated images to the target object.

[0258] For example, if a fine poster or wallpaper needs to be generated for the scene label of a certain film and television drama, the target object inputs the labels related to the film and television drama, such as smoke and rain in West Lake, Leifeng Pagoda, spring, peach blossoms, willow, White Snake Legend, etc. After the target image generation model generates multiple candidate generated images related to spring in West Lake and Leifeng Pagoda, the candidate generated images are sorted through aesthetic degree sorting, so that the operation interface displays the top K candidate generated images in the order from high to low according to the aesthetic degree.

[0259] Referring to Figure 17 As shown, it is a schematic diagram of an image generation model training process provided in an embodiment of the present application. For each candidate image extracted from a film and television drama, first, when it is determined that the scene type of the candidate image is the target scene type and the candidate image contains a person, the person is erased for the candidate object to obtain an erased image. Then, the erased image and the candidate image are both used as sample images. Then, for each sample image, Chinese label prediction and English label prediction are performed respectively to obtain the prediction probability of each label, and label accuracy evaluation of each label is performed to obtain the ACC of each label. Then, based on the ACC, the prediction threshold and the prediction probability of each label, the keyword set of Chinese and English of the sample image is determined. Then, the noise label enhancement is performed on the noise words in the keyword set to obtain multiple groups of keywords. Each group of keywords in the multiple groups of keywords contains each keyword in the keyword set, and one or more groups of keywords after deleting the noise words. Then, the sample image and the multiple groups of keywords corresponding thereto are combined to obtain multiple image-text pairs. Then, the image generation model is trained based on the multiple image-text pairs. In the training process, the model loss corresponding to a group of keywords containing all noise words is called original loss, and the model loss corresponding to a group of keywords not containing noise words (partially or completely) is called noise enhancement loss. The original loss and the noise enhancement loss are combined to adjust the parameters of the model to complete the iterative training.

[0260] Based on the same inventive concept, an image generation model training device is provided in an embodiment of the present application. As Figure 18 As shown, it is a structural schematic diagram of the image generation model training device 1800, which can include:

[0261] The label prediction unit 1801 is configured to, based on the obtained sample images, perform the following operations for the first language and the second language respectively: obtaining, based on the reference words set for the current language, the prediction probability of each sample image for each reference word, and determining the accuracy evaluation value of each reference word in combination with the prediction threshold of each reference word; and based on the obtained prediction probability, obtaining at least one group of keywords corresponding to each sample image in combination with the prediction threshold and the accuracy evaluation value of each reference word;

[0262] The image-text pair construction unit 1802 is configured to, based on the sample images and the groups of keywords corresponding to the sample images respectively, construct an image-text pair set, each image-text pair comprising: a sample image and a group of keywords corresponding to the sample image, and the keywords obtained for the second language in each image-text pair are keywords obtained by language conversion from the first language;

[0263] The training unit 1803 is configured to, based on the image-text pair set, iteratively train an image generation model pre-trained in the first language to obtain a target image generation model.

[0264] As a possible implementation manner, the label prediction unit 1801 is further configured to:

[0265] obtain candidate images in a target application scenario, and obtain the scene categories corresponding to the candidate images respectively by using a scene category classification model;

[0266] When at least one candidate image meeting a set erasing condition is filtered out from the candidate images based on the scene categories corresponding to the candidate images respectively, perform object erasing on the at least one candidate image respectively to obtain corresponding erasing images;

[0267] use the obtained at least one erasing image and the other candidate images except the at least one candidate image in the candidate images as sample images.

[0268] As a possible implementation manner, when the at least one candidate image meeting the set erasing condition is filtered out from the candidate images based on the scene categories corresponding to the candidate images respectively, the label prediction unit 1801 is specifically configured to:

[0269] When a candidate image whose corresponding scene category is a target scene category is filtered out from the candidate images based on the scene categories corresponding to the candidate images respectively, perform object detection on the filtered candidate images by using an object detection model;

[0270] Based on the detection result, at least one candidate image containing the target object in each of the screened candidate images is determined as at least one candidate image meeting the set erasing condition.

[0271] As a possible implementation, when the accuracy evaluation value of each reference word is determined based on the prediction probability of each sample image for each reference word and in combination with the prediction threshold of each reference word, the label prediction unit 1801 is specifically configured to:

[0272] For each of the reference words, the following operations are performed respectively:

[0273] Based on the prediction probability of each sample image for one reference word, sample images with a prediction probability greater than the prediction threshold of the one reference word are screened from the sample images;

[0274] According to the set sample extraction quantity, labeled samples are extracted from the screened sample images, and the accuracy evaluation value of the one reference word is determined based on the corresponding label result of each labeled sample, wherein each label result is used to represent whether the corresponding labeled sample has the one reference word.

[0275] As a possible implementation, the label prediction unit 1801 is further configured to:

[0276] Obtain each label sample labeled for one reference word, and perform reference word prediction on the label samples to obtain the prediction probability of each label sample for the one reference word;

[0277] Based on the prediction probability of each label sample for the one reference word and in combination with the corresponding label result of each labeled sample, the accuracy evaluation value of the one reference word under each candidate threshold is determined.

[0278] Based on the accuracy evaluation value under each candidate threshold, the prediction threshold of the one reference word is determined from the candidate thresholds.

[0279] As a possible implementation, when the at least one group of keywords corresponding to each sample image is obtained based on the obtained prediction probabilities in combination with the prediction threshold and the accuracy evaluation value of each reference word, the label prediction unit 1801 is specifically configured to:

[0280] For each sample image in the sample images, the following operations are performed respectively:

[0281] When there is at least one reference word in the reference words with a prediction probability not less than the corresponding prediction threshold, the at least one reference word is taken as at least one non-noise word of the current sample image.

[0282] when there is at least one other reference word corresponding to an accuracy evaluation value less than the evaluation value threshold in each of the other reference words except for the at least one reference word, taking the at least one other reference word as at least one noise word of the current sample image;

[0283] combining the at least one non-noise word and the at least one noise word to obtain at least one keyword set corresponding to the current sample image.

[0284] As a possible implementation, when the at least one target word and the at least one noise word are combined to obtain at least one keyword set corresponding to the current sample image, the label prediction unit 1801 is specifically configured to:

[0285] combining the at least one non-noise word and the at least one noise word to obtain an initial word set, and taking the initial word set as a keyword set;

[0286] deleting the at least one noise word from the initial word set respectively to obtain at least one enhanced word set, and taking the at least one enhanced word set as at least one keyword set.

[0287] As a possible implementation, the training unit 1803 is further configured to, in each iteration process, perform the following operations on each keyword set in each selected sample image-text pair:

[0288] if a keyword set contains a noise word, directly obtaining a model loss corresponding to the keyword set based on a predicted image and a sample image corresponding to the keyword set;

[0289] if a keyword set does not contain a noise word, performing image erasing processing on a corresponding sample image based on a deleted noise word in the keyword set to obtain a processed sample image, and obtaining a model loss corresponding to the keyword set based on a predicted image corresponding to the keyword set and the processed sample image.

[0290] As a possible implementation, when the processed sample image is obtained by performing image erasing processing on the corresponding sample image based on the deleted noise word in the keyword set, the training unit 1803 is specifically configured to:

[0291] determining an image erasing position of the sample image based on a heat map position of the deleted noise word in the sample image;

[0292] performing image erasing processing on the sample image based on the image erasing position to obtain a processed sample image.

[0293] As a possible implementation manner, the label prediction unit 1801 is further configured to:

[0294] When it is determined that there is at least one similar word group meeting the similarity condition based on the similarity between the reference words set for the first language and the reference words set for the second language, the two reference words respectively included in the at least one similar word group are merged to obtain a corresponding merged word;

[0295] Based on the obtained at least one merged word, at least one group of keywords corresponding to each of the sample images is obtained.

[0296] As a possible implementation manner, the image generation model comprises: a noise adding model, a noise reducing model, a text encoding sub-model, an image encoding sub-model, and an image decoding sub-model. The training unit 1803 is configured to, in each iteration process, for each group of keywords in each sample image-text pair selected, respectively perform the following operations:

[0297] inputting a random image into the image encoding sub-model to obtain image encoding, and inputting the image encoding into the noise adding model to obtain noise-added image encoding;

[0298] inputting a group of keywords into the text encoding sub-model to obtain text encoding, and inputting the text encoding and the noise-added image encoding into the noise reducing model to obtain image encoding of a predicted image corresponding to the group of keywords;

[0299] inputting the image encoding of the predicted image into the image decoding sub-model to obtain the predicted image corresponding to the group of keywords.

[0300] As a possible implementation manner, the training unit 1803 is further configured to:

[0301] based on the model loss, adjusting parameters of each model parameter included in the noise reducing model.

[0302] As a possible implementation manner, the training unit 1803 is further configured to:

[0303] obtaining text input by a target object, and based on the text, obtaining at least one keyword included in the text;

[0304] inputting the at least one keyword into the target image generation model to obtain at least one candidate generated image;

[0305] based on the aesthetic evaluation information corresponding to each of the at least one candidate generated image, determining a target generated image.

[0306] For ease of description, the above parts are divided into modules (or units) according to functions and are described respectively. Of course, functions of the modules (or units) can be implemented in one or more software or hardware in the implementation of the present application.

[0307] As to the apparatus in the above embodiments, the specific manners in which the respective units perform the requests have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0308] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as follows: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software, which can be collectively referred to as "circuitry", "module" or "system".

[0309] Based on the same inventive concept, the embodiments of the present application also provide an electronic device. In an embodiment, the electronic device can be a server or a terminal device. Referring to Figure 19 As shown in the figure, it is a structural schematic diagram of a possible electronic device provided in the embodiments of the present application, Figure 19 In the embodiment, the electronic device 1900 includes a processor 1910 and a memory 1920.

[0310] The memory 1920 stores a computer program executable by the processor 1910. The processor 1910 can execute the steps of the image generation model training method by executing the instructions stored in the memory 1920.

[0311] The memory 1920 can be a volatile memory, for example, a random-access memory (RAM); the memory 1920 can also be a non-volatile memory, for example, a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1920 can be any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 1920 can also be a combination of the above memories.

[0312] The processor 1910 can include one or more central processing units (CPUs), or be a digital processing unit, etc. The processor 1910 is configured to execute the computer program stored in the memory 1920, so as to implement the above-mentioned image generation model training method.

[0313] In some embodiments, the processor 1910 and the memory 1920 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0314] The specific connection medium between the processor 1910 and the memory 1920 in the embodiments of the present application is not limited. In the embodiments of the present application, the connection between the processor 1910 and the memory 1920 through the bus is taken as an example. The connection mode between other components is only schematically described, and is not limited. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, only one thick line is described in the Figure 19 , but it is not described that there is only one bus or only one type of bus. Figure 19

[0315] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium, which includes a computer program, when the computer program runs on an electronic device, the computer program is configured to enable the electronic device to execute the steps of the above-mentioned image generation model training method. In some possible implementation manners, each aspect of the image generation model training method provided by the present application can also be implemented in the form of a program product, which includes a computer program, when the program product runs on an electronic device, the computer program is configured to enable the electronic device to execute the steps in the above-mentioned image generation model training method, for example, the electronic device can execute the steps as shown in Figure 11 .

[0316] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0317] ​The program product of the embodiments of the present application can employ a CD-ROM and include a computer program, and can be executed on an electronic device. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with a command execution system, device, or apparatus.

[0318] The readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a readable computer program is borne. Such a propagated data signal can take on various forms, including but not limited to electro-magnetic signal, optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can be used to carry or transmit a computer program for use by or in connection with a command execution system, device, or apparatus.

[0319] Although the preferred embodiments of the present application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they grasp the fundamental inventive concept. Therefore, the appended claims are intended to encompass all such changes and modifications as falling within the scope of the present application.

[0320] Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if such modifications and changes of the present application fall within the scope of the claims and their equivalents, the present application is also intended to include such modifications and changes.

Claims

1. A method for training an image generation model, characterized in that, The method comprises the following steps: Based on the obtained sample images, the following operations are performed for the first language and the second language respectively: based on the reference words set for the current language, the prediction probability of each sample image for each reference word is obtained, and the accuracy evaluation value of each reference word is determined by combining the prediction threshold value of each reference word; based on the obtained prediction probability, at least one group of keywords corresponding to each sample image is obtained by combining the prediction threshold value and the accuracy evaluation value of each reference word; the reference word represents a candidate label, and the keyword represents a label possessed by the corresponding sample image; Based on the sample images and the corresponding groups of keywords, a set of image-text pairs is constructed, each image-text pair comprising a sample image and a group of keywords corresponding to the sample image, and the keywords obtained for the second language in each image-text pair further include keywords obtained by converting the first language to the second language; Based on the set of image-text pairs, the image generation model pre-trained in the first language is iteratively trained to obtain a target image generation model.

2. The method of claim 1, wherein, The sample images are obtained by the following method: Obtain each candidate image in the target application scenario, and use a scene category classification model to obtain the scene category corresponding to each candidate image; When at least one candidate image meeting the set erasing condition is screened out from the candidate images based on the scene categories corresponding to the candidate images, the at least one candidate image is subjected to object erasing to obtain a corresponding erasing image; The obtained at least one erasing image and the other candidate images in the candidate images except the at least one candidate image are used as sample images.

3. The method of claim 2, wherein, The method comprises the following steps: When the candidate images corresponding to the target scene category are screened out from the candidate images based on the scene categories corresponding to the candidate images, an object detection model is used to perform object detection on the screened candidate images; Based on the detection result, at least one candidate image containing a target object in the screened candidate images is used as at least one candidate image meeting the set erasing condition.

4. The method of claim 1, 2 or 3, wherein, Based on the prediction probability of each sample image for each reference word, the accuracy evaluation value of each reference word is determined by combining the prediction threshold value of each reference word, which comprises the following steps: For each reference word in the reference words, the following operations are performed respectively: Based on the prediction probability of each sample image for a reference word, sample images with a prediction probability greater than the prediction threshold value of the reference word are screened out from the sample images; According to the set sample extraction quantity, each labeled sample is extracted from the screened sample images, and the accuracy evaluation value of the reference word is determined based on the labeled result corresponding to each labeled sample, wherein each labeled result is used to represent whether the corresponding labeled sample possesses the reference word.

5. The method of claim 1, 2 or 3, wherein, The prediction threshold of each of the reference words is determined by the following steps: Obtaining each label sample labeled for one reference word, and performing reference word prediction on the label samples to obtain the prediction probability of each label sample for the one reference word; Based on the prediction probability of each label sample for the one reference word, and combining the label result corresponding to each label sample, the accuracy evaluation value of the one reference word under each candidate threshold is determined; Based on the accuracy evaluation value under each candidate threshold, the prediction threshold of the one reference word is determined from the candidate thresholds.

6. The method of claim 1, 2 or 3, wherein, Based on the obtained prediction probability, and combining the prediction threshold and the accuracy evaluation value of each reference word, at least one group of keywords corresponding to each sample image is obtained, including: For each sample image in the sample images, the following operations are performed respectively: When there is at least one reference word in the reference words whose corresponding prediction probability is not less than the corresponding prediction threshold, the at least one reference word is taken as at least one non-noise word of the current sample image; When there is at least one other reference word in the other reference words except the at least one reference word, whose corresponding accuracy evaluation value is less than the evaluation value threshold, the at least one other reference word is taken as at least one noise word of the current sample image; Combining the at least one non-noise word and the at least one noise word, at least one group of keywords corresponding to the current sample image is obtained.

7. The method of claim 6, wherein, The combination of the at least one target word and the at least one noise word to obtain at least one group of keywords corresponding to the current sample image includes: Combining the at least one non-noise word and the at least one noise word to obtain an initial word group, and taking the initial word group as a group of keywords; From the initial word group, the at least one noise word is deleted respectively to obtain at least one group of enhanced words, and the at least one group of enhanced words is taken as at least one group of keywords.

8. The method of claim 1, 2, or 3, wherein, In each iteration process, for each group of keywords in the selected sample image-text pair, the following operations are performed respectively: If a group of keywords contains noise words, the model loss corresponding to the group of keywords is directly obtained based on the prediction image and the sample image corresponding to the group of keywords; If a group of keywords does not contain noise words, the sample image is subjected to image erasing processing based on the deleted noise words in the group of keywords, to obtain a processed sample image, and the model loss corresponding to the group of keywords is obtained based on the prediction image corresponding to the group of keywords and the processed sample image.

9. The method of claim 8, wherein, The image erasing processing of the sample image based on the deleted noise words in the group of keywords to obtain a processed sample image includes: Based on the heat map position of the deleted noise words in the group of keywords in the corresponding sample image, the image erasing position of the sample image is determined; Based on the image erasing position, the sample image is subjected to image erasing processing to obtain a processed sample image.

10. The method of claim 1, 2, or 3, wherein, Before the constructing the set of image-text pairs based on the sample images and the respective groups of keywords corresponding to the sample images, the method further comprises: When it is determined that there is at least one similar keyword group meeting the similarity condition based on the similarity between the reference keywords set for the first language and the reference keywords set for the second language, the two reference keywords included in the at least one similar keyword group are merged respectively to obtain corresponding merged keywords; Based on the obtained at least one merged keyword, at least one group of keywords corresponding to the sample images is obtained.

11. The method of claim 1, 2, or 3, wherein, The image generation model comprises a noise adding sub-model, a noise reducing sub-model, a text encoding sub-model, an image encoding sub-model, and an image decoding sub-model. In each iteration process, for each group of keywords in the selected sample image-text pairs, the following operations are performed respectively: The random image is input into the image encoding sub-model to obtain image encoding, and the image encoding is input into the noise adding sub-model to obtain noise-added image encoding; A group of keywords is input into the text encoding sub-model to obtain text encoding, and the text encoding and the noise-added image encoding are input into the noise reducing sub-model to obtain image encoding of a predicted image corresponding to the group of keywords; The image encoding of the predicted image is input into the image decoding sub-model to obtain the predicted image corresponding to the group of keywords.

12. The method of claim 11, wherein, Further comprising: Based on the model loss, the model parameters included in the noise reducing sub-model are adjusted.

13. The method of claim 1, 2, or 3, wherein, After the iterative training of the image generation model pre-trained in the first language based on the set of image-text pairs to obtain a target image generation model, the method further comprises: Obtaining text input by a target object, and based on the text, obtaining at least one keyword included in the text; Inputting the at least one keyword into the target image generation model to obtain at least one candidate generated image; Based on the aesthetic evaluation information corresponding to the at least one candidate generated image, determining a target generated image.

14. An image generation model training apparatus characterized by comprising: Comprise: The label prediction unit is configured to, based on the obtained sample images, perform the following operations for the first language and the second language respectively: based on the reference keywords set for the current language, obtaining the prediction probability of the sample images for the reference keywords, and combining the prediction threshold and the accuracy evaluation value of the reference keywords to determine the accuracy evaluation value of the reference keywords; and based on the obtained prediction probability, combining the prediction threshold and the accuracy evaluation value of the reference keywords to obtain at least one group of keywords corresponding to the sample images; the reference keywords represent candidate labels, and the keywords represent labels possessed by the corresponding sample images; The image-text pair construction unit is configured to, based on the sample images and the respective groups of keywords corresponding to the sample images, construct a set of image-text pairs, each image-text pair comprising a sample image and a group of keywords corresponding to the sample image, and the keywords obtained for the second language in each image-text pair further comprise keywords obtained by language conversion from the first language. The training unit is configured to perform iterative training on the image generation model pre-trained in the first language based on the set of image-text pairs, to obtain a target image generation model.

15. An electronic device, comprising: It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method in any one of claims 1-13.

17. A computer program product, characterised in that, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method in any one of claims 1-13.

Images