Image generation model training method, device, computer equipment and storage medium

Abstract

The present application provides a training method, device, computer equipment and storage medium for an image generation model, which belongs to the field of computer technology. The method includes: obtaining multiple sample images generated by multiple first image generation models; for any sample image, processing the sample noise text, the sample description text of the sample image and the sample image series information of the sample image by the second image generation model to obtain a first latent space feature, and predicting a first predicted image based on the first latent space feature; processing the sample noise text, the sample description text and the sample image series information by the first image generation model that generates the sample image to obtain a second latent space feature, and predicting a second predicted image based on the second latent space feature; training the second image generation model based on the first predicted image and the second predicted image. The above technical solution not only improves the accuracy of the first image generation model, but also protects the privacy of the user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, apparatus, computer equipment, and storage medium for an image generation model. Background Art

[0002] With the advancement of computer technology, clients can train image generation models obtained from servers based on multiple images uploaded by users, resulting in customized image generation models. To improve the generalization capabilities of the server-side image generation model, the server needs to obtain training data from the client-side image generation model to train the server-side image generation model. However, to protect user privacy, the client cannot send training data directly to the server. Therefore, how to train the server-side image generation model while protecting user privacy is a technical problem that needs to be solved.

[0003] In related technologies, horizontal federated learning is often used to train server-side image generation models. During training, multiple clients do not need to send training data for the image generation model to the server. Instead, they send gradient information from the training process. The server can then update the server-side image generation model based on this gradient information.

[0004] However, the above solution only works when the client-side image generation model is trained synchronously with the server-side image generation model. If the client has already trained the image generation model, the client can only send gradient information to the server once. Because a single gradient information cannot overcome some errors caused by random operations within the image generation model, if the server only updates the server-side image generation model based on this gradient information, it will cause deviations in the image generation model, reducing the accuracy of image generation. Summary of the Invention

[0005] The present invention provides a method, apparatus, computer device, and storage medium for training an image generation model, which can improve the accuracy of a first image generation model while protecting user privacy. The technical solution is as follows:

[0006] In one aspect, a method for training an image generation model is provided, the method comprising:

[0007] Acquire a plurality of sample images generated by a plurality of first image generation models, the plurality of first image generation models being trained based on object images from a plurality of clients, the first image generation models being configured to generate images based on image series information and descriptive text, the image series information being configured to represent common features of the plurality of object images;

[0008] For any sample image, the second image generation model processes the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image to obtain a first latent space feature, and predicts a first predicted image based on the first latent space feature. The second image generation model has the same structure as the first image generation model but has different model parameters. The first latent space feature is the feature obtained by removing the first reference noise from the global image latent space feature. The global image latent space feature is the feature generated based on the sample description text and the sample image series information. The first reference noise is the noise generated based on the sample noise text and corresponds to the sample description text.

[0009] The first image generation model that generates the sample image processes the sample noise text, the sample description text, and the sample image series information to obtain a second latent space feature, and predicts a second predicted image based on the second latent space feature, where the second latent space feature is a feature of the sample image after removing a second reference noise, and the second reference noise is noise generated based on the sample noise text and corresponding to the sample description text;

[0010] The second predicted image is used as supervisory information and the second image generation model is trained based on the first predicted image and the second predicted image.

[0011] In another aspect, a training device for an image generation model is provided, the device comprising:

[0012] an acquisition module, configured to acquire a plurality of sample images generated by a plurality of first image generation models, the plurality of first image generation models being trained based on object images from a plurality of clients, the first image generation models being configured to generate images based on image series information and descriptive text, the image series information being configured to represent common features of the plurality of object images;

[0013] A first prediction module is configured to, for any sample image, process the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image using a second image generation model to obtain a first latent space feature, and predict a first predicted image based on the first latent space feature, wherein the second image generation model has the same structure as the first image generation model but has different model parameters, the first latent space feature is the feature obtained by removing the first reference noise from the global image latent space feature, the global image latent space feature is the feature generated based on the sample description text and the sample image series information, and the first reference noise is the noise generated based on the sample noise text and corresponding to the sample description text;

[0014] a second prediction model configured to process the sample noise text, the sample description text, and the sample image series information using the first image generation model to obtain second latent space features, and predict a second predicted image based on the second latent space features, where the second latent space features are features of the sample image after removing a second reference noise, and the second reference noise is noise generated based on the sample noise text and corresponding to the sample description text;

[0015] A training module uses the second predicted image as supervisory information and trains the second image generation model based on the first predicted image and the second predicted image.

[0016] In some embodiments, the first prediction module includes:

[0017] a first processing unit, configured to process, for any sample image, the sample description text and the sample image series information by the second image generation model to obtain the global image latent space feature;

[0018] a second processing unit, configured to process the sample noise text and the sample image series information using the second image generation model to obtain a first noise latent space feature, where the first noise latent space feature is used to represent the first reference noise;

[0019] The third processing unit is configured to subtract the global image latent space feature from the first noise latent space feature to obtain the first latent space feature, and predict the first predicted image based on the first latent space feature.

[0020] In some embodiments, the second prediction module includes:

[0021] a fourth processing unit, configured to process, for any sample image, the sample description text and the sample image series information using the first image generation model to obtain an image latent space feature, where the image latent space feature is a global feature of the sample image;

[0022] a fifth processing unit, configured to process the sample noise text and the sample image series information using the first image generation model to obtain a second noise latent space feature, wherein the second noise latent space feature is used to represent the second reference noise;

[0023] The sixth processing unit is configured to subtract the image latent space feature from the second noise latent space feature to obtain the second latent space feature, and predict the second predicted image based on the second latent space feature.

[0024] In some embodiments, the training module is used to perform object detection on the first predicted image and the second predicted image respectively, and determine a first area and a second area, where the first area is the area where the target object is located in the first predicted image, and the second area is the area where the target object is located in the second predicted image; based on the first area, feature extraction is performed on the first predicted image to obtain a first object latent space feature, which is the feature of the target object in the first predicted image; based on the second area, feature extraction is performed on the second predicted image to obtain a second object latent space feature, which is the feature of the target object in the second predicted image; using the second object latent space feature as supervision information, the second image generation model is trained based on the first object latent space feature and the second object latent space feature.

[0025] In some embodiments, the apparatus further comprises:

[0026] a first processing module, configured to process the sample noise text and the sample image series information using a third image generation model to obtain a third noise latent space feature, wherein the third image generation model is an initial image generation model, the second image generation model is trained by the third image generation model, and the third noise latent space feature is a feature generated based on the sample noise text and the sample image series information;

[0027] a second processing module, configured to process the sample noise text and the sample image series information using the second image generation model to obtain a fourth noise latent space feature, wherein the fourth noise latent space feature is a feature generated based on the sample noise text and the sample image series information;

[0028] The training module is used to train the second image generation model based on the first predicted image, the second predicted image, the third noise latent space features and the fourth noise latent space features, using the second predicted image and the third noise latent space features as supervisory information.

[0029] In some embodiments, the training module is used to determine a first loss based on the first predicted image and the second predicted image, where the first loss is used to indicate the difference between the first predicted image and the second predicted image; determine a second loss based on the third noise latent space feature and the fourth noise latent space feature, where the second loss is used to indicate the difference between the third noise latent space feature and the fourth noise latent space feature; perform weighted summation of the first loss and the second loss to obtain the training loss of the second image generation model; and update the model parameters of the second image generation model based on the training loss.

[0030] In some embodiments, the apparatus further comprises:

[0031] a selection module, configured to randomly select a first number of sample images from the plurality of sample images as detection samples based on a selection ratio, where the first number is a product of the selection ratio and the number of the plurality of sample images;

[0032] a first sampling module, configured to perform balanced sampling on the first number of detection samples based on multiple clarity levels to which the first number of detection samples belong, to obtain the first batch of training data, when the second image generation model is trained in a first batch;

[0033] The second sampling module is used to sample the first number of detection samples based on the level loss of multiple clarity levels belonging to multiple sample images in the n-1th batch of training data when the second image generation model is trained for the nth batch, to obtain the nth batch of training data, where n is an integer greater than 1.

[0034] In some embodiments, the first sampling module is used to randomly select a second number of classes from multiple classes of clarity levels for any clarity level among the multiple clarity levels to which the first number of detection samples belong; randomly select a third number of sample images from multiple sample images of the class for any class; and use the multiple sample images randomly selected from the multiple clarity levels as training data for the first batch.

[0035] In some embodiments, the nth batch of training data includes sampled samples of the plurality of definition levels, and for any definition level, the sampled samples of the definition level include sampled samples of a plurality of classes in the definition level;

[0036] The second sampling module is used to determine the sampling probabilities of the multiple clarity levels based on the level losses of the multiple clarity levels when the second image generation model is trained for the nth batch, and the level loss is positively correlated with the sampling probability; for any clarity level, determine the sampling probabilities of the multiple classes based on the sampling probability of the clarity level and the class losses of the multiple classes of the clarity level; for any class, sample multiple sample images of the class based on the sampling probability of the class to obtain sampling samples of the class.

[0037] In some embodiments, the apparatus further comprises:

[0038] a loss determination module, configured to determine, for any sample image in the (n-1)th batch of training data, a metric loss of the sample image based on the sample image and a predicted image, wherein the predicted image is an image predicted by the second image generation model based on the sample description text of the sample image and the sample image series information of the sample image, and the metric loss is used to indicate a difference between the sample image and the predicted image;

[0039] The loss determination module is further configured to determine, for any image series, an average value of the metric losses of a plurality of sample images of the image series as the series loss of the image series;

[0040] The loss determination module is further configured to determine, for any class, an average of series losses of multiple image series of the class as the class loss of the class;

[0041] The loss determination module is further configured to determine, for any clarity level, an average value of class losses of multiple classes of the clarity level as the level loss of the clarity level.

[0042] In some embodiments, the apparatus further comprises:

[0043] a determination module, configured to determine, for any sample image among the plurality of sample images, the image series to which the sample image belongs based on the sample description text of the sample image;

[0044] The determination module is further configured to determine, for any image series, an average value of the sharpness of a plurality of sample images in the image series as the sharpness of the image series;

[0045] The determining module is further configured to determine an average feature of semantic features of a plurality of sample images of the image series as the semantic feature of the image series, wherein the semantic feature of the sample image is used to indicate semantic information of the sample image;

[0046] a level determination module, configured to determine a plurality of clarity levels to which the plurality of image series belong based on the clarity of the plurality of image series and the number of the image series;

[0047] The clustering module is configured to cluster the multiple image series at any clarity level based on the semantic features of the multiple image series at the clarity level to obtain multiple classes to which the multiple image series belong.

[0048] In some embodiments, the level determination module is used to sort the multiple image series based on the clarity of the multiple image series; based on the sorting results of the multiple image series and the number of the image series, equally divide the multiple image series to obtain multiple clarity ranges to which the multiple image series belong, and the multiple clarity ranges correspond one-to-one to the multiple clarity levels; for any clarity range, determine the clarity level corresponding to the clarity range based on the clarity range.

[0049] In some embodiments, the clustering module is used to randomly select a fourth number of image series from multiple image series of any clarity level, where the fourth number is the number of classes; use the semantic features of the fourth number of image series as clustering centers of the multiple classes, cluster the multiple image series, and obtain the multiple classes.

[0050] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the training method of the image generation model in the embodiment of the present application.

[0051] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement a training method for an image generation model as in an embodiment of the present application.

[0052] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is executed by a processor to implement the training method of the image generation model provided in the embodiments of the present application.

[0053] The embodiment of the present application provides a method for training an image generation model, which obtains multiple sample images generated by multiple first image generation models trained based on object images in multiple clients as training data for a second image generation model, so that the server does not need to directly obtain multiple object images from multiple clients, but indirectly learns multiple object images by learning multiple sample images. Compared with directly obtaining the gradient information of multiple first image generation models, the above method processes the sample noise text and the sample description text and sample image series information of the sample image through the second image generation module and the first image generation model respectively for any sample image, and can train the second image model based on the first predicted image and the second predicted image generated by the features after removing the reference noise. The second image generation model after training can learn more accurate image information from multiple sample images, thereby improving the accuracy of the second image generation model and protecting the privacy of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 Schematic diagram of an implementation environment of a training method for an image generation model provided in an embodiment of the present application;

[0056] Figure 2 This is a flow chart of a training method for an image generation model provided in an embodiment of the present application;

[0057] Figure 3 This is a flow chart of another method for training an image generation model according to an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of an image generation interface provided according to an embodiment of the present application;

[0059] Figure 5 is a schematic diagram of a first image generation model provided according to an embodiment of the present application;

[0060] Figure 6 is a schematic diagram of a sample image grading process provided according to an embodiment of the present application;

[0061] Figure 7 is a schematic diagram of a sample image sampling process provided according to an embodiment of the present application;

[0062] Figure 8 This is a training block diagram of a second image generation model provided according to an embodiment of the present application;

[0063] Figure 9 is a training block diagram of another second image generation model provided according to an embodiment of the present application;

[0064] Figure 10 This is a schematic diagram of interaction between multiple clients and a server according to an embodiment of the present application;

[0065] Figure 11 This is a block diagram of a training device for an image generation model provided according to an embodiment of the present application;

[0066] Figure 12 is a block diagram of another image generation model training device provided according to an embodiment of the present application;

[0067] Figure 13 This is a structural block diagram of a terminal provided according to an embodiment of the present application;

[0068] Figure 14 This is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0070] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0071] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0072] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample images, sample description text, and sample image series information involved in this application were all obtained with full authorization.

[0073] The following explains the terms used in this application.

[0074] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0075] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0076] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0077] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to make them more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, map construction, autonomous driving, and smart transportation.

[0078] AIGC (AI Generated Content) is a production method that uses artificial intelligence technology to automatically generate content. AI painting, AI writing, AI imaging, etc. are all branches of AIGC.

[0079] The following will explain the training method of the image retrieval model provided in the embodiment of the present application based on artificial intelligence technology and computer vision technology.

[0080] The training method of the image generation model provided in the embodiment of the present application can be executed by a computer device. In some embodiments, the computer device is a terminal or a server. The following takes the computer device as an example to introduce the implementation environment of the training method of the image generation model provided in the embodiment of the present application. Figure 1 Schematic diagram of an implementation environment for a training method of an image generation model provided in accordance with an embodiment of the present application. Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0081] In some embodiments, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a car terminal, etc., but is not limited thereto. A client that supports AIGC is installed on the terminal 101. The client can be a social client, a video information client, etc., and the embodiments of the present application are not limited to this. Schematically, the terminal 101 can train the first image generation model through multiple object images in the above-mentioned social client, and the terminal 101 can provide AI painting services to users through the first image generation model. The terminal 101 can also regularly upload the first image generation model to the server 102, and the server 102 trains the second image generation model in the server through multiple sample images generated by the first image generation model.

[0082] Those skilled in the art will appreciate that the number of the terminals may be greater or lesser, for example, there may be only one terminal, or there may be dozens or hundreds of terminals, or a greater number. The embodiments of the present application do not limit the number of terminals and device types.

[0083] In some embodiments, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 is used to provide backend services for clients that support the above-mentioned AIGC. Server 102 can send the trained second image generation module to the client of terminal 101.

[0084] In some embodiments, the server 102 undertakes the main computing work and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture to perform collaborative computing.

[0085] Figure 2 This is a flow chart of a training method for an image generation model provided in an embodiment of the present application, such as Figure 2 As shown, in the embodiment of the present application, the training method of the image generation model is described as follows:

[0086] 201. Obtain multiple sample images generated by multiple first image generation models, where the multiple first image generation models are trained based on object images in multiple clients, respectively. The first image generation models are used to generate images based on image series information and descriptive text, and the image series information is used to represent common features of the multiple object images.

[0087] In an embodiment of the present application, multiple first image generation models are image generation models uploaded to a server by multiple clients. For any client, the client can train the first image generation model based on multiple object images in multiple image series uploaded by the user and upload the trained first image generation model to the server. After the first image generation model is trained, the first image generation model can generate a predicted image based on the input descriptive text and image series information of the image series. The predicted image includes the common features of the multiple object images described in the descriptive text and indicated by the image series information. The common features include common features of the target objects in the multiple object images and common features of the image style of the object images. The target objects can be plants, buildings, animals, people, etc., and the image style can be scene lighting, scene environment, etc. After obtaining the multiple first image generation models, the server can generate multiple sample images using the multiple first image generation models. For any first image generation model, the input of the first image model is the sample descriptive text and sample image series information. The sample descriptive text is the same as the image series information of the multiple image series in the first image generation model. The sample image series information is a random seed set by the server.

[0088] 202. For any sample image, the second image generation model processes the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image to obtain a first latent space feature, and predicts a first predicted image based on the first latent space feature. The second image generation model has the same structure as the first image generation model but different model parameters. The first latent space feature is the feature after removing the first reference noise from the global image latent space feature. The global image latent space feature is the feature generated based on the sample description text and the sample image series information. The first reference noise is the noise generated based on the sample noise text and corresponds to the sample description text.

[0089] In an embodiment of the present application, the second image generation model is located on the server and is an image generation model with the same structure as the first image generation model but different model parameters. For any sample image, the server can input the sample noise text and the sample description text and sample image series information of the sample image into the second image generation model, and the second image generation model processes the sample description text and the sample image series information to obtain the global image latent space features obtained in the processing process. The second image generation model processes the sample noise text and the sample image series information to obtain the first reference noise obtained in the processing process. Since the first reference noise is the noise corresponding to the sample noise text and the sample description text, the server can remove the first reference noise from the global image latent space features to obtain the first latent space features. Based on the first latent space features, the server decodes the first latent space features by the second image generation model to obtain a first predicted image, which includes the content described by the sample description text.

[0090] 203. The first image generation model for generating the sample image processes the sample noise text, the sample description text, and the sample image series information to obtain a second latent space feature, and predicts a second predicted image based on the second latent space feature. The second latent space feature is a feature of the sample image after removing the second reference noise. The second reference noise is noise generated based on the sample noise text and corresponds to the sample description text.

[0091] In an embodiment of the present application, for any sample image, the server can input the sample noise text and the sample description text and sample image series information of the sample image into the first image generation model that generates the sample image, and the first image generation model processes the sample description text and the sample image series information to obtain the image latent space features of the sample image obtained in the processing process. The first image generation model processes the sample noise text and the sample image series information to obtain the second reference noise obtained in the processing process. Since the second reference noise is the noise corresponding to the sample noise text and the sample description text, the server can remove the second reference noise from the image latent space features to obtain the features of the sample image after removing the second reference noise, that is, the second latent space features. Based on the second latent space features, the server decodes the second latent space features by the first image generation model to obtain a second predicted image, which includes the content described by the sample description text.

[0092] 204. Using the second predicted image as supervisory information, and based on the first predicted image and the second predicted image, train a second image generation model.

[0093] In an embodiment of the present application, the structures of the first image generation model and the second image generation model are the same, but because the training data used by the first image generation model and the second image generation model are different, the model parameters of the first image generation model and the second image generation model are different. Therefore, in the process of training the second image generation model, the sample noise text, the sample description text and the sample image series information are respectively input into the first image generation model and the second image generation model, and the obtained second predicted image and the first predicted image are different from each other. The server can use the second predicted image generated by the trained first image generation model as supervision information, and train the second image generation model based on the second predicted image and the first predicted image, so that the second image generation model after training can generate images based on the image series information and the description text. As the difference between the first predicted image and the second predicted image becomes smaller and smaller, the accuracy of the second image generation model becomes higher and higher, and the effect of generating images based on the image series information and the description text becomes better and better.

[0094] The embodiment of the present application provides a method for training an image generation model, which obtains multiple sample images generated by multiple first image generation models trained based on object images in multiple clients as training data for a second image generation model, so that the server does not need to directly obtain multiple object images from multiple clients, but indirectly learns multiple object images by learning multiple sample images. Compared with directly obtaining the gradient information of multiple first image generation models, the above method processes the sample noise text and the sample description text and sample image series information of the sample image through the second image generation module and the first image generation model respectively for any sample image, and can train the second image model based on the first predicted image and the second predicted image generated by the features after removing the reference noise. The second image generation model after training can learn more accurate image information from multiple sample images, thereby improving the accuracy of the second image generation model and protecting the privacy of the user.

[0095] Figure 3 This is a flow chart of another method for training an image generation model according to an embodiment of the present application. Figure 3 As shown, in the embodiment of the present application, the training method of the image generation model is described as follows:

[0096] 301. Generate multiple sample images using multiple first image generation models, where the multiple first image generation models are trained based on object images in multiple clients respectively. The first image generation models are used to generate images based on image series information and descriptive text, where the image series information is used to represent common features of the multiple object images.

[0097] In an embodiment of the present application, multiple first image generation models are image generation models uploaded to the server by multiple clients. For any client, the client can train the first image generation model based on multiple object images in multiple image series uploaded by the user, and upload the trained first image generation model to the server. After the training of the first image generation model is completed, the first image generation model can generate a predicted image based on the input description text and the image series information of the image series. The predicted image includes the content described in the description text and the common features of the multiple object images indicated by the image series information. Among them, the common features include the common features of the target objects in the multiple object images and the common features of the image style of the object images. The target object can be a plant, a building, an animal, a task, etc., and the image style can be scene light, scene environment, etc.

[0098] After obtaining multiple first image generation models, the server can generate multiple sample images through the multiple first image generation models. In the process of generating multiple sample images, for any first image generation model, the input of the first image model is the sample description text and the sample image series information. When the server obtains multiple first image generation models, it also obtains the image series information of the multiple image series in each image generation model. For any first image generation model, in the process of generating sample images, the server can use the image series information of the multiple image series corresponding to the first image generation model as multiple sample description texts, and associate k random seeds, that is, k sample image series information, with each sample description text. For any sample description text, the server inputs the sample description text and k random seeds into the first image generation model, and the first image generation model processes the sample description text and the k random seeds, and can predict k sample images. Among them, the k sample images correspond one to one to the k random seeds. Since the sample description text used when generating k sample images is image series information used to indicate common features of multiple object images, the k sample images generated by the first image generation model are images similar to multiple object images in the training data of the first image generation model, and the image series information of the image series to which the multiple object images belong is the same as the sample description text.

[0099] For example, Figure 4 This is a schematic diagram of an image generation interface provided by an embodiment of the present application. Figure 4 As shown, Figure 4(1) is the initial image generation interface 401, which includes a material preparation interface and a generation interface. In the material preparation interface, the user can enter the name of the image series to be created in the first control 401a, and then upload multiple images for the image series through the second control 401b. Finally, the user triggers the third control 401c. In response to the triggering operation of the third control 401c, the client can train the first image model based on multiple images. After the first image model training is completed, the first image generation model can generate images based on the image series information and the description text. In the generation interface, the user can select any image series from the multiple image series created by the user in the fourth control 401d, enter a description word, that is, a description text, in the fifth control 401e, and then trigger the sixth control 401f. In response to the triggering operation of the sixth control 401f, the client can process the input description text and the image series information corresponding to the image series by the first image generation model to generate a predicted image. The predicted image includes the common features of the multiple images indicated by the image series information and the content described by the description text. Figure 4 (2) is the image generation interface 402 displayed on the client after the user triggers the sixth control 401f. Figure 4 As shown in (2), the image series selected by the user is "Xiao Ming" and the description text input is "Xiao Ming holding a sword". The predicted image 402a generated by the first image generation model includes the features indicated by "Xiao Ming" and the content described by "Xiao Ming holding a sword".

[0100] For example, Figure 5 Schematic diagram of a first image generation model provided in an embodiment of the present application. Figure 5 As shown, the left-side input of this first image generation model is the sample image series information, namely the random seed i, and the right-side input is the sample description text. First, the random noise image X, initialized with the random seed i, is encoded by the first encoder, mapping the random noise image into the latent space. Then, the sample description text is encoded by the second encoder, mapping the sample description text into the latent space. In the latent space, the image features Z of the random noise image undergo forward diffusion to obtain the image features Zt at time T. Zt and the text features of the sample description text are repeatedly input into multiple cross-attention modules in the U-Net denoising network to predict the image features Zt-1 at time T. This cycle is repeated multiple times to ultimately predict the image features Z' at time 0. Finally, Z' is decoded by the decoder, outputting the sample image Y. By inputting the text features of the sample description text into multiple cross-attention modules in the U-Net denoising network, the generated sample image can be enriched with text information through the textual attention constraint during the prediction process.

[0101] In some embodiments, since each first image generation model includes at least one image series, the series names of image series from different first image generation models may overlap. To prevent server confusion between image series with the same series name, the server can add random numbers to distinguish between image series with the same series name. For example, if the first image generation model of a first client includes two image series named "Handsome Man" and "Old Man," and the second client includes two image series named "Beautiful Woman" and "Old Man," the server can modify the series name of the "Old Man" series in the first client to "Old Man 1" and modify the series name of the "Old Man" series in the second client to "Old Man 2."

[0102] 302. For any sample image among the multiple sample images, determine the image series to which the sample image belongs based on the sample description text of the sample image.

[0103] In this embodiment of the present application, since the sample image description texts for the sample images generated by different first image generation models are different, and the server uses the image series information of the image series in the first image generation model as the sample description text, for any sample image among the multiple sample images, the server can determine the image series to which the sample image belongs based on the sample description text of the sample image. For example, if the sample description text of the sample image is "student", the image series required for the sample image is the "student series".

[0104] 303. For any image series, determine an average value of the sharpness of multiple sample images in the image series as the sharpness of the image series.

[0105] In this embodiment of the present application, each image series includes multiple sample images. The server determines the clarity of each sample image in the image series and uses the average clarity of the multiple sample images as the clarity of the image series. The clarity of the image series can be used to reflect the average quality of the multiple sample images in the image series. The higher the clarity, the higher the average quality of the multiple sample images in the image series.

[0106] In some embodiments, for any sample image, the server uses the Laplace operator to calculate the Laplace gradient sum of each pixel in the sample image, and calculates the average value of the Laplace gradient sum of all pixels in the entire image to obtain the clarity of the sample image.

[0107] 304. Determine an average feature of the semantic features of the plurality of sample images in the image series as the semantic feature of the image series. The semantic feature of the sample image is used to indicate semantic information of the sample image.

[0108] In an embodiment of the present application, for multiple sample images in any image series, the server can use a feature extraction model to extract semantic features from the multiple sample images. The semantic features are used to represent the semantic information contained in the sample images. Based on the semantic features of the multiple sample images in the image series, the server can determine the semantic features of the image series. The semantic features of the image series are the average features of the semantic features of the multiple sample images, and are used to reflect the semantic information contained in the image series.

[0109] In some embodiments, the server can use Resnet101 (residual network model) trained based on Imagenet to extract sample features, and extract the feature vector output after the pooling layer from Resnet101 as the semantic feature of the sample image. The model structure of Resnet101 is shown in Table 1.

[0110] Table 1

[0111]

[0112]

[0113] ResNet101 consists of five convolutional layers and one pooling layer. These five convolutional layers are named Conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x. Conv1 has an output size of 224×224 and includes 64 convolution kernels of size 7×7 and stride 2. Conv2_x has an output size of 112×112 and includes a max pooling layer and three residual blocks. The max pooling layer has a pooling kernel of size 3×3 and stride 2, and each residual block includes 64 convolution kernels of size 1×1, 64 convolution kernels of size 3×3, and 256 convolution kernels of size 1×1. The output size of Conv3_x is 56×56. Conv3_x includes 4 residual blocks, each of which includes 128 1×1 convolution kernels, 128 3×3 convolution kernels, and 512 1×1 convolution kernels. The output size of Conv4_x is 28×28. Conv4_x includes 23 residual blocks, each of which includes 256 1×1 convolution kernels, 256 3×3 convolution kernels, and 1024 1×1 convolution kernels. The output size of Conv5_x is 14×14. Conv5_x includes 3 residual blocks, each of which includes 512 1×1 convolution kernels, 512 3×3 convolution kernels, and 2048 1×1 convolution kernels. The output size of the pooling layer is 1×2048. The pooling layer is a maximum pooling layer. The maximum pooling layer is a layer that extracts the point with the largest eigenvalue in the sample image as the output.

[0114] 305. Determine a plurality of definition levels to which the plurality of image series belong based on the definitions of the plurality of image series and the number of the image series.

[0115] In this embodiment of the present application, the server can classify multiple image series with similar clarity into the same clarity level based on the clarity of the multiple image series to which the multiple sample images generated by the multiple first image generation models belong, with the sum of the multiple image series in all clarity levels being equal to the number of image series. Multiple image series at a clarity level have similar quality levels; the higher the clarity level, the higher the clarity of the multiple image series at that clarity level, and the higher the average quality of the multiple sample images in the multiple image series.

[0116] In some embodiments, the server can divide the multiple image sequences into several equal parts based on the clarity ranking results of the multiple image series, and determine the clarity level based on the clarity range of the image series in each equal part. Accordingly, the multiple image series are ranked based on their clarity; based on the ranking results of the multiple image series and the number of image series, the multiple image series are equally divided to obtain multiple clarity ranges to which the multiple image series belong, and the multiple clarity ranges correspond one-to-one to multiple clarity levels; for any clarity range, the clarity level corresponding to the clarity range is determined based on the clarity range. The server can rank the multiple image series from low to high according to clarity based on the clarity of the multiple image series. Since the clarity of an image series is used to reflect the average quality of multiple sample images in the image series, the average quality of multiple sample images in the image series ranked higher in the multiple image series is lower, while the average quality of multiple sample images in the image series ranked lower in the multiple image series is higher. Based on the sorting results and the number of image series, the server can divide the multiple image series into several equal parts. Based on the clarity of the first and last image series in each part, the server can determine the clarity range of the multiple image series in each part. Each clarity range corresponds to a clarity level, with a higher clarity range corresponding to a higher clarity level.

[0117] For example, if there are 100 image series, the server will divide them into five equal parts based on the clarity ranking results, with each part containing 20 image series. The clarity ranges of the 20 image series in each part are 1-5, 6-8, 9-10, 11-14, and 15-20, respectively. These five clarity ranges correspond to clarity levels of 1, 2, 3, 4, and 5, respectively.

[0118] 306. For any definition level, cluster the multiple image series based on semantic features of the multiple image series at the definition level to obtain multiple classes to which the multiple image series belong.

[0119] In this embodiment of the present application, the number of image series in each definition level is the same. For any definition level, the server can cluster the multiple image series based on the semantic features of the multiple image series in that definition level to obtain multiple classes. For any class, a close distance between the semantic features of the multiple image series in that class indicates that the multiple sample images in the multiple image series are highly similar. Correspondingly, a long distance between the semantic features of multiple image series in different classes indicates that the multiple sample images in the multiple image series are less similar.

[0120] In some embodiments, the server can cluster multiple image series by determining the semantic features of a fourth number of image series as cluster centers. Accordingly, for any clarity level, a fourth number of image series are randomly selected from the multiple image series of the clarity level, where the fourth number is the number of classes; the semantic features of the fourth number of image series are respectively used as cluster centers of multiple classes, and the multiple image series are clustered to obtain multiple classes. The server can randomly select the semantic features of the fourth number of image series as initial cluster centers, and then calculate the distance between the semantic features of each image series and each initial cluster center, and assign each image series to the cluster center closest to it. The image series represented by the cluster center and the image series assigned to the cluster center represent a class. Each time an image series is assigned, the cluster center of each class will be re-determined based on the existing image series in the class. This process will be repeated until all image series are assigned, resulting in multiple classes to which the multiple image series belong.

[0121] For example, Figure 6 This is a schematic diagram of a sample image grading process provided by an embodiment of the present application. Figure 6 As shown, each first image generation model includes multiple image series, and the series name is the image series information of the multiple image series. The server uses the image series information as sample description text, and inputs the sample description text into each first image generation model. The first image generation model processes the sample description text and the random seed (sample image series information) to generate multiple sample images. Among them, the first image generation model 1 generates 3 sample images based on 3 series names, and the first image generation model 2 and the first image generation model 3 generate a sample image based on one series name. Then, the multiple sample images generated by the multiple image generation models are graded, and the image series to which the multiple sample images belong, the class required by the image series, and the clarity level to which the class belongs can be obtained.

[0122] In summary, the server's classification results for multiple sample images include clarity levels, classes, image series, and sample images. The multiple sample images correspond to multiple clarity levels, each clarity level includes multiple classes, each class includes multiple image series, and each image series includes multiple sample images.

[0123] 307. Based on the selection ratio, randomly select a first number of sample images from the multiple sample images as detection samples, where the first number is the product of the selection ratio and the number of the multiple sample images.

[0124] In an embodiment of the present application, the server can select a first number of sample images as detection samples from a plurality of sample images generated by a plurality of first image generation models according to a preset selection ratio, and use the first number of detection samples to train the second image generation model, where the first number is the product of the selection ratio and the number of sample images. For any image series, the server can select a target number of sample images as detection samples from the plurality of sample images in the image series based on the selection ratio, where the target number is the product of the selection ratio and the number of sample images in the image series.

[0125] For example, the plurality of sample images correspond to 100 image series, and the selection ratio is 0.3. Then, the server can randomly select 30% of the sample images from each image series as detection samples.

[0126] 308. When the second image generation model is trained for the first batch, the first number of detection samples are evenly sampled based on the multiple clarity levels to which the first number of detection samples belong to obtain the first batch of training data. The second image generation model has the same structure as the first image generation model but has different model parameters.

[0127] In an embodiment of the present application, when the second image generation model is training a first batch, the server can sample the first number of detection samples based on the grading results of the first number of detection samples selected from the plurality of sample images. The grading results include the plurality of definition levels to which the first number of detection samples belong. Based on the plurality of definition levels, the server can extract the same number of detection samples from each definition level as the first batch of training data.

[0128] In some embodiments, the grading results of the detection samples include clarity levels, classes, and sample images. The server can sample a first number of detection samples based on the grading results. Accordingly, when the second image generation model performs the first batch training, for any clarity level among the multiple clarity levels to which the first number of detection samples belong, a second number of classes are randomly selected from the multiple classes of the clarity level; for any class, a third number of sample images are randomly selected from the multiple sample images of that class; and the multiple sample images randomly selected from the multiple clarity levels serve as the first batch of training data. The first number of detection samples corresponds to multiple clarity levels, each clarity level includes multiple classes, and each class includes multiple sample images. The server can randomly extract a second number of classes from each clarity level, and then extract a third number of sample images from each class, ultimately obtaining the first batch of training data. The number of sample images in the training data is the product of the number of clarity levels, the second number, and the third number.

[0129] For example, the first number of detection samples corresponds to 5 definition levels. The server can randomly extract 2 classes from each definition level and randomly extract 1 sample image from each extracted class, thus extracting 10 sample images in total.

[0130] 309. When the second image generation model is trained for the nth batch, a first number of detection samples are sampled based on the level losses of multiple clarity levels to which multiple sample images in the n-1th batch of training data belong to obtain the nth batch of training data, where n is an integer greater than 1.

[0131] In an embodiment of the present application, when the second image generation model is trained for the n-1th batch, for any sample image in the n-1th batch of training data, the second image generation model processes the sample description text and sample image series information of the sample image to obtain a predicted image. Based on the sample image and the predicted image, the server can determine the metric loss of the sample image, which is used to indicate the difference between the sample image and the predicted image. The smaller the metric loss, the smaller the difference between the sample image and the predicted image, and the higher the accuracy of the predicted image generated by the second image generation model. Since the second image generation model has different learning abilities for different types of sample images, the generation effect of the predicted image is also uneven. For multiple clarity levels to which multiple sample images in the n-1th batch of training data belong, the server can determine the level loss of the clarity level based on the metric loss of multiple sample images in any clarity level. The level loss is used to indicate the difference between multiple sample images and multiple predicted images in the clarity level, and the level loss is positively correlated with the metric loss. Based on the level losses of the multiple clarity levels to which the multiple sample images in the n-1th batch of training data belong, the server can extract a certain proportion of sample images from the sample images at each clarity level as the training data for the nth batch. The sampling proportions of the multiple clarity levels are positively correlated with the level losses. Based on the magnitude relationship between the multiple level losses, the server can extract more sample images from clarity levels with greater level losses and fewer sample images from clarity levels with less level losses. This can increase the number of difficult samples in the training data, helping to improve the learning ability of the first human feature extraction model.

[0132] In some embodiments, based on the grading results of the sample images, the nth batch of training data includes sampled samples at multiple clarity levels. For any clarity level, the sampled samples of the clarity level include sampled samples from multiple classes within the clarity level. Accordingly, when the second image generation model performs nth batch training, the sampling probabilities of the multiple clarity levels are determined based on the level losses of the multiple clarity levels, and the level losses are positively correlated with the sampling probabilities. For any clarity level, the sampling probabilities of the multiple classes are determined based on the sampling probabilities of the clarity levels and the class losses of the multiple classes of the clarity levels. For any class, based on the sampling probabilities of the classes, multiple sample images of the class are sampled to obtain sampled samples of the class. The server can determine the sampling probabilities of the multiple clarity levels based on the loss ratios of the multiple level losses of the multiple clarity levels. The greater the loss ratio of the level loss of any clarity level compared to the level losses of other clarity levels, the higher the sampling probability of that clarity level. Each clarity level includes multiple classes. The server can further determine the sampling probability of each class in the clarity level based on the loss ratio of the class losses of multiple classes in the clarity level and the sampling probability of the clarity level. Therefore, based on the sampling probability of multiple classes, sampling samples of multiple classes are extracted from multiple classes in each clarity level. The collection of sampling samples of multiple classes is the sampling sample of the clarity level.

[0133] For example, the loss ratios for the five resolution levels are 1:1:3:2:3. The sampling probabilities for the five resolution levels, determined based on the loss ratios, are 10%, 10%, 30%, 20%, and 30%, respectively. The loss ratios for the four classes within each resolution level are 3:2:1:4. For a resolution level with a sampling probability of 10%, the sampling probabilities for the four classes within that resolution level are 3%, 2%, 1%, and 4%, respectively. The server can generate random numbers to determine which class within which resolution level to draw sample images from. If the loss ratios for the five resolution levels are 1:1:3:2:3, the value range from 0 to 1 is divided into five segments: 0 to 0.1, 0.1 to 0.2, 0.2 to 0.5, 0.5 to 0.7, and 0.7 to 1. During sampling, a random number is generated from 0 to 1. The i-th clarity level is selected based on whether the random number is in the i-th segment of the five segments. For example, if the random number is 0.5, the fourth level is selected. Within the i-th clarity level, the 0-1 range is segmented again based on the loss ratio of each class, and a random number is generated to obtain the j-th class that follows the random number. One sample image is randomly selected from this class as the training data for the batch.

[0134] In some embodiments, the server is capable of determining the level losses of multiple clarity levels to which the multiple sample images belong based on the metric losses of the multiple sample images. Accordingly, for any sample image in the n-1th batch of training data, the metric loss of the sample image is determined based on the sample image and the predicted image, where the predicted image is the image predicted by the second image generation model based on the sample description text of the sample image and the sample image series information of the sample image; for any image series, the average value of the metric losses of the multiple sample images of the image series is determined as the series loss of the image series; for any class, the average value of the series losses of the multiple image series of the class is determined as the class loss of the class; for any clarity level, the average value of the class losses of the multiple classes of the clarity level is determined as the level loss of the clarity level. The metric loss is used to indicate the difference between the sample image and the predicted image. The smaller the metric loss, the smaller the difference between the sample image and the predicted image, and the higher the accuracy of the predicted image generated by the second image generation model. To determine the accuracy of the predicted images generated by the second image generation model after processing sample images from different classification levels, for any image series, the server can determine the series loss of the image series as the average of the metric losses of multiple sample images in the image series. The series loss indicates the difference between the multiple sample images in the image series and the multiple predicted images. The smaller the series loss, the stronger the second image generation model's ability to learn the sample images in the image series, and the better the predicted image generation. For any class, the server can determine the class loss of the class as the average of the metric losses of multiple image series for that class. The class loss indicates the difference between the multiple sample images in that class and the multiple predicted images. The smaller the class loss, the stronger the second image generation model's ability to learn the sample images in the image series for that class, and the better the predicted image generation. For any definition level, the server can determine the class loss of the definition level as the average of the class losses of multiple classes within that definition level. The class loss indicates the difference between the multiple sample images in that definition level and the multiple predicted images.

[0135] For example, Figure 7 This is a schematic diagram of a sample image sampling process provided by an embodiment of the present application. Figure 7As shown, during the first batch of training, the server evenly samples multiple detection samples based on the multiple clarity levels to which they belong, obtaining the first batch of training data, and trains the second image generation model to obtain multiple predicted images. The metric loss of the sample images is then determined based on the multiple sample images and predicted images generated by the multiple first image generation models in the first batch of training data. The losses at each level are then obtained based on the grading results of the first batch of training data, reflecting the generation effect of the predicted images at each level. The losses at each level include series losses for image series, class losses for classes, and class losses for clarity levels. The server then resamples the multiple detection samples based on the losses at each level, and uses the sampled sample images as the training data for the next batch.

[0136] 310. For any sample image among the multiple sample images in the current batch, the second image generation model processes the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image to obtain a first latent space feature, and predicts a first predicted image based on the first latent space feature. The first latent space feature is the feature after removing the first reference noise from the global image latent space feature. The global image latent space feature is the feature generated based on the sample description text and the sample image series information. The first reference noise is the noise generated based on the sample noise text and corresponds to the sample description text.

[0137] In an embodiment of the present application, the server can perform multiple batches of training on the second image generation model based on the above-mentioned multiple batches of training data. In the current batch training, for any sample image in the batch of training data, the server can input the sample noise text and the sample description text and sample image series information of the sample image into the second image generation model, and the second image generation model processes the sample description text and the sample image series information to obtain the global image latent space features obtained in the processing process. The second image generation model processes the sample noise text and the sample image series information to obtain the first reference noise obtained in the processing process. Since the first reference noise is the noise corresponding to the sample noise text and the sample description text, the server can remove the first reference noise from the global image latent space features to obtain the first latent space features. Based on the first latent space features, the server decodes the first latent space features by the second image generation model to obtain a first predicted image, which includes the content described by the sample description text.

[0138] In some embodiments, the server can process the text pairs consisting of the sample description text and the sample image series information and the text pairs consisting of the sample noise text and the sample image series information through the second image generation model to obtain a first predicted image. Accordingly, for any sample image, the second image generation model processes the sample description text and the sample image series information to obtain a global image latent space feature; the second image generation model processes the sample noise text and the sample image series information to obtain a first noise latent space feature, which is used to represent the first reference noise; the global image latent space feature and the first noise latent space feature are subtracted to obtain the first latent space feature, and the first predicted image is predicted based on the first latent space feature. For any sample image, the server inputs the sample description text and the sample image series information of the sample image into the second image generation model. The second image generation model can map the sample description text and the sample image series information to the latent space through the encoder, and obtain the global image latent space feature generated based on the text features of the sample description text and the image features of the sample image series information from the latent space. Similarly, the server inputs the sample noise text and sample image series information into the second image generation model. The second image generation model can map the sample noise text and sample image series information to the latent space respectively through the encoder, and obtain the first noise latent space feature generated based on the text features of the sample noise text and the image features of the sample image series information from the latent space. The first noise latent space feature is used to represent the noise corresponding to the sample noise text and the sample description text, that is, the first reference noise. By subtracting the global image latent space feature from the first noise latent space feature, the server can obtain the first latent space feature after removing the first reference noise from the global image latent space feature.

[0139] For example, since the structure of the second image generation model is consistent with the first image generation model, the structure of the second image generation model is consistent with the first image generation model. Figure 5 The structure of the first image generation model shown in FIG is the same as that of the first image generation model shown in FIG. Figure 5As shown, the second image generation model first inputs a sample image sequence, i.e., a random seed i, and a sample description text on the left. The first encoder encodes the random noise image initialized with the random seed i, mapping the random noise image into the latent space. The sample description text is then encoded by the second encoder, mapping the sample description text into the latent space. In the latent space, the image feature Z of the random noise image undergoes forward diffusion to obtain the image feature Zt at time T. Zt and the text features of the sample description text are repeatedly input into multiple cross-attention modules within the U-Net denoising network, predicting the image feature Zt-1 at time T. This cycle is repeated multiple times to ultimately predict the image feature Z' at time 0, which is the global image latent space feature. Then, the second image generation model, when inputted from the left side is a sample image sequence, i.e., a random seed i, and from the right side is a sample noise text, undergoes this process to obtain the first noise latent space feature. The server subtracts the global image latent space features from the first noise latent space features to obtain the first latent space features after removing the first reference noise from the global image latent space features. Finally, the decoder decodes the first latent space features and inputs the first predicted image. The sample noise text is empty, meaning that the sample noise text is input to the second encoder on the right side of the model. The extracted feature vector of the sample noise text is a 0 vector.

[0140] 311. The first image generation model for generating the sample image processes the sample noise text, the sample description text, and the sample image series information to obtain a second latent space feature, and predicts a second predicted image based on the second latent space feature. The second latent space feature is the feature of the sample image after removing the second reference noise. The second reference noise is noise generated based on the sample noise text and corresponds to the sample description text.

[0141] In an embodiment of the present application, for any sample image, the server can input the sample noise text and the sample description text and sample image series information of the sample image into the first image generation model that generates the sample image, and the first image generation model processes the sample description text and the sample image series information to obtain the image latent space features of the sample image obtained in the processing process. The first image generation model processes the sample noise text and the sample image series information to obtain the second reference noise obtained in the processing process. Since the second reference noise is the noise corresponding to the sample noise text and the sample description text, the server can remove the second reference noise from the image latent space features to obtain the features of the sample image after removing the second reference noise, that is, the second latent space features. Based on the second latent space features, the server decodes the second latent space features by the first image generation model to obtain a second predicted image, which includes the content described by the sample description text.

[0142] In some embodiments, the server can process the text pairs consisting of the sample description text and the sample image series information and the text pairs consisting of the sample noise text and the sample image series information, respectively, through the first image generation model to obtain a second predicted image. Accordingly, for any sample image, the first image generation model processes the sample description text and the sample image series information to obtain image latent space features, which are global features of the sample image; the first image generation model processes the sample noise text and the sample image series information to obtain second noise latent space features, which are used to represent the second reference noise; the image latent space features and the second noise latent space features are subtracted to obtain second latent space features, and the second predicted image is predicted based on the second latent space features. For any sample image, the server inputs the sample description text and the sample image series information of the sample image into the first image generation model that generates the sample image. The first image generation model can map the sample description text and the sample image series information to the latent space through an encoder, and obtain image latent space features generated from the latent space based on the text features of the sample description text and the image features of the sample image series information, i.e., the global features of the sample image. Similarly, the server inputs the sample noise text and sample image series information into the first image generation model. The first image generation model maps the sample noise text and sample image series information to the latent space respectively through the encoder, and obtains the second noise latent space features generated from the latent space based on the text features of the sample noise text and the image features of the sample image series information. The second noise latent space features are used to represent the noise corresponding to the sample noise text and the sample description text, that is, the second reference noise. By subtracting the image latent space features from the second noise latent space features, the server can obtain the second latent space features after removing the second reference noise from the image latent space features.

[0143] 312. The third image generation model processes the sample noise text and the sample image series information to obtain a third noise latent space feature. The third image generation model is an initial image generation model. The second image generation model is trained by the third image generation model. The third noise latent space feature is a feature generated based on the sample noise text and the sample image series information.

[0144] In an embodiment of the present application, the third image generation model is the initial image generation model. The server can send the third image generation model to multiple clients. The multiple clients train the third image generation model based on multiple object images and obtain multiple first image generation models. The multiple clients upload the trained multiple first image generation models to the server. The server can train the third image generation model based on multiple sample images generated by the multiple first image generation models to obtain the second image generation model. The model structures of the first image generation model, the second image generation model and the third image generation model are consistent, but due to the different training data, the model parameters are different. In the process of training the second image generation model, for any sample image, the server can input the sample noise text and the sample image series information of the sample image into the third image generation model. The third image generation model maps the sample noise text and the sample image series information to the latent space respectively through the encoder, and obtains the third noise latent space features generated based on the text features of the sample noise text and the image features of the sample image series information from the latent space.

[0145] 313. The second image generation model processes the sample noise text and the sample image series information to obtain a fourth noise latent space feature, where the fourth noise latent space feature is a feature generated based on the sample noise text and the sample image series information.

[0146] In an embodiment of the present application, during the training of the second image generation model, for any sample image, the server can input the sample noise text and the sample image series information of the sample image into the second image generation model. The second image generation model uses an encoder to map the sample noise text and the sample image series information to a latent space, and obtains a fourth noise latent space feature generated from the latent space based on the text features of the sample noise text and the image features of the sample image series information. Optionally, the fourth noise latent space feature is the same as the first noise latent space feature.

[0147] 314. Using the second predicted image and the third noise latent space features as supervisory information, train a second image generation model based on the first predicted image, the second predicted image, the third noise latent space features, and the fourth noise latent space features.

[0148] In an embodiment of the present application, the structures of the first image generation model, the second image generation model, and the third image generation model are the same, but due to the different training data used, the model parameters of the first image generation model, the second image generation model, and the third image generation model are different. Therefore, in the process of training the second image generation model, the sample noise text, the sample description text, and the sample image series information are respectively input into the first image generation model and the second image generation model, and the second predicted image and the first predicted image obtained are different from each other. Similarly, the sample noise text and the sample image series information are respectively input into the third image generation model and the second image generation model, and the third noise latent space features and the fourth noise latent space features obtained are different from each other. The server can use the second predicted image and the third noise latent space features as supervision information, and train the second image generation model based on the first predicted image, the second predicted image, the third noise latent space features, and the fourth noise latent space features. While improving the accuracy of the image generated by the second image generation model, it can also improve the generalization ability of the second image generation model.

[0149] In some embodiments, the server can update the model parameters of the second image generation model based on the training loss determined based on the first predicted image, the second predicted image, the third noise latent space feature, and the fourth noise latent space feature. Accordingly, a first loss is determined based on the first predicted image and the second predicted image, indicating the difference between the first predicted image and the second predicted image; a second loss is determined based on the third noise latent space feature and the fourth noise latent space feature, indicating the difference between the third noise latent space feature and the fourth noise latent space feature; a weighted sum of the first and second losses is performed to obtain the training loss of the second image generation model; and the model parameters of the second image generation model are updated based on the training loss. The smaller the first loss, the smaller the difference between the first and second predicted images, and the higher the accuracy of the predicted images generated by the second image generation model. The smaller the second loss, the smaller the difference between the third and fourth noise latent space features, and the more generalized global information can be learned by the second image generation model. Based on the training loss obtained by weighted summation of the first loss and the second loss, the model parameters of the second image generation model are updated. This can improve the accuracy of image generation by the second image generation model while also improving the generalization ability of the second image generation model. It can also avoid the impact of conflicting information between multiple first image models on the global generalization of the second image generation model when multiple first image models guide the training of the second image generation model.

[0150] In some embodiments, to improve the accuracy of the first and second predicted images, the server can extract features of the target object from the first and second predicted images, respectively. Accordingly, object detection is first performed on the first and second predicted images, determining a first region and a second region. The first region is the region where the target object is located in the first predicted image, and the second region is the region where the target object is located in the second predicted image. Feature extraction is then performed on the first predicted image based on the first region to obtain first object latent space features, which are features of the target object in the first predicted image. Feature extraction is then performed on the second predicted image based on the second region to obtain second object latent space features, which are features of the target object in the second predicted image. Finally, the server can use the second object latent space features and the third noise latent space features as supervisory information to train a second image generation model based on the first object latent space features, the second object latent space features, the third noise latent space features, and the fourth noise latent space features. For any sample image, each sample image contains a target object. For example, if the sample image is a person image, the target object is a person; if the sample image is an object image, the target object is an object. The first image generation model and the second image generation model generate the first predicted image and the second predicted image based on the sample noise text, the sample description text of the sample image, and the sample image series information, which also include the target object. The server performs object detection on the first predicted image and the second predicted image, respectively, and can determine the area where the target object is located in the first predicted image and the area where the target object is located in the second predicted image, that is, the first area and the second area. Based on the first area and the second area, feature extraction is performed on the first predicted image and the second predicted image, respectively, to obtain the first object latent space feature and the second object latent space feature. The server trains the second image generation model based on the extracted more accurate first object latent space feature and second object latent space feature together with the third noise latent space feature and the fourth noise latent space feature, which can further improve the accuracy of the image generated by the second image generation model.

[0151] In some embodiments, the first loss and the second loss are both mean square error losses, and the server can determine the first loss and the second loss respectively through the following formula 1.

[0152] Formula 1:

[0153]

[0154] Among them, when loss is the first loss, z is the first object latent space feature, and y is the second object latent space feature; when loss is the second loss, z is the fourth noise latent space feature, and y is the third noise latent space feature.

[0155] In some embodiments, the server performs a weighted summation of the first loss and the second loss using the following formula 2 to obtain the training loss of the second image generation model.

[0156] Formula 2:

[0157] loss = loss1 + a × loss2

[0158] Here, loss1 is the first loss, and loss2 is the second loss. Because loss2 is a generalization loss, it corresponds to the entire image and generates a much larger loss than loss1. Therefore, to ensure that the latent space features in loss1 can be fed back to the second image generation model, the scaling factor a is set to a small value. A value of 0.001 is acceptable.

[0159] For example, Figure 8 This is a training block diagram of a second image generation model provided according to an embodiment of the present application. Figure 8 As shown, for any sample image, the random seed x is the sample image series information of the sample image, and the series name is the sample description text of the sample image. The server first uses (random seed x, series name) and (random seed x, noise word) through the second image generation model to obtain the global image latent space representation and the first noise latent space feature. Then, the server predicts the first predicted image based on the global image latent space representation and the first noise latent space feature. Then, the server uses (random seed x, series name) and (random seed x, noise word) through the first image generation model to obtain the image latent space representation and the second noise latent space feature. Then, the server predicts the second predicted image based on the global image latent space representation and the second noise latent space feature. The expert information is the feature information of the target object in the second predicted image, and the expert prediction information is the feature information of the target object in the first predicted image. The random seed x and the noise word are then passed through the third and second image generation models to obtain third and fourth noise latent space features. The non-expert information is the global image information of the noise image encoded using the random seed x extracted by the third image generation model, and the non-expert prediction information is the global image prediction information of the noise image encoded using the random seed x extracted by the second image generation model. The server determines a first loss based on the first and second predicted images, and a second loss based on the third and fourth noise latent space features. The server then performs a weighted sum of the first and second losses to obtain the training loss of the second image generation model. The model parameters of the second image generation model are updated based on the training loss. Figure 9 is a training block diagram of another second image generation model provided according to an embodiment of the present application, Figure 8In comparison, after obtaining the first predicted image based on the global image latent space representation and the first noise latent space feature prediction, the server can perform object detection on the first predicted image to determine the region where the target object is located, thereby extracting more accurate expert prediction information, namely, the feature information of the target object in the first predicted image. Similarly, the server extracts more accurate expert information from the second predicted image, namely, the feature information of the target object in the second predicted image, and determines the first loss based on the feature information of the target object in the first predicted image and the feature information of the target object in the second predicted image.

[0160] In order to more intuitively understand the process of using the first image generation model of multiple clients to guide the second image generation model of the server proposed in the embodiment of the present application, the embodiment of the present application provides a schematic diagram of the interaction between multiple clients and the server. Figure 10 As shown, the embodiment of the present application includes multiple clients, and each user uses one client. In the client, the user first prepares the materials according to the interface prompts displayed by the client, that is, the user creates an image series and uploads the images corresponding to the image series. For example, a user creates an image series and uploads multiple avatars of himself, then the series "Myself" represents the user's own avatar. When the user wants to create a avatar for his child, he can create a new image series "Baby" and upload multiple avatars belonging to the child. Each time the client collects an image series, it trains the first image generation model in the client. The training process is not shown in the figure. The first image generation model of the client in the figure is a trained model. Multiple clients can upload multiple first image generation models to the server regularly (such as during service maintenance every morning). The server uses multiple first image generation models to guide the training of the second image model. The training process is as follows: Figure 8 and Figure 9 As shown. The trained second image model is then sent to each client, and each client replaces the first image generation model with the second image generation model. The client then trains the first image generation model in the client again based on the collected image series, and then submits the first image generation model to the server again, repeating the above process. For privacy protection, multiple clients will not upload the images of the multiple image series uploaded by users to the server to train the second image generation model. Instead, they use the image generation model training method provided by this application to migrate the expert information in the multiple first image generation models to the second image model in the absence of the multiple image series uploaded by users.

[0161] The embodiment of the present application provides a method for training an image generation model, which obtains multiple sample images generated by multiple first image generation models trained based on object images in multiple clients as training data for a second image generation model, so that the server does not need to directly obtain multiple object images from multiple clients, but indirectly learns multiple object images by learning multiple sample images. Compared with directly obtaining the gradient information of multiple first image generation models, the above method processes the sample noise text and the sample description text and sample image series information of the sample image through the second image generation module and the first image generation model respectively for any sample image, and can train the second image model based on the first predicted image and the second predicted image generated by the features after removing the reference noise. The second image generation model after training can learn more accurate image information from multiple sample images, thereby improving the accuracy of the second image generation model and protecting the privacy of the user.

[0162] Figure 11 This is a block diagram of a training device for an image generation model according to an embodiment of the present application. The device is used to perform the training method for the above-mentioned image generation model, see Figure 11 The device includes: an acquisition module 1101, a first prediction module 1102, a second prediction module 1103 and a training module 1104.

[0163] An acquisition module 1101 is configured to acquire a plurality of sample images generated by a plurality of first image generation models, wherein the plurality of first image generation models are trained based on object images from a plurality of clients, the first image generation models being configured to generate images based on image series information and descriptive text, wherein the image series information is configured to represent common features of the plurality of object images;

[0164] A first prediction module 1102 is configured to, for any sample image, process the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image using a second image generation model to obtain a first latent space feature, and predict a first predicted image based on the first latent space feature. The second image generation model has the same structure as the first image generation model but has different model parameters. The first latent space feature is the feature obtained by removing the first reference noise from the global image latent space feature. The global image latent space feature is the feature generated based on the sample description text and the sample image series information. The first reference noise is the noise generated based on the sample noise text and corresponding to the sample description text.

[0165] A second prediction model 1103 is configured to process the sample noise text, the sample description text, and the sample image series information using the first image generation model to obtain second latent space features, and predict a second predicted image based on the second latent space features, where the second latent space features are features of the sample image after removing a second reference noise, and the second reference noise is noise generated based on the sample noise text and corresponding to the sample description text;

[0166] The training module 1104 uses the second predicted image as supervisory information and trains a second image generation model based on the first predicted image and the second predicted image.

[0167] In some embodiments, Figure 12 This is a block diagram of another image generation model training device provided according to an embodiment of the present application. Figure 12 The first prediction module 1102 includes:

[0168] The first processing unit 1201 is configured to process the sample description text and the sample image series information by the second image generation model for any sample image to obtain a global image latent space feature;

[0169] The second processing unit 1202 is configured to process the sample noise text and the sample image series information using a second image generation model to obtain a first noise latent space feature, where the first noise latent space feature is used to represent a first reference noise;

[0170] The third processing unit 1203 is configured to subtract the global image latent space feature from the first noise latent space feature to obtain a first latent space feature, and predict a first predicted image based on the first latent space feature.

[0171] In some embodiments, continued participation Figure 12 The second prediction module 1103 includes:

[0172] The fourth processing unit 1204 is configured to process the sample description text and the sample image series information using the first image generation model for any sample image to obtain an image latent space feature, where the image latent space feature is a global feature of the sample image.

[0173] a fifth processing unit 1204 configured to process the sample noise text and the sample image series information using the first image generation model to obtain a second noise latent space feature, where the second noise latent space feature is used to represent a second reference noise;

[0174] The sixth processing unit 1206 is configured to subtract the image latent space feature from the second noise latent space feature to obtain the second latent space feature, and predict the second predicted image based on the second latent space feature.

[0175] In some embodiments, the training module 1104 is used to perform object detection on the first predicted image and the second predicted image respectively, determine a first area and a second area, the first area is the area where the target object is located in the first predicted image, and the second area is the area where the target object is located in the second predicted image; based on the first area, perform feature extraction on the first predicted image to obtain a first object latent space feature, which is the feature of the target object in the first predicted image; based on the second area, perform feature extraction on the second predicted image to obtain a second object latent space feature, which is the feature of the target object in the second predicted image; and use the second object latent space feature as supervision information to train a second image generation model based on the first object latent space feature and the second object latent space feature.

[0176] In some embodiments, see Figure 12 , the device further comprises:

[0177] A first processing module 1105 is configured to process the sample noise text and the sample image series information using a third image generation model to obtain a third noise latent space feature, wherein the third image generation model is an initial image generation model, the second image generation model is trained by the third image generation model, and the third noise latent space feature is a feature generated based on the sample noise text and the sample image series information;

[0178] A second processing module 1106 is configured to process the sample noise text and the sample image series information using a second image generation model to obtain a fourth noise latent space feature, where the fourth noise latent space feature is a feature generated based on the sample noise text and the sample image series information;

[0179] The training module 1104 is configured to train a second image generation model based on the first predicted image, the second predicted image, the third noise latent space features, and the fourth noise latent space features, using the second predicted image and the third noise latent space features as supervisory information.

[0180] In some embodiments, the training module 1104 is used to determine a first loss based on the first predicted image and the second predicted image, where the first loss is used to indicate the difference between the first predicted image and the second predicted image; determine a second loss based on the third noise latent space feature and the fourth noise latent space feature, where the second loss is used to indicate the difference between the third noise latent space feature and the fourth noise latent space feature; perform weighted summation of the first loss and the second loss to obtain the training loss of the second image generation model; and update the model parameters of the second image generation model based on the training loss.

[0181] In some embodiments, see Figure 12 , the device further comprises:

[0182] A selection module 1107 is configured to randomly select a first number of sample images from the plurality of sample images as detection samples based on a selection ratio, where the first number is a product of the selection ratio and the number of the plurality of sample images;

[0183] A first sampling module 1108 is configured to perform balanced sampling on the first number of detection samples based on the multiple clarity levels to which the first number of detection samples belong, to obtain a first batch of training data, when the second image generation model is trained for the first batch;

[0184] The second sampling module 1109 is used to sample the first number of detection samples based on the level loss of multiple clarity levels belonging to multiple sample images in the n-1th batch of training data when the second image generation model is trained in the nth batch, to obtain the nth batch of training data, where n is an integer greater than 1.

[0185] In some embodiments, the first sampling module 1108 is used to randomly select a second number of classes from multiple classes of the clarity level for any clarity level among the multiple clarity levels to which the first number of detection samples belong; randomly select a third number of sample images from multiple sample images of the class for any class; and use the multiple sample images randomly selected from multiple clarity levels as training data for the first batch.

[0186] In some embodiments, the nth batch of training data includes sampled samples of multiple definition levels, and for any definition level, the sampled samples of the definition level include sampled samples of multiple classes in the definition level;

[0187] The second sampling module 1109 is used to determine the sampling probabilities of multiple clarity levels based on the level losses of multiple clarity levels when the second image generation model is trained for the nth batch, where the level loss is positively correlated with the sampling probability; for any clarity level, determine the sampling probabilities of multiple classes based on the sampling probability of the clarity level and the class losses of multiple classes of the clarity level; for any class, sample multiple sample images of the class based on the sampling probability of the class to obtain sampling samples of the class.

[0188] In some embodiments, see Figure 12 , the device further comprises:

[0189] a loss determining module 1110 configured to determine, for any sample image in the (n-1)th batch of training data, a metric loss of the sample image based on the sample image and a predicted image, where the predicted image is an image predicted by the second image generation model based on the sample description text of the sample image and the sample image series information of the sample image, and the metric loss is used to indicate the difference between the sample image and the predicted image;

[0190] The loss determination module 1110 is further configured to determine, for any image series, an average value of the metric losses of a plurality of sample images of the image series as the series loss of the image series;

[0191] The loss determination module 1110 is further configured to determine, for any class, an average of series losses of multiple image series of the class as the class loss of the class;

[0192] The loss determining module 1110 is further configured to determine, for any definition level, an average value of class losses of multiple classes of the definition level as the level loss of the definition level.

[0193] In some embodiments, see Figure 12 , the device further comprises:

[0194] A determination module 1111 is configured to determine, for any sample image among the plurality of sample images, the image series to which the sample image belongs based on the sample description text of the sample image;

[0195] The determination module 1111 is further configured to determine, for any image series, an average value of the sharpness of a plurality of sample images in the image series as the sharpness of the image series;

[0196] The determining module 1111 is further configured to determine an average feature of semantic features of a plurality of sample images in the image series as the semantic feature of the image series, wherein the semantic feature of the sample image is used to indicate semantic information of the sample image;

[0197] a level determination module 1112 for determining a plurality of clarity levels to which the plurality of image series belong based on the clarity of the plurality of image series and the number of image series;

[0198] The clustering module 1113 is configured to cluster the multiple image series at any definition level based on the semantic features of the multiple image series at the definition level to obtain multiple classes to which the multiple image series belong.

[0199] In some embodiments, the level determination module 1112 is used to sort multiple image series based on the clarity of the multiple image series; based on the sorting results of the multiple image series and the number of image series, the multiple image series are evenly divided to obtain multiple clarity ranges to which the multiple image series belong, and the multiple clarity ranges correspond one-to-one to multiple clarity levels; for any clarity range, based on the clarity range, the clarity level corresponding to the clarity range is determined.

[0200] In some embodiments, the clustering module 1113 is used to randomly select a fourth number of image series from multiple image series of the clarity level for any clarity level, where the fourth number is the number of classes; and use the semantic features of the fourth number of image series as clustering centers of multiple classes, clustering the multiple image series to obtain multiple classes.

[0201] The embodiment of the present application provides a training device for an image generation model. By obtaining multiple sample images generated by multiple first image generation models trained based on object images in multiple clients as training data for a second image generation model, the server does not need to directly obtain multiple object images from multiple clients, but indirectly learns multiple object images by learning multiple sample images. Compared with directly obtaining the gradient information of multiple first image generation models, the above method processes the sample noise text and the sample description text and sample image series information of the sample image through the second image generation module and the first image generation model respectively for any sample image, and can train the second image model based on the first predicted image and the second predicted image generated by the features after removing the reference noise. The second image generation model after training can learn more accurate image information from multiple sample images, thereby improving the accuracy of the second image generation model and protecting the privacy of the user.

[0202] It should be noted that the image generation model training device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal can be divided into different functional modules to complete all or part of the functions described above. In addition, the image generation model training device provided in the above embodiment and the image generation model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0203] In the embodiments of the present application, the computer device can be configured as a terminal or a server. When the computer device is configured as a server, the server can be used as the execution subject to implement the technical solution provided in the embodiments of the present application. When the computer device is configured as a terminal, the terminal can be used as the execution subject to implement the technical solution provided in the embodiments of the present application. Of course, the technical solution provided in the embodiments of the present application can also be implemented collaboratively by the service and the terminal, and the embodiments of the present application do not limit this.

[0204] Figure 13 This is a structural block diagram of a terminal 1300 provided according to an embodiment of the present application.

[0205] Typically, the terminal 1300 includes a processor 1301 and a memory 1302 .

[0206] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0207] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one computer program, which is used to be executed by the processor 1301 to implement the training method of the image generation model provided in the method embodiment of the present application.

[0208] In some embodiments, terminal 1300 may optionally include a peripheral device interface 1303 and at least one peripheral device. Processor 1301, memory 1302, and peripheral device interface 1303 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1308.

[0209] The peripheral device interface 1303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0210] The RF circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the RF circuit 1304 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 1304 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1304 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0211] The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch screen display, the display screen 1305 also has the ability to collect touch signals on the surface or above the surface of the display screen 1305. The touch signal can be input as a control signal to the processor 1301 for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 1305, which is set on the front panel of the terminal 1300; in other embodiments, there can be at least two display screens 1305, which are respectively set on different surfaces of the terminal 1300 or in a folding design; in other embodiments, the display screen 1305 can be a flexible display screen, which is set on the curved surface or folding surface of the terminal 1300. Even more, the display screen 1305 can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0212] The camera assembly 1306 is used to capture images or videos. In some embodiments, the camera assembly 1306 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0213] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 1301 for processing, or input into the radio frequency circuit 1304 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 1300. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0214] Power supply 1308 is used to power various components in terminal 1300. Power supply 1308 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1308 includes a rechargeable battery, the rechargeable battery can be wired or wirelessly rechargeable. A wired rechargeable battery is charged via a wired line, while a wireless rechargeable battery is charged via a wireless coil. The rechargeable battery can also support fast charging technology.

[0215] In some embodiments, the terminal 1300 further includes one or more sensors 1309 , including but not limited to: an acceleration sensor 1310 , a gyroscope sensor 1311 , a pressure sensor 1312 , an optical sensor 1313 , and a proximity sensor 1314 .

[0216] The accelerometer 1310 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 1300. For example, the accelerometer 1310 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 1301 can control the display screen 1305 to display the user interface in either a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1310. The accelerometer 1310 can also be used to collect game or user motion data.

[0217] The gyroscope sensor 1311 can detect the orientation and rotation angle of the terminal 1300. It can also work with the accelerometer 1310 to collect the user's 3D movements on the terminal 1300. Based on the data collected by the gyroscope sensor 1311, the processor 1311 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0218] The pressure sensor 1312 can be provided on the side frame of the terminal 1300 and / or below the display screen 1305. When the pressure sensor 1312 is provided on the side frame of the terminal 1300, it can detect the user's gripping signal of the terminal 1300. The processor 1301 then performs left-hand or right-hand recognition or shortcut operations based on the gripping signal collected by the pressure sensor 1312. When the pressure sensor 1312 is provided below the display screen 1305, the processor 1301 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1305. Operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0219] Optical sensor 1313 is used to detect ambient light intensity. In one embodiment, processor 1301 can control the display brightness of display screen 1305 based on the ambient light intensity detected by optical sensor 1313. Specifically, when the ambient light intensity is high, the display brightness of display screen 1305 is increased; when the ambient light intensity is low, the display brightness of display screen 1305 is decreased. In another embodiment, processor 1301 can also dynamically adjust the shooting parameters of camera assembly 1306 based on the ambient light intensity detected by optical sensor 1313.

[0220] Proximity sensor 1314, also known as a distance sensor, is typically located on the front panel of terminal 1300. Proximity sensor 1314 is used to measure the distance between the user and the front of terminal 1300. In one embodiment, when proximity sensor 1314 detects that the distance between the user and the front of terminal 1300 is gradually decreasing, processor 1301 controls display screen 1305 to switch from the screen-on state to the screen-off state. When proximity sensor 1304 detects that the distance between the user and the front of terminal 1300 is gradually increasing, processor 1301 controls display screen 1305 to switch from the screen-off state to the screen-on state.

[0221] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0222] Figure 141 is a schematic diagram of the structure of a server provided according to an embodiment of the present application. The server 1400 may vary significantly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 1401 and one or more memories 1402. The memories 1402 store at least one computer program, which is loaded and executed by the processor 1401 to implement the image generation model training methods provided in the above-mentioned various method embodiments. Of course, the server may also include other components for implementing device functions, which will not be described in detail here.

[0223] The present application also provides a computer-readable storage medium that stores at least one computer program, which is loaded and executed by a processor to implement the image generation model training method described in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0224] An embodiment of the present application also provides a computer program product, including a computer program product, which is executed by a processor to implement the training method of the image generation model in the above embodiment.

[0225] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0226] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Acquire a plurality of sample images generated by a plurality of first image generation models, the plurality of first image generation models being trained based on object images from a plurality of clients, the first image generation models being configured to generate images based on image series information and descriptive text, the image series information being configured to represent common features of the plurality of object images; For any sample image, the second image generation model processes the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image to obtain a first latent space feature, and predicts a first predicted image based on the first latent space feature. The second image generation model has the same structure as the first image generation model but has different model parameters. The first latent space feature is the feature obtained by removing the first reference noise from the global image latent space feature. The global image latent space feature is the feature generated based on the sample description text and the sample image series information. The first reference noise is the noise generated based on the sample noise text and corresponds to the sample description text. The first image generation model that generates the sample image processes the sample noise text, the sample description text, and the sample image series information to obtain a second latent space feature, and predicts a second predicted image based on the second latent space feature, where the second latent space feature is a feature of the sample image after removing a second reference noise, and the second reference noise is noise generated based on the sample noise text and corresponding to the sample description text; The second predicted image is used as supervisory information and the second image generation model is trained based on the first predicted image and the second predicted image.

2. The method according to claim 1, characterized in that For any sample image, the second image generation model processes the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image to obtain a first latent space feature, and predicts a first predicted image based on the first latent space feature, including: For any sample image, the second image generation model processes the sample description text and the sample image series information to obtain the global image latent space feature; The second image generation model processes the sample noise text and the sample image series information to obtain a first noise latent space feature, where the first noise latent space feature is used to represent the first reference noise; The global image latent space feature and the first noise latent space feature are subtracted to obtain the first latent space feature, and the first predicted image is predicted based on the first latent space feature.

3. The method according to claim 1, characterized in that The first image generation model that generates the sample image processes the sample noise text, the sample description text, and the sample image series information to obtain a second latent space feature, and predicts a second predicted image based on the second latent space feature, including: For any sample image, the first image generation model processes the sample description text and the sample image series information to obtain image latent space features, where the image latent space features are global features of the sample image; The first image generation model processes the sample noise text and the sample image series information to obtain a second noise latent space feature, where the second noise latent space feature is used to represent the second reference noise; The image latent space feature and the second noise latent space feature are subtracted to obtain the second latent space feature, and the second predicted image is predicted based on the second latent space feature.

4. The method according to claim 1, wherein The step of training the second image generation model based on the first predicted image and the second predicted image using the second predicted image as supervisory information includes: performing object detection on the first predicted image and the second predicted image respectively to determine a first region and a second region, wherein the first region is a region where a target object is located in the first predicted image, and the second region is a region where the target object is located in the second predicted image; Based on the first region, feature extraction is performed on the first predicted image to obtain a first object latent space feature, where the first object latent space feature is a feature of the target object in the first predicted image; Based on the second region, feature extraction is performed on the second predicted image to obtain a second object latent space feature, where the second object latent space feature is a feature of the target object in the second predicted image; The second object latent space features are used as supervisory information and the second image generation model is trained based on the first object latent space features and the second object latent space features.

5. The method according to claim 1, characterized in that The method further comprises: The sample noise text and the sample image series information are processed by a third image generation model to obtain a third noise latent space feature, wherein the third image generation model is an initial image generation model, the second image generation model is trained by the third image generation model, and the third noise latent space feature is a feature generated based on the sample noise text and the sample image series information; The second image generation model processes the sample noise text and the sample image series information to obtain a fourth noise latent space feature, where the fourth noise latent space feature is a feature generated based on the sample noise text and the sample image series information; The step of training the second image generation model based on the first predicted image and the second predicted image using the second predicted image as supervisory information includes: The second predicted image and the third noise latent space features are used as supervisory information, and the second image generation model is trained based on the first predicted image, the second predicted image, the third noise latent space features, and the fourth noise latent space features.

6. The method according to claim 5, characterized in that The method of training the second image generation model based on the first predicted image, the second predicted image, the third noise latent space feature, and the fourth noise latent space feature using the second predicted image and the third noise latent space feature as supervisory information includes: determining a first loss based on the first predicted image and the second predicted image, the first loss being used to indicate a difference between the first predicted image and the second predicted image; determining a second loss based on the third noise latent space feature and the fourth noise latent space feature, wherein the second loss is used to indicate a difference between the third noise latent space feature and the fourth noise latent space feature; Performing a weighted summation on the first loss and the second loss to obtain a training loss of the second image generation model; Based on the training loss, model parameters of the second image generation model are updated.

7. The method according to claim 1, characterized in that The method further comprises: Based on a selection ratio, randomly selecting a first number of sample images from the plurality of sample images as detection samples, where the first number is the product of the selection ratio and the number of the plurality of sample images; When the second image generation model is trained in a first batch, performing balanced sampling on the first number of detection samples based on multiple clarity levels to which the first number of detection samples belong, to obtain the first batch of training data; When the second image generation model is trained for the nth batch, the first number of detection samples are sampled based on the level loss of multiple clarity levels belonging to multiple sample images in the n-1th batch of training data to obtain the nth batch of training data, where n is an integer greater than 1.

8. The method according to claim 7, characterized in that In the case where the second image generation model performs the first batch training, performing balanced sampling on the first number of detection samples based on multiple clarity levels to which the first number of detection samples belong to, to obtain the first batch of training data, includes: When the second image generation model is trained in a first batch, for any one of the plurality of definition levels to which the first number of detection samples belong, randomly selecting a second number of classes from the plurality of classes of the definition level; For any class, randomly selecting a third number of sample images from the plurality of sample images of the class; A plurality of sample images randomly selected from the plurality of definition levels are used as the training data of the first batch.

9. The method according to claim 7, characterized in that The nth batch of training data includes sampled samples of the plurality of definition levels, and for any definition level, the sampled samples of the definition level include sampled samples of a plurality of classes in the definition level; The method further comprises: sampling the first number of detection samples based on the level losses of multiple definition levels to which multiple sample images in the n-1th batch of training data belong when the second image generation model is trained in the nth batch, to obtain the nth batch of training data, including: determining sampling probabilities of the plurality of sharpness levels based on level losses of the plurality of sharpness levels when the second image generation model is trained for an n-th batch, wherein the level losses are positively correlated with the sampling probabilities; For any clarity level, determining sampling probabilities of the plurality of classes based on the sampling probability of the clarity level and the class losses of the plurality of classes of the clarity level; For any class, based on the sampling probability of the class, multiple sample images of the class are sampled to obtain sampled samples of the class.

10. The method according to claim 9, characterized in that The method further comprises: for any sample image in the (n-1)th batch of training data, determining a metric loss of the sample image based on the sample image and a predicted image, where the predicted image is an image predicted by the second image generation model based on the sample description text of the sample image and the sample image series information of the sample image, and the metric loss is used to indicate a difference between the sample image and the predicted image; For any image series, determining an average of the metric losses of a plurality of sample images of the image series as the series loss of the image series; For any class, the average of the series losses of multiple image series of the class is determined as the class loss of the class; For any definition level, the average of the class losses of a plurality of classes of the definition level is determined as the class loss of the definition level.

11. The method according to claim 8 or 9, characterized in that The method further comprises: For any sample image among the plurality of sample images, determining, based on a sample description text of the sample image, an image series to which the sample image belongs; For any image series, determining an average value of the sharpness of a plurality of sample images of the image series as the sharpness of the image series; determining an average feature of semantic features of a plurality of sample images of the image series as the semantic feature of the image series, wherein the semantic feature of the sample image is used to indicate semantic information of the sample image; determining, based on the sharpness of the plurality of image series and the number of the image series, a plurality of sharpness levels to which the plurality of image series belong; For any definition level, the multiple image series are clustered based on semantic features of the multiple image series at the definition level to obtain multiple classes to which the multiple image series belong.

12. The method according to claim 11, characterized in that The determining, based on the sharpness of the plurality of image series and the number of the image series, the plurality of sharpness levels to which the plurality of image series belong, comprises: sorting the plurality of image series based on the clarity of the plurality of image series; Based on the sorting results of the multiple image series and the number of the image series, the multiple image series are equally divided to obtain multiple definition ranges to which the multiple image series belong, where the multiple definition ranges correspond one-to-one to the multiple definition levels; For any clarity range, a clarity level corresponding to the clarity range is determined based on the clarity range.

13. The method according to claim 11, characterized in that The step of clustering the multiple image series at any clarity level based on the semantic features of the multiple image series at the clarity level to obtain the multiple classes to which the multiple image series belong comprises: For any definition level, randomly selecting a fourth number of image series from the plurality of image series of the definition level, where the fourth number is the number of the classes; The semantic features of the fourth number of image series are respectively used as cluster centers of the multiple classes, and the multiple image series are clustered to obtain the multiple classes.

14. A training device for an image generation model, characterized in that: The device comprises: an acquisition module, configured to acquire a plurality of sample images generated by a plurality of first image generation models, the plurality of first image generation models being trained based on object images from a plurality of clients, the first image generation models being configured to generate images based on image series information and descriptive text, the image series information being configured to represent common features of the plurality of object images; A first prediction module is configured to, for any sample image, process the sample noise text, the sample description text of the sample image, and the sample image series information of the sample image using a second image generation model to obtain a first latent space feature, and predict a first predicted image based on the first latent space feature, wherein the second image generation model has the same structure as the first image generation model but has different model parameters, the first latent space feature is the feature obtained by removing the first reference noise from the global image latent space feature, the global image latent space feature is the feature generated based on the sample description text and the sample image series information, and the first reference noise is the noise generated based on the sample noise text and corresponding to the sample description text; a second prediction model configured to process the sample noise text, the sample description text, and the sample image series information using the first image generation model to obtain second latent space features, and predict a second predicted image based on the second latent space features, where the second latent space features are features of the sample image after removing a second reference noise, and the second reference noise is noise generated based on the sample noise text and corresponding to the sample description text; A training module uses the second predicted image as supervisory information and trains the second image generation model based on the first predicted image and the second predicted image.

15. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the training method of the image generation model described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store at least one computer program, and the at least one computer program is used to execute the training method of the image generation model described in any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the training method of the image generation model according to any one of claims 1 to 13.

Images