Image generation model training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an image generation model training method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a sample image set; the sample image set comprises sample images with the same visual effect type and a plurality of objects; performing image marking processing on the sample images in the sample image set to obtain marked texts of each sample image under a plurality of representation dimensions; based on each sample image and the annotation text of each sample image, constructing a sample data set; and training a to-be-trained image generation model by using the sample data set to obtain a trained image generation model. According to the method and the device, model redundancy can be avoided, and computing resources of the electronic equipment can be greatly saved.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a training method, device, electronic device, computer-readable storage medium, and computer program product for an image generation model. Background Art

[0002] The development of deep learning has laid the foundation for text-to-image technology. Deep learning models can effectively process text and image information, mapping text descriptions to image content. People are also keen to incorporate their favorite concepts or characters into generated works by fine-tuning the models. However, related technologies can typically only generate one character at a time. If multiple characters are generated for an intellectual property (IP) ecosystem, a corresponding model must be trained for each character. This not only easily leads to model redundancy, but also increases the waste of computing resources in electronic devices during model training and loading. Summary of the Invention

[0003] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium and computer program product for an image generation model. A sample data set is constructed by using a sample image set after image labeling processing. The image generation model is trained using the sample data set. The same image generation model obtained through training can generate images with different types of objects, thereby avoiding model redundancy and greatly saving computing resources of electronic devices.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a training method for an image generation model, the method comprising: obtaining a sample image set; the sample image set comprising sample images having the same visual effect type and having multiple objects; performing image labeling processing on the sample images in the sample image set to obtain labeled text for each sample image under multiple representation dimensions; constructing a sample data set based on each sample image and the labeled text for each sample image; and using the sample data set to train the image generation model to be trained to obtain a trained image generation model.

[0006] An embodiment of the present application provides a training device for an image generation model, comprising: a sample acquisition module for acquiring a sample image set; the sample image set includes sample images having the same visual effect type and having multiple objects; an image annotation module for performing image labeling processing on the sample images in the sample image set to obtain annotated text for each sample image under multiple representation dimensions; a sample construction module for constructing a sample data set based on each sample image and the annotated text of each sample image; and a model training module for using the sample data set to train the image generation model to be trained to obtain a trained image generation model.

[0007] In the above scheme, the sample acquisition module is also used to obtain multiple sample images with the same visual effect type; the multiple sample images have different shooting angles, and each sample image includes at least one object; each sample image is cropped according to a preset object proportion value to obtain multiple cropped images; the multiple cropped images are determined as sample images in the sample image set.

[0008] In the above scheme, the image annotation module is also used to perform image labeling processing on the sample images in the sample image set to obtain the first annotation text of each sample image; deduplicate the first annotation text to obtain the second annotation text; perform text recognition on the second annotation text to obtain the target dimension to which the second annotation text belongs; and divide the second annotation text into any one of the multiple representation dimensions according to the target dimension to which the second annotation text belongs, to obtain the annotation text of each sample image under multiple representation dimensions.

[0009] In the above scheme, the image annotation module is also used to perform image labeling processing on the sample images in the sample image set for each representation dimension to obtain the third annotation text of each sample image under the representation dimension; deduplicate the third annotation text to obtain the fourth annotation text under the representation dimension; and summarize the fourth annotation texts under multiple representation dimensions to obtain the annotation text of each sample image under the multiple representation dimensions.

[0010] In the above scheme, the sample construction module is also used to generate an image label for each sample image based on the objects included in the sample image; edit the image label and the annotation text of the sample image in multiple representation dimensions according to a preset text format to obtain structured annotation text; construct an association relationship between the sample image and the structured annotation text to obtain sample data; and add the sample data to the sample data set.

[0011] In the above scheme, when there are multiple objects included in the sample image, the image label of the sample image includes: the object identifier of at least one object among the multiple objects; the annotation text of the sample image under multiple representation dimensions includes: the annotation sub-text for each object in the sample image under the multiple representation dimensions; the sample construction module is also used to obtain the target annotation sub-text of the object from the annotation text for each object in the multiple objects; according to a preset text format, the object identifier of the object and the target annotation sub-text of the object are text-edited to obtain the structured annotation sub-text of the object; the structured annotation sub-texts of all objects in the sample image are text-aggregated to obtain the structured annotation text.

[0012] In the above scheme, the model training module is also used to obtain the training parameters of the image generation model to be trained; the training parameters include at least the parameters of the model to be trained and the learning rate; the image generation model to be trained is called to perform data mapping on the sample data in the sample data set to obtain a generated image; based on the learning rate and the generated image, the parameters of the model to be trained are updated to obtain the trained image generation model.

[0013] In the above scheme, the model training module is also used to obtain the model parameters of the trained image generation model; add the model parameters to a preset model file; and call the trained image generation model to generate images by loading the model parameters in the model file.

[0014] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the training method of the image generation model provided in the embodiment of the present application.

[0015] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training method of the image generation model provided in the embodiment of the present application when executed by a processor.

[0016] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the training method of the image generation model provided in the embodiment of the present application is implemented.

[0017] The embodiments of the present application have the following beneficial effects:

[0018] First, a sample image set is obtained, which is formed by sample images with the same visual effect type and multiple objects; then, image labeling is performed on the sample images in the sample image set to obtain labeled texts for each sample image under multiple representation dimensions. The labeled texts under multiple representation dimensions can increase the features required by the model; then, a sample data set is constructed based on each sample image and the labeled text of each sample image; finally, the sample data set is used to train the image generation model to be trained to obtain a trained image generation model. In this way, a sample data set is constructed by the sample image set after image labeling, and the image generation model is trained using the sample data set, so that the image generation model can learn the features of multiple objects. Moreover, since the features of each object come from the labeled texts under multiple representation dimensions, the features of each object are more accurate, which improves the discrimination of objects in the images generated by the image generation model, and thus the same image generation model obtained through training can generate images with different types of objects, which not only avoids model redundancy but also greatly saves computing resources of electronic devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is an optional flowchart of the training method of the image generation model provided in the embodiment of the present application;

[0020] Figure 2 This is another optional flowchart of the training method of the image generation model provided in the embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an implementation flow of the image marking process provided in an embodiment of the present application;

[0022] Figure 4 This is another implementation flow diagram of the image marking process provided in the embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the implementation process of constructing a sample data set provided in an embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of the implementation process of the training image generation model provided in the embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of the implementation process of the image generation model training method provided in the embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a sample image provided in an embodiment of the present application;

[0027] Figure 9 is a structural diagram of a text encoder provided in an embodiment of the present application;

[0028] Figure 10 Schematic diagram of the structure of the encoder of the variational autoencoder provided in the embodiment of the present application;

[0029] Figure 11 Schematic diagram of the structure of the cross attention module provided in the embodiment of the present application;

[0030] Figure 12 This is a structural block diagram of a training device for an image generation model provided in an embodiment of the present application;

[0031] Figure 13 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0040] 2) Human-computer interaction interface, which is used to provide an interface for human-computer interaction functions / an interface for displaying image information.

[0041] For example, graphical user interface (GUI) display, such as augmented reality (AR) interface, virtual reality (VR) interface, voice user interface (VUI), interactive projection interface (using projection technology to display information on a plane), eye movement detection interface (interface controlled by detecting the user's line of sight), holographic interface (three-dimensional hologram formed by projecting images through holographic projection technology, so that stereoscopic images can be seen without wearing special glasses), multimodal interface (interface that combines multiple interaction methods, such as touch, vision, hearing, etc.), brain-machine interface (BMI) interface, etc.

[0042] 3) Low-Rank Adaptation (LoRA): This is a model fine-tuning method specifically designed for large language models. By introducing two small, decomposable matrices with far fewer parameters than the original model, it constructs an efficient bypass structure. Fine-tuning this bypass effectively adjusts the weights of the pre-trained model without affecting its overall structure. Requiring only minimal computing resources, LoRA enables rapid fine-tuning, and the performance of the model fine-tuned with LoRA is comparable to that of a full-model fine-tuning, achieving efficient model optimization.

[0043] 4) Latent Space: It is an abstract multidimensional space formed by the model by compressing data features.

[0044] 5) Variational Autoencoder (VAE): A generative AI method that combines deep learning and probabilistic graphical models. It is primarily used for data generation, feature learning, and latent variable inference. The core idea of ​​VAE is to map data into a latent space through variational inference and generate new data from the latent space. For example, it can generate images (such as faces and handwritten digits), text, and music.

[0045] 6) Variational Autoencoder Encoder (VAE-Encoder): Responsible for mapping the input data to the probability distribution parameters (e.g., mean and variance) of the latent space.

[0046] 7) Prompt: It is a text instruction or question input by the user to the model to guide the model to generate a specific type of response or content.

[0047] In order to better understand the training method of the image generation model provided in the embodiment of the present application, the training method of the image generation model in the related art is first described below.

[0048] Related technologies can generate an image of a person, an object, or even a specific style from dozens of images. However, these technologies typically only generate a single character. If multiple characters are generated for an intellectual property (IP) ecosystem, a corresponding model must be generated for each character, which can easily lead to model redundancy.

[0049] Based on the problems existing in the related art, an embodiment of the present application provides a training method for an image generation model. When training the image generation model, first, a sample image set formed by sample images with the same visual effect type and multiple objects is obtained; then, image labeling is performed on the sample images in the sample image set to obtain labeled texts for each sample image under multiple representation dimensions. The labeled texts under multiple representation dimensions can increase the features required by the model; then, a sample data set is constructed based on each sample image and the labeled text of each sample image; finally, the sample data set is used to train the image generation model to be trained to obtain a trained image generation model. In this way, the image generation model can learn the features of multiple objects, and since the features of each object come from the labeled texts under multiple representation dimensions, the features of each object are more accurate, which improves the discrimination of objects in the images generated by the image generation model, and thereby increases the number of objects in the images generated by the image generation model.

[0050] The training method of the image generation model provided in the embodiment of the present application can be applied to electronic devices such as laptops, tablet computers, and desktop computers. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0051] The following describes in detail the training method of the image generation model provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0052] Figure 1 This is an optional flow chart of the training method of the image generation model provided in the embodiment of the present application. The method can be applied to an electronic device, which can be a server or a terminal. That is, the training method of the image generation model in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal. The following will be explained by taking the electronic device as an example. Figure 1 As shown, the method includes the following steps S101 to S104:

[0053] Step S101: Acquire a sample image set.

[0054] Here, the sample image set includes sample images having the same visual effect type and having multiple objects.

[0055] In an embodiment of the present application, the sample images in the sample image set may have the same visual effect type, that is, the sample images in the sample image set have the same painting style. The visual effect type may be a painting style. Painting style refers to the unique style and characteristics presented by an image, including: brushwork and composition, color and tone, expression technique and overall visual perception. Brushwork and composition refer to the painting style reflected in the brushstrokes, line processing, color application and overall composition of the painting. Color and tone refer to the matching of colors, light and dark contrast, warm and cold tones, etc. Expression technique refers to different understandings and ways of expressing the same theme, such as realism, cartoon, abstraction, etc. Overall visual perception refers to the spatial sense, layering, light processing, etc. of the picture. Each sample image in the sample image set may have one object or multiple objects. In this way, different objects in the sample image set may have the same visual effect type.

[0056] Methods for acquiring sample images may include using public datasets, web crawlers, and data synthesis. For example, a set of sample images with the same visual effect type can be downloaded from the dataset's official website or related platforms. The objects in the sample images can be any entities with clear boundaries, shapes, features, or semantic meaning, or abstract elements, etc., and this embodiment of the application does not limit this. For example, the objects in the sample images can be people, animals, plants, etc.

[0057] Step S102 : performing image labeling processing on the sample images in the sample image set to obtain labeled text for each sample image in multiple representation dimensions.

[0058] In an embodiment of the present application, image labeling processing may refer to the process of annotating objects or background areas in an image. The image labeling processing method may include using a target detection model or a scene understanding model to detect or understand the image for labeling, or may use a large language model to label the image by inputting specified prompt words. The embodiment of the present application does not limit this.

[0059] Representation dimensions are a collection of features used to describe different aspects or attributes of an object. Each dimension represents information about a specific aspect of the object, and multiple dimensions together constitute a complete representation of the object. For example, if the object is a person, the multiple dimensions might include appearance description, action description, and background description. If the object is an animal, the multiple dimensions might include physiological characteristics, behavioral characteristics, environmental description, and health status. If the object is an object, the multiple dimensions might include geometric characteristics, material characteristics, and functional status.

[0060] The annotation text can be the text corresponding to each feature word. For example, if the subject is a girl, the annotation text for the appearance description may include: female, short black hair, big eyes, wearing glasses, shirt, skirt, sneakers. The annotation text for the action description may include: standing, reaching out, opening mouth. The annotation text for the background description may include: park, tree, trash can, rainy day, daytime.

[0061] Step S103: constructing a sample data set based on each sample image and the annotated text of each sample image.

[0062] In an embodiment of the present application, a sample data set is a collection of sample data for model training, verification, and testing of an image generation model. Each sample data may include each sample image and the annotated text of each sample image. Each sample image and the annotated text of each sample image may first be matched one-to-one, thereby associating each sample image with the annotated text of each sample image to obtain the sample data corresponding to each sample image, and then the sample data corresponding to all sample images may be aggregated to obtain a sample data set. For example, an image and text may be directly associated by a file name, an identifier (ID), or a unique identifier in metadata (e.g., image_001.jpg corresponds to text_001.txt).

[0063] Step S104: train the image generation model to be trained using the sample data set to obtain a trained image generation model.

[0064] In embodiments of the present application, an image generation model can learn the inherent patterns and characteristic distributions of data from a large amount of sample data and generate new images based on what it has learned. The generated images are visually similar to the sample data and can be completely fictitious or generated based on specific conditions (such as text descriptions, labels, or other images). Image generation models can include: generative adversarial networks, variational autoencoders, diffusion models, autoregressive models, conditional generation models, and stable diffusion models. For example, a pre-trained stable diffusion model can be selected as the base model; then, training parameters such as the learning rate, number of training steps, and batch size are set; then, the sample data set is input into the base model to obtain the corresponding predicted values; finally, the loss between the predicted values ​​and the target values ​​is calculated, and the model parameters of the base model are updated through backpropagation. When the loss value fluctuates slightly within a certain range and no longer decreases significantly, training can be terminated, or when the quality and diversity of the generated images have reached the expected goals, training can be stopped. After training is completed, the model weights and parameters can be saved for subsequent use.

[0065] In an embodiment of the present application, first, a sample image set is obtained, which is formed by sample images having the same visual effect type and multiple objects; then, image labeling is performed on the sample images in the sample image set to obtain labeled texts for each sample image under multiple representation dimensions, and the features required by the model can be increased through the labeled texts under multiple representation dimensions; then, a sample data set is constructed based on each sample image and the labeled text of each sample image; finally, the sample data set is used to train the image generation model to be trained to obtain a trained image generation model. In this way, the image generation model can learn the features of multiple objects, and since the features of each object come from the labeled texts under multiple representation dimensions, the features of each object are more accurate, which improves the discrimination of objects in the images generated by the image generation model, and thereby increases the number of objects in the images generated by the image generation model.

[0066] The following examples illustrate the application scenarios of the training method of the image generation model provided in the embodiments of the present application.

[0067] Artists and designers can use the image generation models trained using the embodiments of this application (such as StableDiffusion) to generate creative images, illustrations, and concept art. For example, by inputting a specific text description, they can generate futuristic science fiction scenes or retro-style illustrations. They can also use a pre-trained StableDiffusion model and fine-tune it with a sample dataset of a specific style to generate images that conform to a specific artistic style.

[0068] Game developers can use the image generation model trained by the embodiments of this application to quickly create game assets such as characters, environments, and props, accelerating prototyping and content iteration. They can also train the image generation model to generate high-quality game assets by collecting a large amount of image data in game styles.

[0069] In film and animation production, the image generation model trained using the embodiments of this application can be used to generate backgrounds, special effects, or character designs, helping artists and directors realize their visual creativity. The model can also be trained to generate images that match specific scenes or styles by combining sample data from film and television styles.

[0070] The image generation model trained using the embodiments of this application can also be used to generate architectural renderings for urban construction or architectural design presentations. Alternatively, an image generation model can be trained using an image dataset of architectural styles to enable the image generation model to generate detailed architectural renderings from simple line drawings.

[0071] The following will describe the training method of the image generation model in the embodiment of the present application in combination with the above scenario.

[0072] Figure 2 This is another optional flow chart of the training method of the image generation model provided in the embodiment of the present application, such as Figure 2 As shown, the method includes the following steps S201 to S212:

[0073] Step S201: The terminal receives a training operation of an image generation model input by a user.

[0074] The training operation of the image generation model includes an input operation, which is used to input the identification of the visual effect type of the sample image. For example, the user operates on a terminal (such as a computer, mobile phone, etc.) through input devices such as a keyboard, mouse, and touch screen, such as pressing a key, moving the mouse, or touching the screen.

[0075] In step S202 , the terminal encapsulates the identifier of the visual effect type of the sample image into the model training request.

[0076] Model training requests are used to request the server to perform model training.

[0077] Step S203: The terminal sends a model training request to the server.

[0078] In an embodiment of the present application, the terminal sends a model training request to the server to request the server to perform model training.

[0079] Step S204: The server obtains multiple sample images with the same visual effect type in response to the model training request sent by the terminal.

[0080] Here, the plurality of sample images have different shooting angles, and each sample image includes at least one object.

[0081] In an embodiment of the present application, the server parses the model training request sent by the terminal, obtains the identification of the visual effect type, and then obtains multiple sample images with the same visual effect type based on the identification of the visual effect type. These sample images have different shooting angles, so that sample images of the same object can show different angles of the object. For example, if the object is a chair, the sample images may include images taken from the front, images taken from above, and images taken from the back, etc. In addition, each sample image includes at least one object. For example, an image may include two people talking.

[0082] In step S205 , the server crops each sample image according to a preset object ratio value to obtain a plurality of cropped images.

[0083] In the embodiment of the present application, the object proportion value is pre-set. The object proportion value can be the proportion of the area of ​​the object in the sample image, or the proportion of the height of the object in the height of the sample image, or the proportion of the width of the object in the width of the sample image. The embodiment of the present application does not limit this and can be set according to actual conditions. When the object proportion value of the object in the sample image is less than the preset object proportion value, each sample image can be cropped. For example, if the preset object proportion value is 80%, and the object proportion value of the object in the sample image is 50%, the sample image can be cropped to obtain a cropped image that meets the preset object proportion value.

[0084] In step S206 , the server determines the multiple cropped images as sample images in the sample image set.

[0085] In the embodiment of the present application, the obtained multiple cropped images are used as sample images in the sample image set.

[0086] Through steps S204 to S206, by collecting sample images from different shooting angles, the image generation model can learn more features, and then cropping the sample images according to the object ratio can also enable the image generation model to pay more attention to the features of the objects in the sample images, reduce background noise interference, and thus increase the diversity and accuracy of the data in the sample image set.

[0087] In step S207 , the server performs image labeling on the sample images in the sample image set to obtain labeled text for each sample image in multiple representation dimensions.

[0088] In some embodiments, see Figure 3 , Figure 3 It is shown that step S207 can be implemented by the following steps S2071A to S2074A:

[0089] Step S2071A: perform image labeling on the sample images in the sample image set to obtain a first labeled text for each sample image.

[0090] In the embodiment of the present application, image labeling is performed on each sample image in the sample image set to obtain a first labeled text for each sample image. The first labeled text is the entire labeled text of the sample image. For example, taking a sample image as an example, image labeling can obtain the first labeled text: "Girl, reading a book in bed, black hair, blue top, red skirt, yellow top, night, purple pillow, stuffed bunny, rabbit."

[0091] Step S2072A: remove duplicates from the first annotated text to obtain a second annotated text.

[0092] In an embodiment of the present application, duplicate annotation text in a first annotation text is removed to obtain a second annotation text. For example, the first annotation text can be hashed, and first annotation texts with the same hash value can be deleted, retaining one first annotation text for each hash value; or the first annotation text can be vectorized and similarity calculated, and multiple first annotation texts with similarities greater than a similarity threshold are deleted, retaining one first annotation text. Incorrect annotation texts can also be deleted to obtain a second annotation text. For example, multiple image labeling processes are performed on sample images in a sample image set, and first annotation texts that appear less than a threshold number of times are identified as incorrect annotation texts and deleted. For example, the first annotation text is obtained as follows: "Girl, reading a book in bed, blue eyes, black hair, blue top, red skirt, yellow top, night, purple pillow, plush bunny, rabbit", where "blue top" and "yellow top" have the same annotation text, but different colors. One of the annotation texts is incorrect. The image labeling process can be repeated. "Blue top" only appears once, so it can be confirmed that "blue top" is an incorrect annotation text and the incorrect annotation text "blue top" is deleted. By calculating text similarity, we can confirm that "plush bunny" and "rabbit" are duplicates. We can then delete the annotated text "rabbit" and retain the more refined annotated text "plush bunny." Finally, we get the second annotated text: "girl, reading a book in bed, black hair, red skirt, yellow top, night, purple pillow, plush bunny."

[0093] Step S2073A: perform text recognition on the second annotated text to obtain the target dimension to which the second annotated text belongs.

[0094] In this embodiment, text recognition is performed on each second annotated text in the second annotated text in turn to obtain the target dimension to which each second annotated text belongs. For example, text recognition is performed on the second annotated text "reading books in bed" to obtain the target dimension as the character behavior dimension, "blue eyes" as the character appearance dimension, and "stuffed bunny" as the background dimension.

[0095] Step S2074A: Divide the second annotated text into any one of a plurality of representation dimensions according to the target dimension to which the second annotated text belongs, and obtain the annotated text of each sample image in the plurality of representation dimensions.

[0096] In an embodiment of the present application, if the target dimension is the same as any one of the multiple representation dimensions, the second annotation text can be divided into the representation dimension that is the same as the target dimension among the multiple representation dimensions according to the target dimension to which the second annotation text belongs. If the target dimension is a sub-dimension of any one of the multiple representation dimensions, the second annotation text can be divided into the representation dimension to which the target dimension belongs among the multiple representation dimensions according to the target dimension to which the second annotation text belongs. If there are multiple representation dimensions to which the target dimension belongs, the target dimension can be randomly assigned to any one of the multiple representation dimensions. After all the second annotation texts are divided, the annotation texts for each sample image under multiple representation dimensions can be obtained. For example, the multiple representation dimensions are character behavior dimension, character appearance dimension, and background dimension. The target dimension is the same as any one of the multiple representation dimensions. For example, "reading a book in bed" is a character behavior dimension, "blue eyes" is a character appearance dimension, and "stuffed bunny" is a background dimension. The second annotated text: "girl, reading a book in bed, black hair, red skirt, yellow top, night, purple pillow, stuffed bunny" can be divided according to the target dimension to which it belongs, resulting in the annotated text "girl, black hair, red skirt, yellow top" for the character appearance dimension, the annotated text "reading a book in bed" for the character behavior dimension, and the annotated text "night, purple pillow, stuffed bunny" for the background dimension. For example, the multiple representation dimensions are the character behavior dimension, the character appearance dimension, and the background dimension. The target dimension is a sub-dimension of any one of the multiple representation dimensions. For example, the target dimension included in the character appearance dimension can be the facial dimension, the clothing dimension, etc. "Black hair" is a facial dimension and a sub-dimension of the character appearance dimension. Therefore, "black hair" can be classified into the character appearance dimension. The division process for other annotated texts can refer to the above content and is not listed here. Finally, the labeled text of the sample image in multiple representation dimensions is obtained, namely, the labeled text of the character appearance dimension "girl, black hair, red skirt, yellow top", the labeled text of the character behavior dimension "reading a book in bed", and the labeled text of the background dimension "night, purple pillow, plush rabbit".

[0097] Through steps S2071A to S2074A, the redundancy and errors in the annotated text can be eliminated through the deduplication and error correction mechanism to ensure the accuracy of the annotated text. Then, the annotated text is layered to achieve feature decoupling, so that the image generation model can independently learn features at different semantic levels and enhance the fine-grained learning ability of the image generation model.

[0098] In some embodiments, see Figure 4 , Figure 4 It shows that step S207 can be implemented by the following steps S2071B to S2073B:

[0099] Step S2071B: for each representation dimension, perform image labeling processing on the sample images in the sample image set to obtain a third labeled text for each sample image under the representation dimension.

[0100] In an embodiment of the present application, the specific types of multiple representation dimensions are first determined, and then, for each dimension, the sample images in the sample image set are subjected to image labeling processing to obtain the third labeled text of each sample image under the representation dimension. For example, the multiple representation dimensions are the character behavior dimension, the character appearance dimension, and the background dimension. Under the character behavior dimension, the sample images in the sample image set are subjected to image labeling processing to obtain the third labeled text "reading books in bed", under the character appearance dimension, the sample images in the sample image set are subjected to image labeling processing to obtain the third labeled text "girl, black hair, red skirt, yellow top, blue top", and under the background dimension, the sample images in the sample image set are subjected to image labeling processing to obtain the third labeled text "night, purple pillow, plush rabbit, rabbit".

[0101] Step S2072B: deduplicate the third annotation text to obtain the fourth annotation text under the representation dimension.

[0102] In this embodiment of the present application, the third annotated text can be deduplicated with reference to the implementation of step S2072A to obtain fourth annotated text for each representation dimension. For example, after deduplicating the third annotated text, the fourth annotated text for the character appearance dimension can be "girl, black hair, red skirt, yellow top", the fourth annotated text for the character behavior dimension can be "reading a book in bed", and the fourth annotated text for the background dimension can be "night, purple pillow, plush bunny".

[0103] Step S2073B: Summarize the fourth annotation texts under multiple representation dimensions to obtain the annotation texts of each sample image under multiple representation dimensions.

[0104] In the embodiment of the present application, the fourth annotation text under each representation dimension is summarized to obtain the annotation text of each sample image under multiple representation dimensions.

[0105] Through steps S2071B to S2073B, independent image labeling is performed according to dimensions, which reduces cross-dimensional semantic interference and achieves feature decoupling. Then, the annotated text is deduplicated to ensure the accuracy of the annotated text, enabling the image generation model to independently learn features at different semantic levels and enhance the fine-grained learning ability of the image generation model.

[0106] In step S208 , the server constructs a sample data set based on each sample image and the annotated text of each sample image.

[0107] In some embodiments, see Figure 5 , Figure 5 It shows that step S208 can be implemented by following steps S2081 to S2084:

[0108] Step S2081 : For each sample image, generate an image label for the sample image based on the object included in the sample image.

[0109] In an embodiment of the present application, for each sample image, an image label can be generated for the sample image based on the objects included in the sample image. When the number of objects included in the sample image is one, the object identifier of the object (such as a name, type, or other identifying text) can be used as the label of the sample image. When the number of objects included in the sample image is multiple, the object identifiers of the multiple objects can be used together as the label of the sample image. For example, if there is only one chair in the sample image, "chair" can be used as the image label of the sample image. If there is a chair and a table in the sample image, "chair-table" can be used as the image label of the sample image.

[0110] Step S2082 : Edit the image labels and the annotation texts of the sample images in multiple representation dimensions according to a preset text format to obtain structured annotation texts.

[0111] In the embodiment of the present application, the text format is pre-set, and the text format can limit the sorting method of the image labels and the annotation texts of the sample images under multiple representation dimensions. For example, the image label can be limited to the first place, and the annotation texts under multiple representation dimensions can be sorted according to preset rules. The image label can also be limited to the first line, and the annotation texts under multiple representation dimensions can be sorted according to preset rules, and each representation dimension is located in a separate line. The specific text format can be limited according to actual conditions, and the embodiment of the present application does not limit this. By editing the image labels and annotation texts according to the preset text format, a structured annotation text can be obtained.

[0112] In some embodiments, when the sample image includes multiple objects, the image label of the sample image includes: the object identifier of at least one object among the multiple objects; the annotation text of the sample image under multiple representation dimensions includes: the annotation sub-text for each object in the sample image under multiple representation dimensions; the above step S2082 can be implemented in the following way: first, for each object among the multiple objects, obtain the target annotation sub-text of the object from the annotation text; then, according to the preset text format, the object identifier of the object and the target annotation sub-text of the object are text-edited to obtain the structured annotation sub-text of the object; finally, the structured annotation sub-texts of all objects in the sample image are text-aggregated to obtain the structured annotation text.

[0113] In an embodiment of the present application, first, when there are multiple objects included in the sample image, for each of the multiple objects in the sample image, the target annotation subtext of each object is obtained from the annotation text. For example, the sample image includes a chair and a table, and the annotation text of the sample image under multiple representation dimensions (such as shape dimension, color dimension and material dimension) is "square, round, brown, black, wood, stainless steel", the target annotation subtext of the chair is "square, brown, wood", and the target annotation subtext of the table is "round, black, stainless steel". Then, according to the preset text format, the object identifier of the object and the target annotation subtext of the object are text edited to obtain the structured annotation subtext of the object. For example, the preset text format is arranged in the order of "object identifier-material dimension-shape dimension-color dimension", and the structured annotation subtext of each object is "chair-wood-square-brown" and "table-stainless steel-round-black". Finally, the structured annotation subtexts of all objects in the sample image are textually summarized to obtain the structured annotation text. For example, all structured annotated sub-texts are summarized, and the obtained structured annotated text is "chair-wood-square-brown; table-stainless steel-round-black".

[0114] This approach generates precise and structured target annotation text for each object in a sample image. This helps clearly distinguish objects in multi-object scenarios. Furthermore, by formatting the annotation information in a specific format, it improves data usability and the efficiency of subsequent image generation model training.

[0115] Step S2083: construct an association relationship between the sample image and the structured annotated text to obtain sample data.

[0116] It should be noted that the implementation method of step S2083 can refer to step S103, and the implementation details of step S2083 will not be repeated in this embodiment of the application.

[0117] Step S2084: Add the sample data to the sample data set.

[0118] Through steps S2081 to S2084, accurate and structured annotation text can be generated for each object in the sample image, and then the image label and annotation text can be edited using a preset text format, thereby enhancing data consistency, optimizing the structured representation of image data, and providing high-quality annotation information for subsequent image generation model training, thereby improving model performance.

[0119] In step S209 , the server trains the image generation model to be trained using the sample data set to obtain a trained image generation model.

[0120] In some embodiments, see Figure 6 , Figure 6 It shows that step S209 can be implemented by the following steps S2091 to S2093:

[0121] Step S2091, obtaining training parameters of the image generation model to be trained.

[0122] Here, the training parameters include at least the parameters of the model to be trained and the learning rate.

[0123] In this embodiment of the present application, the parameters of the model to be trained can be the U-Net architecture parameters of the diffusion model, including the convolution kernel weights of each residual block, the matrix parameters of the temporal embedding layer, and the projection matrix of the attention mechanism. The learning rate can be set according to the actual situation, for example, to 0.001.

[0124] Step S2092: Call the image generation model to be trained to perform data mapping on the sample data in the sample data set to obtain a generated image.

[0125] In the embodiments of the present application, data mapping may refer to the process of converting input sample data into a generated image through an image generation model. For example, if the image generation model to be trained is Stable Diffusion, the input sample data will be used, and Stable Diffusion will gradually remove noise in the latent space through a multi-step denoising process to ultimately generate an image.

[0126] Step S2093: Based on the learning rate and the generated image, the parameters of the model to be trained are updated to obtain a trained image generation model.

[0127] In the embodiments of the present application, a loss function is calculated using the generated and real images. Based on the loss function results, the model parameters are updated using gradient descent. For example, if the Adam optimizer is used, the gradient of the loss function with respect to the model parameters is calculated, and the parameters are updated based on the learning rate and the gradient. When the loss function value no longer decreases significantly, or the model training cycle reaches a preset number of rounds, the training ends, and the trained image generation model is obtained.

[0128] By accurately setting the training parameters through steps S2091 to S2093 and then using the sample data set to train the parameters of the model to be trained, the model training efficiency can be improved.

[0129] In step S210 , the server obtains model parameters of the trained image generation model.

[0130] In an embodiment of the present application, the server can obtain the model parameters of the trained image generation model from the training environment. After the training is completed, the server obtains the model parameters from the training environment through an application programming interface (API) call or a file transfer protocol.

[0131] In step S211 , the server adds the model parameters to a preset model file.

[0132] In the embodiment of the present application, the model file may be a container that may include structural information of the image generation model, optimizer status, etc. The server may add the acquired model parameters to the preset model file.

[0133] In step S212, the server loads the model parameters in the model file to call the trained image generation model to generate the image.

[0134] In an embodiment of the present application, the server can use a deep learning framework to load the model parameters in the model file to call the trained image generation model for image generation. For example, the server uses Python and PyTorch frameworks to load the model file. After loading, the server can initialize a Stable Diffusion model instance and set the StableDiffusion parameters to the model parameters read from the model file. The server then receives the user's text description input, such as "a kitten playing on the grass", and calls the loaded Stable Diffusion model to generate the corresponding image.

[0135] Through steps S210 to S212, the model parameters can be obtained to ensure the accurate transmission of the model parameters, and then the model parameters are added to the preset model file, which provides guarantees for the integrity and availability of the image generation model. Finally, the image generation model is called to generate images, providing users with efficient and accurate image generation services.

[0136] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0137] Generating multiple character tasks in a diffusion model is not difficult, but it is difficult to make the model generate several fixed characters in each generation task. In the currently popular LoRA model training version, the faces of the characters produced by the same model are too similar. To distinguish the characters, different LoRA models need to be used. However, the embodiment of the present application performs special annotation on the training images, and then combines the text encoder (Text-Encoder) module and the cross-attention (Cross-Attention) module to decouple the features in the image. The model learns the meaning and weight of each feature in the image through the prompt-image pairs (i.e., sample data) of the training data set, and completes the control of multiple characters by a single LoRA model.

[0138] See also Figure 7 , Figure 7 It is a schematic diagram of the implementation process of the training method of the image generation model provided in the embodiment of the present application. The following will be exemplified by taking the execution subject as the server as an example. Step S701, obtain the training picture (i.e., sample image). Step S702, perform special annotation on the training picture (i.e., image labeling processing). Step S703, use the annotated training picture to train the model to obtain a trained low-rank adaptive model (i.e., image generation model). Step S704, save the low-rank adaptive model. Step S705, test the model training results.

[0139] The embodiment of the present application uses a specific image annotation method to generate a multi-role prompt vocabulary. First, according to the character's personality, several pictures containing the character from different angles are collected. Then, the image set is labeled according to the character using a specific image marker. The features required by the model are added to the prompt (to reduce the features that the model does not expect to appear). The added features are defined as unique features of each character, and a special trigger word (Trigger, i.e., image label) is assigned to each character.

[0140] When using special image annotation methods for special annotation, first perform rough data labeling. You can use machine labeling tools to automatically label and add a person's name trigger (all relevant files can be placed in a folder called person's name). Then clean the data, remove duplicate annotations (i.e., annotated text), and retain more detailed annotations. Finally, organize the annotation format into a unified paragraph format (i.e., preset text format). For example, the first line is the name; the second line is the description of the person's appearance, annotating the characteristics of the person's appearance, such as hair and eye color, clothing color and style, hat type, upper body or full body portrait, etc.; the third line is the description of the person's actions, annotating the person's actions and expressions; the fourth line is the description of the background, annotating what appears in the background, location, day and night, other objects, etc.

[0141] See also Figure 8 , Figure 8 is a schematic diagram of a sample image provided in an embodiment of the present application. Figure 8 For example, we first perform rough labeling to obtain the labeled text: "Kixx, 1girl, kixx (xxxCartoon), black hair, closed eyes, bow, hair bow, arms behind head, solo, grass, red bow, lying, short hair, on back, hairband, red hairband, short sleeves, blush stickers, shirt, closed mouth, dress, flower, upper body, radio, sleeping" (i.e., the first labeled text). This can be a name or a tag representing an element in the image. These tags are disordered and may contain problems such as mislabeling, repeated labeling, missing labels, and overly simple labeling.

[0142] The resulting annotations are then cleaned. In the sentence "Kixx, 1girl, kixx (xxxCartoon), blackhair, closed eyes, bow, hair bow, arms behind head, solo, grass, red bow, lying, shorthair, on back, hairband, red hairband, short sleeves, blush stickers, shirt, closedmouth, dress, flower, upper body, sleeping," "kixx (xxxCartoon)" and the person's name "Kixx" are duplicates, so only one is retained. "bow" and "red bow" are duplicates, so the more detailed annotation is retained. "Hair bow" and "redbow" describe the same object from different angles and can be modified or not, depending on the specific situation. Here, they can be merged into a single feature, "red hair bow." "Solo" means being alone. Since there are images of people alone in the dataset, this label can be deleted based on the specific situation. "On back" is not clearly represented in the image and is an incorrect annotation, so "on back" is deleted. "shirt" means shirt, which is an incorrect label here, so delete "shirt".

[0143] Finally, the cleaned annotation text (i.e., the second annotation text) is organized into a fixed segment format, as shown below:

[0144] "Kixx,

[0145] 1girl, black hair, red hair bow, solo, short hair, upper body, red hairband, short sleeves, blush, dress,

[0146] closed eyes, arms behind head, lying, sleeping, closed mouth,

[0147] grass, flower, radio" (i.e., labeled text under multiple representation dimensions).

[0148] In the embodiment of the present application, when training the image generation model, the cross-attention mechanism is used to decouple and match the text features and image character features. When using the cross-attention mechanism for image generation, first, the prompt words of the image pair (i.e., sample data) are encoded using the Prompt and Trigger through the Text-Encoder module to extract the semantic information of each image. Figure 9 , Figure 9 901, a tokenizer 902, a word embedding 903, and a text transformer 904. Figure 10 , Figure 10 This is a schematic diagram of the structure of the encoder of the variational autoencoder provided in the embodiment of the present application. The Vae-Encoder module is used at the image end to encode the sample image and extract the potential features of the sample image in the latent space. Figure 11 , Figure 11 This is a schematic diagram of the structure of the cross-attention module provided in an embodiment of the present application. The attention weight is dynamically adjusted through the Cross-Attention module, and the image features (Q) and text embeddings (K, V) (i.e., the text embeddings of the annotated text) are interacted, thereby effectively combining the text information and the features in the image generation process, injecting the key information in the text into the image generation process, and thus generating an image that matches the input text description. The cross-attention mechanism allows the model to dynamically focus on different parts of the input text when generating an image, thereby better capturing the details and semantic information of the text description. The expression of the Cross-Attention module is shown in formula (1):

[0149]

[0150] in, It is the text semantic embedding, obtained by Text-Encoder encoding, which represents the semantic information of the input text. is the image potential feature, which is obtained by VAE-Encoder encoding. The image potential feature represents the abstract features of the input image; K = V = X1*W K , K is the text key matrix, which is obtained by projecting text features and provides matching semantic anchors for image queries. V is the text value matrix, which retains the original text semantic information and is used for weighted aggregation into image generation; Q = X2*W Q is the image query matrix, which is obtained by projecting image features and drives the attention mechanism to focus on the key semantics in the text; W KThe learned projection matrix, namely the key projection matrix, maps the text features to the key space and learns how to “answer” the image query with the text features; W Q The learned projection matrix, i.e., the query projection matrix, maps image features to the query space, and learns how to use image features to "ask" text; d1 is the dimension of the key-value set, and d2 is also the dimension of the key-value set; soft refers to the normalized exponential function (softmax function).

[0151] The embodiments of the present application can implement a method for simply implementing Vincent graph training (i.e., the training method for the above-mentioned image generation model) and a device for Vincent graph training (i.e., the training device for the image generation model). By integrating the optimization of modules such as the image preprocessing module, the parameter adjustment module, the training module, and the generation and display module, the training and testing steps are simplified. Among them, the device for Vincent graph training in the embodiments of the present application includes multiple modules: an image preprocessing module, a parameter adjustment module, a training module, and a testing module.

[0152] The image preprocessing module can first receive input images and collect a data set of images (i.e., sample image sets) containing multiple characters and multiple angles (i.e., shooting angles) of the same style (i.e., visual effect type). The image preprocessing module then performs image cropping. The image preprocessing module crops the original image to ensure the proportion of characters in the image. Finally, the image preprocessing module adds labels to the images. The image preprocessing module uses a special labeler (such as a large language model) to add labels to each image, generates a text description (i.e., annotated text) that conforms to the multi-character image set, and adds a specific Trigger label (i.e., image label) to the Prompt text. These labels will be used together with the image to generate image-label pairs (i.e., sample data) as a training set (i.e., sample data set) in preparation for subsequent input into the training module.

[0153] The parameter adjustment module can be used to obtain the set training parameters and adjust the various parameters required for training. The parameters required for training include: learning rate (dynamically adjusting the learning rate based on the performance during training to improve training efficiency and effectiveness), number of training rounds (setting the number of rounds of model training to ensure that the image generation model reaches optimal performance within an appropriate time), and other hyperparameters (adjusting relevant hyperparameters to optimize the training process based on the specific training tasks and model requirements). The parameter adjustment module can also optimize parameter settings. Based on the performance of the image generation model during training, the parameter adjustment module dynamically adjusts the learning rate and number of training rounds to optimize the training effect.

[0154] The training module can receive preprocessed data and parameters, obtain cropped images and label data from the image preprocessing module, and obtain the set training parameters from the parameter adjustment module. The training module can also perform model training steps. After starting the model training process, the training module uses a deep learning algorithm to train the input data. During the training process, the learning rate and the number of training rounds are dynamically adjusted according to the feedback from the parameter adjustment module to ensure that the image generation model is trained in the optimal state. After the training is completed, the training module outputs the training results and can save the model parameters of the image generation model as a LoRA file (i.e., the preset model file) for subsequent testing and application.

[0155] The test module can load LoRA files and display the LoRA files generated by the module after training. Then, the test module uses the text-to-image technology based on the LoRA model and combines it with the input text prompt to generate multiple character images.

[0156] Based on the training method of the image generation model described in the above embodiment, Figure 12 A structural block diagram of a training device for an image generation model provided in an embodiment of the present application is shown. The training device 100 for the image generation model can be a device in an electronic device (for example, a server). The training device for the image generation model can be implemented in software, which can be software in the form of programs and plug-ins, including the following software modules: a sample acquisition module 101, an image annotation module 102, a sample construction module 103 and a model training module 104. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0157] Among them, the sample acquisition module 101 is used to obtain a sample image set; the sample image set includes sample images with the same visual effect type and multiple objects; the image annotation module 102 is used to perform image labeling processing on the sample images in the sample image set to obtain the annotation text of each sample image under multiple representation dimensions; the sample construction module 103 is used to construct a sample data set based on each sample image and the annotation text of each sample image; the model training module 104 is used to use the sample data set to train the image generation model to be trained to obtain the trained image generation model.

[0158] In some embodiments, the sample acquisition module 101 is also used to acquire multiple sample images with the same visual effect type; the multiple sample images have different shooting angles, and each sample image includes at least one object; each sample image is cropped according to a preset object proportion value to obtain multiple cropped images; the multiple cropped images are determined as sample images in the sample image set.

[0159] In some embodiments, the image annotation module 102 is further used to perform image labeling processing on the sample images in the sample image set to obtain a first annotation text for each sample image; deduplicate the first annotation text to obtain a second annotation text; perform text recognition on the second annotation text to obtain a target dimension to which the second annotation text belongs; and divide the second annotation text into any one of the multiple representation dimensions according to the target dimension to which the second annotation text belongs to obtain the annotation text for each sample image under multiple representation dimensions.

[0160] In some embodiments, the image annotation module 102 is further used to perform image labeling processing on the sample images in the sample image set for each representation dimension to obtain a third annotation text for each sample image under the representation dimension; deduplicate the third annotation text to obtain a fourth annotation text under the representation dimension; and summarize the fourth annotation texts under multiple representation dimensions to obtain annotation texts for each sample image under the multiple representation dimensions.

[0161] In some embodiments, the sample construction module 103 is further used to generate an image label for each sample image based on the objects included in the sample image; perform text editing on the image label and the annotation text of the sample image under multiple representation dimensions according to a preset text format to obtain structured annotation text; construct an association relationship between the sample image and the structured annotation text to obtain sample data; and add the sample data to the sample data set.

[0162] In some embodiments, when the sample image includes multiple objects, the image label of the sample image includes: an object identifier of at least one object among the multiple objects; the annotation text of the sample image under multiple representation dimensions includes: annotation sub-texts for each object in the sample image under the multiple representation dimensions; the sample construction module 103 is also used to obtain the target annotation sub-text of the object from the annotation text for each object in the multiple objects; edit the object identifier of the object and the target annotation sub-text of the object according to a preset text format to obtain the structured annotation sub-text of the object; and summarize the structured annotation sub-texts of all objects in the sample image to obtain the structured annotation text.

[0163] In some embodiments, the model training module 104 is also used to obtain the training parameters of the image generation model to be trained; the training parameters include at least the model parameters to be trained and the learning rate; the image generation model to be trained is called to perform data mapping on the sample data in the sample data set to obtain a generated image; based on the learning rate and the generated image, the parameters of the model to be trained are updated to obtain the trained image generation model.

[0164] In some embodiments, the model training module 104 is also used to obtain model parameters of the trained image generation model; add the model parameters to a preset model file; and call the trained image generation model to generate images by loading the model parameters in the model file.

[0165] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0166] An embodiment of the present application provides an electronic device, Figure 13 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 13 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 13 Only one is shown), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the computer executable instructions 133, the steps of the training method embodiment of any of the above-mentioned image generation models are implemented.

[0167] The electronic device may include but is not limited to a processor 131 and a memory 132. It will be understood by those skilled in the art that Figure 13 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0168] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0169] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard disk or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard disk equipped on the electronic device 130, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. Furthermore, the memory 132 may include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 132 may also be used to temporarily store data that has been output or is about to be output.

[0170] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image generation model training method described in the present invention.

[0171] The present application embodiment provides a computer-readable storage medium having computer-executable instructions or a computer program stored therein. When the computer-executable instructions or the computer program are executed by a processor, the processor will be caused to execute the training method of the image generation model provided in the present application embodiment, for example, Figure 1 The training method of the image generation model is shown.

[0172] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0173] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0174] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0175] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0176] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Acquire a sample image set; the sample image set includes sample images having the same visual effect type and having multiple objects; Performing image labeling processing on sample images in the sample image set to obtain labeled text for each sample image in multiple representation dimensions; Constructing a sample data set based on each sample image and the annotated text of each sample image; The sample data set is used to train the image generation model to be trained to obtain a trained image generation model.

2. The method according to claim 1, characterized in that The acquiring of the sample image set comprises: Acquire multiple sample images of the same visual effect type; the multiple sample images have different shooting angles, and each sample image includes at least one object; Crop each sample image according to a preset object ratio value to obtain multiple cropped images; The multiple cropped images are determined as sample images in the sample image set.

3. The method according to claim 1, characterized in that The performing image labeling processing on the sample images in the sample image set to obtain labeled text of each sample image in multiple representation dimensions includes: Performing image labeling processing on sample images in the sample image set to obtain a first labeled text for each sample image; Deduplication of the first annotated text to obtain a second annotated text; Performing text recognition on the second annotated text to obtain a target dimension to which the second annotated text belongs; According to the target dimension to which the second annotated text belongs, the second annotated text is divided into any one of the multiple representation dimensions to obtain the annotated text of each sample image under the multiple representation dimensions.

4. The method according to claim 1, wherein The performing image labeling processing on the sample images in the sample image set to obtain labeled text of each sample image in multiple representation dimensions includes: For each representation dimension, performing image labeling processing on the sample images in the sample image set to obtain a third labeled text for each sample image under the representation dimension; Deduplication is performed on the third annotated text to obtain a fourth annotated text under the representation dimension; The fourth annotated texts under multiple representation dimensions are aggregated to obtain annotated text of each sample image under the multiple representation dimensions.

5. The method according to claim 1, characterized in that The constructing of a sample data set based on each sample image and the annotated text of each sample image includes: For each sample image, generating an image label for the sample image based on an object included in the sample image; Editing the image label and the annotation text of the sample image in multiple representation dimensions according to a preset text format to obtain structured annotation text; Constructing an association relationship between the sample image and the structured annotated text to obtain sample data; The sample data is added to the sample data set.

6. The method according to claim 5, characterized in that In the case where the sample image includes multiple objects, the image label of the sample image includes: an object identifier of at least one object among the multiple objects; the annotation text of the sample image under multiple representation dimensions includes: annotation subtexts for each object in the sample image under the multiple representation dimensions; The step of editing the image label and the annotation text of the sample image in multiple representation dimensions according to a preset text format to obtain structured annotation text includes: For each object of the plurality of objects, obtaining a target annotation subtext of the object from the annotation text; Editing the object identifier of the object and the target annotation subtext of the object according to a preset text format to obtain a structured annotation subtext of the object; The structured annotation subtexts of all objects in the sample image are summarized to obtain the structured annotation text.

7. The method according to any one of claims 1 to 6, characterized in that The method of training the image generation model to be trained using the sample data set to obtain the trained image generation model includes: Obtaining training parameters of the image generation model to be trained; the training parameters include at least parameters of the model to be trained and a learning rate; Calling the image generation model to be trained to perform data mapping on the sample data in the sample data set to obtain a generated image; Based on the learning rate and the generated image, the parameters of the model to be trained are updated to obtain a trained image generation model.

8. The method according to any one of claims 1 to 6, characterized in that After the image generation model to be trained is trained using the sample data set to obtain a trained image generation model, the method further includes: Obtaining model parameters of the trained image generation model; Adding the model parameters to a preset model file; The trained image generation model is called to generate images by loading the model parameters in the model file.

9. A training device for an image generation model, characterized in that: The device comprises: A sample acquisition module is used to acquire a sample image set; the sample image set includes sample images of the same visual effect type and having multiple objects; An image annotation module, configured to perform image annotation processing on sample images in the sample image set to obtain annotation text for each sample image in multiple representation dimensions; A sample construction module, configured to construct a sample data set based on each sample image and the annotated text of each sample image; The model training module is used to train the image generation model to be trained using the sample data set to obtain a trained image generation model.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the image generation model training method according to any one of claims 1 to 8 when executing computer-executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the training method of the image generation model described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the training method of the image generation model described in any one of claims 1 to 8 is implemented.

Images