Methods for training image generation model and constructing sample set, and large image generation model

Abstract

The present application discloses methods for training an image generation model and constructing a sample set, and a large image generation model. During training of an image generation model, on the basis of the number of attributes of preset attributes of sample texts in a sample set of a target scenario, the sample texts and sample images are grouped to obtain sample groups, in which changes in the attributes of the sample texts in the same sample group are relatively consistent; and according to the number of attributes corresponding to the sample groups, the sample groups are sorted in ascending order, the sorted sample groups are sequentially used for training an image generation model to be trained, and a target model is obtained.

Description

Image generation model training, sample set construction method and image generation large model

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202311192236.5, filed on September 15, 2023, entitled “Image generation model training, sample set construction method and image generation large model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to the training of image generation models, sample set construction methods, and large image generation models. Background Art

[0004] With the advancement of science and technology, image generation technology is attracting increasing attention. Existing image generation technologies are categorized into image-to-image generation models and text-to-image generation models, depending on the input conditions. Text-to-image generation models, in particular, have attracted widespread attention due to their suitability for everyday user scenarios and their simplicity and convenience. In text-to-image generation models, users input a prompt text into the model, which then generates an image that matches the semantics of the prompt text based on the received prompt text.

[0005] Summary of the Invention

[0006] Some embodiments of the present application provide a method for training an image generation model, including:

[0007] Acquire a sample set corresponding to a target scene, wherein the sample set includes a plurality of sample images, and the sample images correspond to sample texts;

[0008] Determining the number of preset attributes included in the sample texts in the sample set, and grouping the sample texts and corresponding sample images according to the number of attributes to obtain sample groups;

[0009] The sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups, and the image generation model to be trained is trained using the sorted sample groups in sequence to obtain a target model.

[0010] On the other hand, some embodiments of the present application further provide a sample set construction method, including:

[0011] Acquire an image set corresponding to a target scene, and text to be filled saved for the target scene, wherein the text to be filled includes at least one target field to which an attribute value is to be written;

[0012] For any sample image in the image set, determine the target attribute value corresponding to the target field in the sample image, fill the target attribute value into the target field, and obtain the sample text corresponding to the sample image; save the sample text and the corresponding sample image into the sample set.

[0013] On the other hand, some embodiments of the present application provide a large image generation model, which is obtained based on the training method of the above-mentioned image generation model.

[0014] On the other hand, some embodiments of the present application provide an electronic device, including:

[0015] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the training method of the image generation model provided in the first aspect above.

[0016] Some embodiments of the present application provide a computer-readable non-volatile storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the above-mentioned training method of the image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a flow chart of a method for training an image generation model according to some embodiments of the present application;

[0018] FIG2 is a schematic diagram of an image generated according to prompt text provided in some embodiments of the present application;

[0019] FIG3 is a schematic diagram of candidate attributes provided by some embodiments of the present application;

[0020] FIG4 is a schematic diagram of page display provided by some embodiments of the present application;

[0021] FIG5 is a schematic diagram of sample images provided in some embodiments of the present application;

[0022] FIG6 is a schematic diagram of the training process of another image generation model provided in some embodiments of the present application;

[0023] FIG7 is a schematic diagram of model training provided in some embodiments of the present application;

[0024] FIG8 is a schematic diagram of a fine-tuning training process of a general image generation model provided by some embodiments of the present application;

[0025] FIG9 is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0026] To further illustrate some embodiments of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although some embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in some embodiments of the present application. In the actual processing process or when the control device is executed, the method can be executed in the order of the method shown in the embodiments or drawings or in parallel.

[0027] In the related art, when training an image generation model, the model to be trained is typically trained using widely available sample images and sample text. This allows the target model to possess general knowledge. However, because the target model is not trained based on sample images and sample text focused on specific scenes, the target model's image generation performance for specific scenes is poor. For example, the target model's performance in generating detailed images of pedestrians needs to be improved. Furthermore, because the sample text corresponding to the sample images used in training the model to be trained is manually written based on experience, the resulting sample text lacks a unified standard. For example, some sample texts written by some people describe the attributes of the person in the image in very detailed terms, such as "a man wearing a mask, a military green top, and military green pants, aged 20-40." Other sample texts are more coarse, such as "a person wearing a military green top." As a result, the types and number of attributes included in different sample texts vary. Consequently, the diversity of attribute variations in the sample texts leads to extremely slow convergence of the model to be trained.

[0028] Some embodiments of the present application provide a training method for an image generation model, a sample set construction method, and a large image generation model. When training the image generation model, a sample set corresponding to a target scene is obtained, where the sample set includes multiple sample images, and the sample images correspond to sample texts; the number of attributes of preset attributes included in the sample texts in the sample set is determined, and the sample texts and corresponding sample images are grouped according to the number of attributes to obtain sample groups; the sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups, and the sorted sample groups are used in sequence to train the image generation model to be trained to obtain a target model.

[0029] FIG1 is a flow chart of a method for training an image generation model provided in some embodiments of the present application, comprising the following steps:

[0030] S101: Acquire a sample set corresponding to a target scene, wherein the sample set includes a plurality of sample images, and each of the sample images corresponds to a sample text.

[0031] The training method of the image generation model provided in some embodiments of the present application is applied to an electronic device, which may be a server, a PC, etc.

[0032] With the development of technology, image generation models can generate a variety of images. For example, if the prompt text "A sheep in a wine glass" is input into the image generation model, the image generation model can generate an image as shown in Figure 2. Figure 2 is a schematic diagram of an image generated based on the prompt text provided in some embodiments of the present application. It can be seen from Figure 2 that a lamb is lying in a wine glass. In real life, it is impossible to use image acquisition equipment to actually capture such an image. Therefore, the emergence of image generation models has provided more possibilities for technological development.

[0033] In related technologies, image generation models can also be used to address the problem of missing data on specific attributes in models such as video structuring. For example, in video structured attribute recognition, data on pedestrians wearing flip-flops is scarce, resulting in a non-robust attribute recognition model. Therefore, using an image generation model to generate a batch of pedestrians wearing flip-flops can effectively supplement the data and enhance the robustness of the attribute recognition model.

[0034] In order to improve the effect of the images of a specific scene generated by the image generation model, the image generation model to be trained can be trained for a specific scene. In some embodiments of the present application, the image generation model to be trained can be a brand new image generation model, that is, a newly developed image generation model that has not undergone any training; the image generation model to be trained can also be an image generation model obtained through pre-training. Then, in some embodiments of the present application, when training the pre-trained image generation model, the sample set of the specific scene is used to fine-tune the pre-trained image generation model. In some embodiments of the present application, the image generation model to be trained can be a stable diffusion model (SD), ControlNet (CN), Dreambooth, etc.

[0035] In order to train an image generation model with better image generation effects, in some embodiments of the present application, a sample set corresponding to a target scene can be obtained. The target scene can be input by the user of the electronic device or sent to the electronic device by another electronic device. The target scene can be a weather scene such as snowy, rainy, or windy, or a scene containing people, animals, or cars.

[0036] In some embodiments of the present application, the sample set corresponding to the acquired target scene may include a large number of sample images and sample texts corresponding to the sample images. The sample text describes the content included in the corresponding sample image. The sample text can be understood as a prompt text used to prompt the model to generate the content included in the image. Among them, the sample image is an image about the target scene. For example, when the target scene is a snowy day, the sample images in the sample set are all images about snowy days. When the target scene is a scene containing pedestrians, the sample images in the sample set all contain pedestrians. In some embodiments of the present application, the sample images included in the sample set can be acquired from a publicly available image set, or can be acquired by an image acquisition device, or can be generated by other image generation models. Each sample image in the sample set can correspond to one sample text, or can correspond to multiple sample texts.

[0037] S102: Determine the number of preset attributes included in the sample texts in the sample set, and group the sample texts and corresponding sample images according to the number of attributes to obtain sample groups.

[0038] After the sample set corresponding to the target scene is obtained, sample texts in the sample set are obtained one by one, and the number of attributes of the preset attributes included in the obtained sample texts is determined.

[0039] In some embodiments of the present application, the preset attributes may be pre-configured based on the characteristics of the target scene. For example, when the target scene is a scene containing pedestrians, the preset attributes may be gender, age, head and shoulder attributes, upper body attributes, lower body attributes, foot attributes, etc., wherein the head and shoulder attributes may be understood as whether the pedestrian in the image is wearing a hat, glasses, earrings, etc.; the upper body attributes may be understood as whether the pedestrian in the image is wearing a T-shirt, a shirt, or a down jacket, etc.; the lower body attributes may be understood as whether the pedestrian in the image is wearing jeans, shorts, or casual pants, etc.; the foot attributes may be understood as whether the pedestrian in the image is wearing high heels, leather shoes, or sandals, etc.

[0040] When determining the number of preset attributes included in any sample text, it can be determined whether the sample text includes preset fields, for example, whether the sample text includes preset fields such as "male", "female", "hat", and "jeans". If the sample text includes the two fields "male" and "jeans", then it can be determined that the sample text includes two preset attributes, namely gender and lower body attributes, and the number of attributes corresponding to the sample text is 2.

[0041] After determining the number of attributes corresponding to each sample text in the sample set, the sample texts and their corresponding sample images can be grouped according to the number of attributes corresponding to each sample text. In other words, sample texts and their corresponding sample images with the same number of attributes are grouped together, and each sample text in the resulting sample text group includes the same number of preset attributes.

[0042] S103: sorting the sample groups in ascending order according to the number of attributes corresponding to the sample groups, and using the sorted sample groups in sequence to train the image generation model to be trained to obtain a target model.

[0043] If, when the image generation model to be trained is being trained, the sample text used in the previous training is very simple, such as "a man", and the sample text used in the next training is very complex, such as "a man wearing a mask and blue jeans", then the image generation model to be trained will have problems such as slow convergence speed and poor robustness. Therefore, in some embodiments of the present application, after obtaining the sample groupings, the obtained sample groupings are sorted in ascending order according to the number of attributes corresponding to the sample groupings. After the ascending sorting, the sorted sample groupings are used in sequence to train the image generation model to be trained. After the training meets the convergence conditions, the image generation model for the target scene can be obtained. For the convenience of description, the image generation model obtained by training will be referred to as the target model. That is, when training the image generation model to be trained, the image generation model to be trained is first trained using sample texts containing fewer preset attributes, and then the image generation model to be trained is trained using sample texts containing more preset attributes. That is, first, low-information image-text pairs are input to the image generation model to be trained, and then high-information image-text pairs are gradually added to achieve a process from simple to complex.

[0044] For example, assuming that the image generation model to be trained is trained for a total of 1000 rounds, the sample text and sample image in the sample group with an attribute number of 1 can be used for training in rounds 1-300; the sample text and sample image in the sample group with an attribute number of 2 can be used for training in rounds 301-600, and so on.

[0045] Because in some embodiments of the present application, when training the image generation model, a sample set corresponding to the target scene is obtained, the number of attributes of the preset attributes included in the sample text in the sample set is determined, and the sample text and the corresponding sample image are grouped according to the number of attributes to obtain a sample group. The number of attributes in each sample text in the sample group is consistent, so the attribute changes of the sample text in the same sample group are relatively simple, and the sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups. The sorted sample groups are used in turn to train the image generation model to be trained to obtain the target model, that is, the sample texts from simple to detailed are used progressively for training, which avoids the problem of extremely slow convergence speed of the image generation model to be trained due to the diversity of attribute changes in the sample text, and realizes efficient training of the image generation model for a specific scene.

[0046] In order to improve the efficiency of model training, in some embodiments of the present application, the process of obtaining a sample set corresponding to the target scene includes:

[0047] Acquire an image set corresponding to the target scene, and text to be filled saved for the target scene, wherein the text to be filled includes at least one target field to which an attribute value is to be written;

[0048] For any sample image in the image set, determine the target attribute value corresponding to the target field in the sample image, fill the target attribute value into the target field, and obtain the sample text corresponding to the sample image; save the sample text corresponding to the sample image in the sample set.

[0049] Since the amount of data contained in the sample set used to train the image generation model to be trained is extremely large, if the sample text corresponding to each sample image is completely manually input word for word, it will seriously affect the training efficiency of the image generation model to be trained. In addition, different workers have different ideas, so the degree of detail in describing the sample images by different workers is also different. Therefore, in order to standardize the description method of sample text, in some embodiments of the present application, the text to be filled is pre-saved, and the text to be filled may include at least one target field to which an attribute value is to be written. When the staff annotates the sample image, they can write the corresponding attribute value in the target field in the text to be filled.

[0050] Determining the sample text corresponding to the sample image based on the pre-saved text to be filled can improve the efficiency of determining the sample set. Therefore, in some embodiments of the present application, when obtaining the sample set corresponding to the target scene, the image set corresponding to the target scene and the pre-saved text to be filled can be obtained.

[0051] Since the contents included in the sample images of different scenes are different, the contents described in the sample texts corresponding to the sample images of different scenes are also different, while the contents included in the sample images of the same scene are similar, the contents described in the sample texts corresponding to the sample images of the same scene are also similar. Therefore, the pre-saved texts to be filled can be saved for different scenes, that is, multiple texts to be filled are pre-saved, and the scenes corresponding to different texts to be filled are different. The same scene can correspond to one or more texts to be filled.

[0052] In some embodiments of the present application, when obtaining the text to be filled, the text to be filled saved for the target scene can be obtained, and the obtained text to be filled can be one or more. When multiple texts to be filled are saved for the target scene, the number of target fields included in different texts to be filled may be the same or different. When the number of targets included in different texts to be filled is the same, then the target attributes corresponding to the target fields included in the same number of texts to be filled are different.

[0053] In some embodiments of the present application, each text to be filled includes at least one target field to which an attribute value is to be written. The target field is the field that needs to be filled in subsequently. In order to facilitate the subsequent clarification of which preset attribute value corresponds to which target field needs to be filled in, in some embodiments of the present application, the preset attribute corresponding to each target field can be identified in the text to be filled in.

[0054] Specifically, assuming that the target scene is a scene containing pedestrians, the text to be filled in for the target scene can be a person of {gender}, a person of {age}, a person of {gender} of {age}, a person of {gender} of {age} wearing {foot attributes}, etc. Among them, the {} in the text to be filled in and the fields inside {} are the target fields to which the attribute values ​​are to be written. That is to say, the "{gender}", "{age}", and "{foot attributes}" in the above text to be filled in are all target fields, and the content inside "{}" is the preset attribute corresponding to the target field. When subsequent staff fill in the text to be filled in, they can determine which preset attribute's attribute value needs to be filled in the target field corresponding to the position based on the content included in each "{}".

[0055] After obtaining the image set corresponding to the target scene and the text to be filled in for the target scene, the sample text corresponding to each sample image in the image set can be determined. In some embodiments of the present application, when determining the sample text corresponding to any sample image in the image set, the target attribute value corresponding to the target field in the sample image can be determined, and the target attribute value can be filled into the target field to obtain the sample text corresponding to the sample image. After obtaining the sample text corresponding to the sample image, the sample text corresponding to the sample image can be saved in the sample set, and the sample set will be used for subsequent training of the image generation model to be trained.

[0056] In order to improve the accuracy of determining the sample text, based on the above embodiments, determining the target attribute value corresponding to the target field in any sample image in the image set includes:

[0057] Determine the target attribute corresponding to the target field in the text to be filled;

[0058] For any sample image in the image set, the sample image and the target attribute are output, and a prompt is given to input the attribute value corresponding to the target attribute in the sample image; the received attribute value corresponding to the target attribute is determined as the target attribute value; or, for any sample image in the image set, image recognition is performed on the sample image to obtain the attribute value corresponding to the target attribute in the sample image, and the attribute value is determined as the target attribute value.

[0059] In some embodiments of the present application, when determining the target attribute value corresponding to the target field in the sample image, the target attribute corresponding to the target field in each acquired text to be filled can be determined. Specifically, based on text recognition technology, it can be respectively identified whether each text to be filled includes a preset character, and the preset character is the character that identifies the preset attribute. For example, it is identified whether each text to be filled includes "{}". If "{}" is recognized, it means that the field within "{}" is the target attribute.

[0060] In order to facilitate subsequent filling of target attribute values ​​into the corresponding text to be filled, after determining the target attribute corresponding to the target field in any text to be filled, the determined target attribute may be saved corresponding to the text to be filled.

[0061] After determining the target attribute corresponding to the target field in the text to be filled, any sample image in the image set can be obtained, and the sample image and each determined target attribute can be output. For example, the sample image and the determined target attribute can be displayed on a display, and the user of the electronic device can be prompted to enter the attribute value corresponding to the target attribute in the currently displayed sample image. The user of the electronic device can enter the corresponding attribute value in the input box corresponding to each target attribute, and after the input is completed, click a button such as "Submit", "Complete" or "Save" to send the attribute value corresponding to each target attribute to the electronic device. After receiving the attribute value corresponding to the target attribute, the electronic device can determine the attribute value as the target attribute.

[0062] Specifically, assuming that sample image 1 and target attribute A, target attribute B, and target attribute C are output, the attribute value input by the user of the electronic device for target attribute A is attribute value A, the attribute value input for target attribute B is attribute value B, and the attribute value input for target attribute C is attribute value C. After receiving the attribute value corresponding to each target attribute, the electronic device may determine attribute value A as the target attribute value corresponding to target attribute A, attribute value B as the target attribute value corresponding to target attribute B, and attribute value C as the target attribute value corresponding to target attribute C.

[0063] In order to further improve the efficiency of sample text determination, in some embodiments of the present application, before outputting the sample image and the target attribute, the method further includes:

[0064] Obtaining candidate attribute values ​​stored for the target attribute;

[0065] Outputting the sample image and the target attribute includes:

[0066] The sample image, the target attribute and the candidate attribute value are output.

[0067] Since the category of the target attribute is fixed, the candidate range of the attribute value corresponding to each target attribute is also limited. For example, the candidate attribute values ​​corresponding to the target attribute of gender are only male or female. Therefore, in some embodiments of the present application, in order to further improve the efficiency of sample text determination, before outputting the sample image and target attribute, the candidate attribute value saved for the target attribute can also be obtained. The candidate attribute value can be the possible attribute value listed in advance by the staff based on experience.

[0068] After obtaining the candidate attribute values ​​saved for the target attribute, when outputting the sample image and the target attribute, the obtained candidate attribute values ​​can also be output corresponding to the target attribute, so that the staff can choose from the candidate attribute values ​​without having to enter the corresponding attribute values ​​one by one.

[0069] Specifically, Figure 3 is a schematic diagram of candidate attributes provided by some embodiments of the present application. As shown in Figure 3, the target attributes corresponding to the target field in the determined text to be filled may include gender, age, head and shoulder attributes, upper body attributes, lower body attributes and foot attributes, among which the candidate attribute values ​​saved for the target attribute of gender may include male and female; the candidate attribute values ​​saved for the target attribute of age may include less than 20 years old, 20-40 years old, 40-60 years old, and greater than 60 years old; the candidate attribute values ​​saved for the target attribute of head and shoulder attributes may include wearing a hat, wearing glasses, wearing earrings, wearing jewelry, etc.; the candidate attribute values ​​saved for the target attribute of upper body attributes may include T-shirts, shirts, down jackets, etc.; the candidate attribute values ​​saved for the target attribute of lower body attributes may include jeans, shorts, casual pants, etc.; the candidate attribute values ​​saved for the target attribute of foot attributes may include shoes, sandals, etc. As shown in Figure 3, the candidate attribute values ​​for the upper body, lower body, and foot attributes also include the attribute color. Color can be used as a modifier for the candidate attribute values ​​and can be combined with the candidate attribute values ​​for the upper body, lower body, and foot attributes, for example, red T-shirt, blue shirt, yellow down jacket, green down jacket, etc. In some embodiments of the present application, the attribute color has multiple candidate attribute values, including red, orange, yellow, green, cyan, blue, purple, and other colors.

[0070] Since the number of sample images used to train the image generation model to be trained is extremely large, and each sample image includes multiple attributes, although the staff does not have to enter the sample text word for word, the user of the electronic device needs to enter the attribute value word for word, which still consumes a lot of manpower and material resources. Therefore, in some embodiments of the present application, image recognition can be performed on any sample image in the image set to obtain the attribute value corresponding to the target attribute in the sample image, and the attribute value is determined as the target attribute value. When identifying the attribute value corresponding to the target attribute in the sample image, the sample image can be identified based on the trained attribute recognition model. It should be noted that identifying the attribute value corresponding to the attribute in the image is a prior art, and this will not be repeated in some embodiments of the present application.

[0071] In order to ensure the accuracy of determining the sample text, in some embodiments of the present application, after obtaining the attribute value corresponding to the target attribute in the sample image and before determining the attribute value as the target attribute value, the method further includes:

[0072] Outputting the sample image and the obtained attribute value corresponding to the target attribute, and prompting a user to review the attribute value corresponding to the target attribute, to obtain a first attribute value corresponding to the target attribute after review;

[0073] The first attribute value corresponding to the target attribute after review is received, and the attribute value corresponding to the target attribute obtained through image recognition is updated using the first attribute value.

[0074] Since the accuracy of the samples included in the sample set used to train the image generation model to be trained will seriously affect the obtained target model, if the content described in the sample text is inconsistent with the content included in the sample image, the image generation model to be trained will be trained incorrectly. Therefore, in some embodiments of the present application, after performing image recognition on the sample image, the attribute value corresponding to the target attribute in the identified sample image may not be determined as the target attribute value, and the attribute value corresponding to the target attribute obtained based on image recognition and the corresponding sample image may be output. Exemplarily, the attribute value corresponding to the target attribute obtained by image recognition and the corresponding sample image may be displayed on a display, and a prompt may be provided to review the attribute value corresponding to the identified target attribute.

[0075] After receiving the audit prompt, the staff can audit the attribute values ​​corresponding to the target attributes of each sample image. If it is determined that the attribute values ​​obtained by image recognition are incorrect, the corresponding attribute values ​​can be modified, or the incorrect attribute values ​​can be identified. If a certain attribute value is identified as an error, the electronic device can re-identify the target attribute corresponding to the identification after receiving the identification, or not save the attribute value corresponding to the identification. After the audit is completed, the staff can send an audit completion instruction and send the attribute value corresponding to the audited target attribute to the electronic device. For ease of description, in some embodiments of the present application, the attribute value corresponding to the audited target attribute can be referred to as the first attribute value. When sending the first attribute value to the electronic device, the staff can click buttons such as "Submit", "Complete", and "Save" to send the first attribute value.

[0076] After receiving the first attribute value corresponding to the reviewed target attribute, the electronic device can consider that the content included in the first attribute value is correct, and can use the first attribute to update the attribute value corresponding to the target attribute obtained through image recognition, and subsequently perform other operations based on the confirmed correct first attribute value.

[0077] Specifically, assuming that the attribute value corresponding to the target attribute A included in the sample image 1 identified based on image recognition technology is A, and the attribute value corresponding to the target attribute B is B, then the sample image 1 and the attribute value corresponding to each target attribute can be displayed on the visualization page. For example, Figure 4 is a page display schematic diagram provided by some embodiments of the present application. As shown in Figure 4, the sample image to be reviewed can be displayed on the left side of the page (left and right in the figure), that is, the sample image 1 is displayed on the left side of the page (left and right in the figure), and each target attribute and the attribute value corresponding to the target attribute are displayed in sequence on the right side of the page (left and right in the figure), that is, the target attribute A and its corresponding attribute value A, and the target attribute B and its corresponding attribute value B are displayed in sequence on the right side of the page (left and right in the figure). When the staff determines that the attribute value corresponding to the target attribute B is incorrectly identified, for example, the attribute value corresponding to the target attribute B in the sample image 1 is actually M, then the staff can modify the attribute value B corresponding to the target attribute B to M on the page. After the staff has completed the review of each target attribute, they can click the "Save" button to send the reviewed first attribute value to the electronic device.

[0078] The configuration of the text to be filled is described below with reference to a specific embodiment. Based on the target attributes included in FIG3 , the text to be filled can be pre-configured as shown in Table 1:

[0079] Table 1

[0080] According to Table 1 above, it can be seen that in some embodiments of the present application, multiple texts to be filled are constructed according to the number of target attributes, wherein the texts to be filled correspond to texts to be filled that include only a single type of target attribute, texts to be filled that include any two types of target attributes, texts to be filled that include any three types of target attributes, texts to be filled that include any four types of target attributes, texts to be filled that include any five types of target attributes, and texts to be filled that include all six types of target attributes. By configuring the texts to be filled in the above manner, 63 types of texts to be filled can be obtained.

[0081] For example, the text to be filled in for a single target attribute shown in Table 1 above may be "a {gender} person". Then, based on the candidate attribute values ​​shown in FIG3 , the text to be filled in may generate sample texts of "a man" and "a woman".

[0082] For example, the text to be filled for any two target attributes shown in Table 1 above may be "a person with {head and shoulder attributes} wearing {upper body attributes}". Then, based on the candidate attribute values ​​shown in FIG3 , the text to be filled may generate the following sample text:

[0083] A person wearing a hat and a white T-shirt;

[0084] a man in a hat and red shorts;

[0085] A man wearing a hat and a black down jacket;

[0086] A man wearing glasses and a white T-shirt;

[0087] …

[0088] In addition, since the candidate attribute values ​​can be matched with various color information, color input can also be added to the text to be filled to obtain more sample texts.

[0089] FIG5 is a schematic diagram of a sample image provided by some embodiments of the present application. As shown in FIG5 , FIG5 includes a pedestrian. Based on the text to be filled in Table 1, at least the following sample text is determined for the sample image:

[0090] A man;

[0091] A man in a hat and a black coat;

[0092] A man in a hat and black coat;

[0093] a man wearing a mask, a hat, and black pants;

[0094] A man aged 20-40 wearing a hat and a black shirt;

[0095] A man wearing a mask, a black top and black pants;

[0096] A man wearing a mask, a black shirt and black pants;

[0097] A man aged 20-40 wearing a mask, a black shirt and black pants;

[0098] …

[0099] It can be found that this method of generating sample text can effectively increase training data, thereby increasing the robustness of the model. At the same time, due to the diversity of expressions, the generalization ability of the model can be enhanced.

[0100] In order to ensure the effectiveness of model training, in some embodiments of the present application, the sequentially using the sorted sample groups to train the image generation model to be trained includes:

[0101] Sequentially obtaining the first sample group from the sorted sample groups;

[0102] If the first sample group is the first group in the sorted sample groups, the first sample group is used to train the image generation model to be trained.

[0103] Since model training has the characteristic of data forgetting, for example, sample forgetting may occur during the training of neural networks or deep models. Therefore, in order to ensure the effectiveness of model training, in some embodiments of the present application, when the image generation model to be trained is trained using the sorted sample groups in sequence, when the image generation model to be trained is trained using a certain sample group, the image generation model to be trained can be made to "review" the previously learned content before continuing to learn new content. Here, "review" refers to using part of the previous sample data to re-input the model for training.

[0104] In some embodiments of the present application, after the sample groups are sorted in ascending order, sample groups can be obtained sequentially from the sorted sample groups. For ease of description, in some embodiments of the present application, the currently obtained sample group can be referred to as the first sample group.

[0105] After obtaining the first sample grouping, it can be determined whether the first sample grouping is the first grouping in the sorted sample groupings. If the first sample grouping is the first grouping, it means that the image generation model to be trained has not learned other sample images and corresponding sample texts before, and there is no need for "review". The first sample grouping can be used to train the image generation model to be trained.

[0106] In order to further ensure the effect of model training, in some embodiments of the present application, the method further includes:

[0107] If the first sample group is not the first group in the sorted sample groups, obtaining a set number of sample images and corresponding sample texts from other sample groups before the first sample group;

[0108] The image generation model to be trained is trained using the set number of sample images, the sample texts corresponding to the set number of sample images, and the first sample grouping.

[0109] If the first sample group obtained is not the first group in the sorted sample groups, it means that the image generation model to be trained has previously learned sample images and sample texts in other sample groups. Therefore, in order to prevent the image generation model to be trained from forgetting the data of the previously learned content, in some embodiments of the present application, it is possible to determine which other sample groups are ranked before the first sample group in the sorted sample groups. After determining the other sample groups before the first sample group, a set number of sample images and their corresponding sample texts can be selected from each other sample group. The set number of sample images and their corresponding sample texts can be understood as the content that the image generation model to be trained needs to "review". The set number can be any positive integer, such as 10, 30, 55, 100, etc.

[0110] After obtaining the sample texts corresponding to the set number of sample images, the image generation model to be trained may be trained using the set number of sample images, the sample texts corresponding to the set number of sample images, and the first sample grouping.

[0111] Specifically, assume that the image generation model to be trained needs to be trained 1000 times, wherein the image generation model to be trained needs to be trained using a sample group with 3 attributes for the 100th to 200th rounds. In order to prevent the image generation model to be trained from forgetting previously learned content, in some embodiments of the present application, a set number of sample images and corresponding sample texts can be obtained from the sample groups with 1 attribute and 2 attributes. Assuming that the set number is 50, then 50 sample images and the sample texts corresponding to the 50 sample images can be obtained from the sample group with 1 attribute; and 50 sample images and the sample texts corresponding to the 50 sample images can be obtained from the sample group with 2 attributes. After obtaining the above sample images and their corresponding sample texts, the image generation model to be trained can be trained using the above 100 sample images and their corresponding sample texts, as well as the first sample group. Of course, since the set number is 50, in some embodiments of the present application, a total of 50 sample images and their corresponding sample texts can also be randomly obtained from other sample groups before the first sample group.

[0112] The following describes the training process of the image generation model to be trained with reference to a specific embodiment. FIG6 is a schematic diagram of the training process of another image generation model provided in some embodiments of the present application. As shown in FIG6 , the process includes the following steps:

[0113] S601: The sample groups obtained by grouping are sorted in ascending order according to the number of attributes corresponding to the sample groups.

[0114] S602: Obtain the first sample group in sequence from the sorted sample groups.

[0115] S603: Determine whether the first sample group is the first sample group in the sorted sample groups. If yes, execute S604; otherwise, execute S605.

[0116] S604: Use the first sample group to train the image generation model to be trained.

[0117] S605: Obtain a set number of sample images and corresponding sample texts from other sample groups before the first sample group, and train the image generation model to be trained using the set number of sample images, the sample texts corresponding to the set number of sample images, and the first sample group.

[0118] To further ensure the effectiveness of model training, in some embodiments of the present application, obtaining a set number of sample images and corresponding sample texts includes:

[0119] Get the pre-saved target ratio;

[0120] Obtain sample images and corresponding sample texts in a quantity corresponding to the target ratio from the other sample groups.

[0121] Since the number of sample images included in different sample groups is different, if only a set number is pre-configured, it is very likely that the number of sample images included in a certain sample group is less than the set number. Therefore, in some embodiments of the present application, in order to further ensure the effect of model training, a target ratio can be pre-configured. The target ratio can be understood as obtaining sample images and their corresponding sample texts of the target ratio of each sample group from each sample group. In some embodiments of the present application, the target ratio can be one or more. When the target ratio is one, it means obtaining sample images and their corresponding sample texts of the same target ratio from each sample group; when the target ratio is multiple, different target ratios can be configured for sample groups with different numbers of attributes, for example, the target ratio corresponding to the sample group with 1 attribute is 10%, the target ratio corresponding to the sample group with 2 attributes is 30%, the target ratio corresponding to the sample group with 3 attributes is 40%, and so on.

[0122] After the pre-saved target ratio is obtained, sample images and corresponding sample texts of a number corresponding to the target ratio may be obtained from other sample groups preceding the first sample group.

[0123] Specifically, assuming that the pre-saved target ratio is 100%, the image generation model to be trained needs to be trained for 1000 rounds. Figure 7 is a schematic diagram of model training provided by some embodiments of the present application. As shown in Figure 7, when the image generation model to be trained is trained, a sample grouping with an attribute number of 1 is used to train the image generation model to be trained in rounds 1-100, that is, sample texts containing a single type of attribute and their corresponding sample images are used to train the image generation model to be trained; after 100 rounds of training, the image generation model to be trained has acquired certain knowledge, and in the next rounds 101-200, a sample grouping with an attribute number of 2 can be used to train the image generation model to be trained, that is, sample texts containing any two types of attributes and their corresponding sample images are used to train the image generation model to be trained. However, due to the The image generation model also learned the knowledge of the sample texts and corresponding sample images included in the sample group with an attribute number of 1 in the 1st to 100th rounds. In order to prevent the image generation model to be trained from forgetting the learned knowledge, the sample images and their corresponding sample texts of the target ratio can be obtained in the sample group with an attribute number of 1. Since the pre-saved target ratio is 100%, the sample groups with an attribute number of 1 and an attribute number of 2 can be used to train the image generation model to be trained in the 101st to 200th rounds; similarly, in the next 201st to 300th rounds, the sample groups with an attribute number of 1, an attribute number of 2 and an attribute number of 3 can be used to train the image generation model to be trained, and so on.

[0124] The following describes the process of fine-tuning the general image generation model in conjunction with another embodiment to obtain an image generation model specifically for pedestrian images. Figure 8 is a schematic diagram of the fine-tuning training process of the general image generation model in some embodiments of the present application. As shown in Figure 8, in some embodiments of the present application, the general text image generation model can be fine-tuned to obtain a pedestrian text image generation model. The training data used in the fine-tuning training is a text image pair, where the image used for training is an image including pedestrians. The pedestrians in the images used for training are diverse, for example, with different appearances, different clothes, and different postures. Each image used for training corresponds to a sample text, which is determined based on a multi-level text prompt template. The text prompt template is the template to be filled in the above embodiments. After obtaining the above data for training the general text image generation model, the general text image generation model can be fine-tuned based on the fine-tuning training method of course learning to obtain a pedestrian text image generation model.

[0125] Some embodiments of the present application propose a method for converting discrete attribute values ​​of a specific scene into a multi-level text prompt method. Based on this, text-image pair data is constructed, and a curriculum learning method is used to fine-tune the model on the general text image generation model. The general text image generation model is converted into a proprietary pedestrian image generation model, thereby realizing the controllable generation of pedestrian images.

[0126] In some embodiments of the present application, a method for constructing a sample set is further provided, the method comprising:

[0127] Acquire an image set corresponding to a target scene, and text to be filled saved for the target scene, wherein the text to be filled includes at least one target field to which an attribute value is to be written;

[0128] For any sample image in the image set, determine the target attribute value corresponding to the target field in the sample image, fill the target attribute value into the target field, and obtain the sample text corresponding to the sample image; save the sample text corresponding to the sample image in the sample set.

[0129] In some embodiments, a plurality of texts to be filled are stored for the target scenario. If different texts to be filled include the same number of target fields, the target attributes corresponding to the target fields included in the same number of texts to be filled are different.

[0130] In some embodiments, determining, for any sample image in the image set, a target attribute value corresponding to the target field in the sample image includes:

[0131] Determine the target attribute corresponding to the target field in the text to be filled;

[0132] For any sample image in the image set, the sample image and the target attribute are output, and a prompt is given to input the attribute value corresponding to the target attribute in the sample image; the received attribute value corresponding to the target attribute is determined as the target attribute value; or, for any sample image in the image set, image recognition is performed on the sample image to obtain the attribute value corresponding to the target attribute in the sample image, and the attribute value is determined as the target attribute value.

[0133] In some embodiments, before outputting the sample image and the target attributes, the method further includes:

[0134] Obtaining candidate attribute values ​​stored for the target attribute;

[0135] Outputting the sample image and the target attribute includes:

[0136] The sample image, the target attribute and the candidate attribute value are output.

[0137] In some embodiments, after obtaining the attribute value corresponding to the target attribute in the sample image and before determining the attribute value as the target attribute value, the method further includes:

[0138] Outputting the sample image and the obtained attribute value corresponding to the target attribute, and prompting a user to review the attribute value corresponding to the target attribute, to obtain a first attribute value corresponding to the target attribute after review;

[0139] The first attribute value corresponding to the target attribute after review is received, and the attribute value corresponding to the target attribute obtained through image recognition is updated using the first attribute value.

[0140] The sample set construction process provided in some embodiments of the present application is consistent with the process of obtaining the sample set corresponding to the target scene during the training process of the image generation model. Since the process has been described in detail in the above embodiments, the sample set construction process will not be repeated in some embodiments of the present application.

[0141] In some embodiments of the present application, a large image generation model is also provided. The large image generation model is obtained based on the training method of the above-mentioned image generation model. Therefore, the training of the large image generation model can refer to the above-mentioned embodiments.

[0142] In some embodiments of the present application, the "big model" in the image generation big model can be understood as a model based on a transformer architecture; the "big model" can also be understood as a machine learning model with a large parameter scale and complexity, for example, a neural network model with millions to billions of parameters or tens of billions of parameters; the "big model" can also be understood as a deep learning model trained on large-scale training data through semi-(weak) supervision, full supervision, self-supervision or unsupervised techniques. The "big model" can handle a variety of different tasks. When training the "big model", it is generally trained based on training data in a certain target task field. The trained "big model" can generally be migrated to other task fields similar to the target task field for use.

[0143] FIG9 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application. On the basis of the above embodiments, some embodiments of the present application further provide an electronic device, as shown in FIG9 , including: a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other via the communication bus 1104;

[0144] The memory 1103 stores a computer program. When the program is executed by the processor 1101, the processor 1101 performs the following steps:

[0145] Acquire a sample set corresponding to a target scene, wherein the sample set includes a plurality of sample images, and the sample images correspond to sample texts;

[0146] Determining the number of preset attributes included in the sample texts in the sample set, and grouping the sample texts and corresponding sample images according to the number of attributes to obtain sample groups;

[0147] The sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups, and the sorted sample groups are used in sequence to train the image generation model to be trained to obtain a target model.

[0148] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.

[0149] Based on the above embodiments, some embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor. When the program is executed on the processor, the processor implements the following steps:

[0150] Acquire a sample set corresponding to a target scene, wherein the sample set includes a plurality of sample images, and the sample images correspond to sample texts;

[0151] Determining the number of preset attributes included in the sample texts in the sample set, and grouping the sample texts and corresponding sample images according to the number of attributes to obtain sample groups;

[0152] The sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups, and the sorted sample groups are used in sequence to train the image generation model to be trained to obtain a target model.

[0153] Although preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this application. Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if such changes and modifications fall within the scope of the claims of this application and their equivalents, then this application is intended to include such changes and modifications.

Claims

1. A training method for an image generation model, comprising: Acquire a sample set corresponding to a target scene, wherein the sample set includes a plurality of sample images, and the sample images correspond to sample texts; Determining the number of attributes of preset attributes included in the sample text in the sample set, and grouping the sample text and the corresponding sample images according to the number of attributes to obtain sample groups; The sample groups are sorted in ascending order according to the number of attributes corresponding to the sample groups, and the sorted sample groups are used in sequence to train the image generation model to be trained to obtain a target model.

2. According to the method of claim 1, the step of obtaining a sample set corresponding to the target scene comprises: Acquire an image set corresponding to the target scene, and text to be filled saved for the target scene, wherein the text to be filled includes at least one target field to which an attribute value is to be written; For any sample image in the image set, determine the target attribute value corresponding to the target field in the sample image, fill the target attribute value into the target field accordingly, and obtain the sample text corresponding to the sample image; The sample text and the corresponding sample image are saved in a sample set.

3. According to the method of claim 2, a plurality of texts to be filled are saved for the target scenario. If the number of target fields included in different texts to be filled is the same, the target attributes corresponding to the target fields included in the same number of texts to be filled are different.

4. The method according to claim 2 or 3, wherein for any sample image in the image set, determining the target attribute value corresponding to the target field in the sample image comprises: Determine a target attribute corresponding to a target field in the text to be filled; For any sample image in the image set, the sample image and the target attribute are output, and a prompt is given to input an attribute value corresponding to the target attribute in the sample image; the received attribute value corresponding to the target attribute is determined as the target attribute value; or, for any sample image in the image set, image recognition is performed on the sample image to obtain the attribute value corresponding to the target attribute in the sample image, and the attribute value is determined as the target attribute value.

5. The method according to claim 4, before outputting the sample image and the target attribute, further comprising: Obtaining candidate attribute values ​​saved for the target attribute; The outputting the sample image and the target attribute comprises: The sample image, the target attribute and the candidate attribute value are output.

6. The method according to claim 4, after obtaining the attribute value corresponding to the target attribute in the sample image and before determining the attribute value as the target attribute value, further comprising: Outputting the sample image and the obtained attribute value corresponding to the target attribute, and prompting to review the attribute value corresponding to the target attribute to obtain a first attribute value corresponding to the target attribute after review; The first attribute value corresponding to the target attribute after review is received, and the attribute value corresponding to the target attribute obtained through image recognition is updated using the first attribute value.

7. The method according to claim 1, wherein the step of sequentially using the sorted sample groups to train the image generation model to be trained comprises: Sequentially obtain the first sample group from the sorted sample groups; If the first sample group is the first group in the sorted sample groups, the first sample group is used to train the image generation model to be trained.

8. The method according to claim 7, further comprising: If the first sample group is not the first group in the sorted sample groups, then obtaining a set number of sample images and corresponding sample texts from other sample groups before the first sample group; The image generation model to be trained is trained using the set number of sample images, the sample texts corresponding to the set number of sample images, and the first sample grouping.

9. The method according to claim 8, wherein obtaining a set number of sample images and corresponding sample texts comprises: Get the pre-saved target ratio; Obtain sample images and corresponding sample texts of a number corresponding to the target ratio in the other sample groups.

10. A method for constructing a sample set, comprising: Acquire an image set corresponding to a target scene, and text to be filled saved for the target scene, wherein the text to be filled includes at least one target field in which an attribute value is to be written; For any sample image in the image set, determine the target attribute value corresponding to the target field in the sample image, fill the target attribute value into the target field accordingly, and obtain the sample text corresponding to the sample image; The sample text and the corresponding sample image are saved in a sample set.

11. According to the method of claim 10, a plurality of texts to be filled are stored for the target scenario, and if different texts to be filled include the same number of target fields, the target attributes corresponding to the target fields included in the same number of texts to be filled are different.

12. The method according to claim 10 or 11, wherein for any sample image in the image set, determining the target attribute value corresponding to the target field in the sample image comprises: Determine a target attribute corresponding to a target field in the text to be filled; For any sample image in the image set, the sample image and the target attribute are output, and a prompt is given to input an attribute value corresponding to the target attribute in the sample image; the received attribute value corresponding to the target attribute is determined as the target attribute value; or, for any sample image in the image set, image recognition is performed on the sample image to obtain the attribute value corresponding to the target attribute in the sample image, and the attribute value is determined as the target attribute value.

13. The method according to claim 12, before outputting the sample image and the target attribute, further comprising: Obtaining candidate attribute values ​​saved for the target attribute; The outputting the sample image and the target attribute comprises: The sample image, the target attribute and the candidate attribute value are output.

14. The method according to claim 12, after obtaining the attribute value corresponding to the target attribute in the sample image and before determining the attribute value as the target attribute value, further comprising: Outputting the sample image and the obtained attribute value corresponding to the target attribute, and prompting to review the attribute value corresponding to the target attribute to obtain a first attribute value corresponding to the target attribute after review; The first attribute value corresponding to the target attribute after review is received, and the attribute value corresponding to the target attribute obtained through image recognition is updated using the first attribute value.

15. A large image generation model, wherein the large image generation model is obtained based on the training method of the image generation model according to any one of claims 1 to 9.

16. An electronic device comprising a processor and a memory, wherein the processor is used to implement the steps of the image generation model training method as described in any one of claims 1 to 9, or the steps of the sample set construction method as described in any one of claims 10 to 14 when executing a computer program stored in the memory.

17. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the training method of the image generation model as described in any one of claims 1 to 9, or the steps of the sample set construction method as described in any one of claims 10 to 14.

Images