Training method of image generation model, image generation method and related equipment

Abstract

The invention provides a training method of an image generation model, the image generation model comprises an image generation module and a layout control module, the output of the layout control module is one input of the image generation module, and the method comprises the following steps: obtaining first training data; training the image generation module by taking the description information of the first training image as the input of the image generation module and taking the first training image as a label; taking the description information of the second training image and the layout image of the second training image as input of a layout control module, and obtaining layout control features output by the layout control module; and training the layout control module by taking the description information of the second training image and the layout control features output by the layout control module as inputs of the trained image generation module and taking the second training image as a label. The image generation model obtained through training comprehensively considers the relevance between the background picture and the foreground object, and layout controllability is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for an image generation model, an image generation method, a training device for an image generation model, an image generation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the image generation scenario, users (e.g., business parties with image generation needs) can use image generation models to generate images. Usually, an image may include multiple foreground objects. When users need to achieve controllable generation of images, they can construct control information for controlling foreground objects (e.g., text with control information or pictures with control information), input the control information into the image generation model, and make the image generation model generate images that meet the control information according to the control information.

[0003] However, in the related art, it is difficult to achieve accurate control of the position of the foreground object in the generated image by relying on concise control information, the correlation between the background image and the foreground object is low, and the image generation quality is not high. Summary of the invention

[0004] The present disclosure provides a method for training an image generation model. The image generation model trained by the method can comprehensively consider the correlation between the background image and the foreground object in the generated image, and at the same time realize the controllable layout to meet the diverse image generation requirements. The present disclosure also provides an image generation method corresponding to the above method, a training device for the image generation model, an image generation device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, the present disclosure provides a training method for an image generation model, wherein the image generation model comprises an image generation module and a layout control module, wherein an output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of a foreground object in the image; the method comprises:

[0006] Acquire first training data; wherein the first training data includes a first training image and description information of the first training image, and the description information of the first training image is used to describe the background image and foreground object of the first training image;

[0007] Using the description information of the first training image as input to the image generation module, and using the first training image as a label to train the image generation module;

[0008] After completing the training of the image generation module, obtaining second training data; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground objects of the second training image, and the layout image of the second training image is used to indicate an area of ​​the second training image that does not include the foreground object;

[0009] Using the description information of the second training image and the layout image of the second training image as inputs of the layout control module, and obtaining layout control features output by the layout control module;

[0010] The description information of the second training image and the layout control features output by the layout control module are used as inputs of the trained image generation module, and the layout control module is trained by using the second training image as a label.

[0011] In a second aspect, the present disclosure provides an image generation method, the method comprising:

[0012] Acquire first description information and a first layout image; wherein the first description information is used to describe the background picture and foreground object of the desired image, and the first layout image is used to indicate an area of ​​the desired image that does not include the foreground object;

[0013] Inputting the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features;

[0014] Obtain the first image output by the image generation model; wherein the image generation model is trained based on a second image, description information of the second image and a second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area of ​​the second image that does not include the foreground object.

[0015] In a third aspect, the present disclosure provides a training device for an image generation model, wherein the image generation model includes an image generation module and a layout control module, wherein an output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of a foreground object in the image; the device includes:

[0016] An acquisition module, used to acquire first training data; wherein the first training data includes a first training image and description information of the first training image, and the description information of the first training image is used to describe the background image and foreground object of the first training image;

[0017] a training module, configured to use the description information of the first training image as an input of the image generation module, and to use the first training image as a label to train the image generation module;

[0018] The acquisition module is further used to acquire second training data after completing the training of the image generation module; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground objects of the second training image, and the layout image of the second training image is used to indicate an area in the second training image that does not include a foreground object;

[0019] a processing module, configured to use the description information of the second training image and the layout image of the second training image as inputs of the layout control module, and obtain layout control features output by the layout control module;

[0020] The training module is further used to use the description information of the second training image and the layout control features output by the layout control module as inputs of the trained image generation module, and to use the second training image as a label to train the layout control module.

[0021] In a fourth aspect, the present disclosure provides an image generating device, the device comprising:

[0022] An acquisition module, configured to acquire first description information and a first layout image; wherein the first description information is used to describe the background image and the foreground object of the desired image, and the first layout image is used to indicate an area of ​​the desired image that does not include the foreground object;

[0023] an input module, configured to input the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features;

[0024] An output module is used to obtain the first image output by the image generation model; wherein the image generation model is trained based on a second image, description information of the second image and a second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area of ​​the second image that does not include the foreground object.

[0025] In a fifth aspect, the present disclosure provides an electronic device, comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the electronic device executes the training method of the image generation model in the first aspect or any implementation of the first aspect, or executes the image generation method in the second aspect or any implementation of the second aspect.

[0026] In a sixth aspect, the present disclosure provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, wherein the instructions instruct an electronic device to execute the training method of an image generation model as described in the first aspect or any one of the implementations of the first aspect, or to execute the image generation method as described in the second aspect or any one of the implementations of the second aspect.

[0027] In a seventh aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the training method of the image generation model described in the first aspect or any one of the implementations of the first aspect, or to execute the image generation method as described in the second aspect or any one of the implementations of the second aspect.

[0028] Based on the implementations provided in the above aspects, the present disclosure can be further combined to provide more implementations.

[0029] It can be seen from the above technical solutions that the present disclosure has the following advantages:

[0030] The present disclosure provides a training method for an image generation model, wherein the image generation model includes an image generation module and a layout control module, wherein the output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of a foreground object in the image. The method first obtains first training data, wherein the first training data includes a first training image and description information of the first training image, the description information of the first training image is used to describe a background image and a foreground object of the first training image, the description information of the first training image is used as an input of the image generation module, and the first training image is used as a label to train the image generation module. After completing the training of the image generation module, second training data is obtained; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground object of the second training image, and the layout image of the second training image is used to indicate the area in the second training image that does not include the foreground object, the description information of the second training image and the layout image of the second training image are used as inputs of the layout control module, the layout control features output by the layout control module are obtained, the description information of the second training image and the layout control features output by the layout control module are used as inputs of the trained image generation module, and the second training image is used as a label to train the layout control module.

[0031] In this method, the image generation module and the layout control module in the image generation model are trained separately in a step-by-step training manner. The image generation module is trained first. Since the description information of the first training image describes both the background image and the foreground object, the one-time generation of the background image and the foreground object in the generated image can be achieved, thereby enhancing the correlation between the background image and the foreground object. After the training of the image generation module is completed, the layout control module is trained in combination with the trained image generation module. Since the layout image of the second training image indicates an area that does not include the foreground object, the area in the generated image where the foreground object cannot be generated can be controlled to achieve controllable layout. In this way, the trained image generation model can comprehensively consider the correlation between the background image and the foreground object in the generated image, while achieving controllable layout and meeting diverse image generation requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical method of the embodiments of the present disclosure, the drawings required for use in the embodiments are briefly introduced below.

[0033] Figure 1 A schematic diagram of the structure of an image generation model provided by an embodiment of the present disclosure;

[0034] Figure 2A flowchart of a method for training an image generation model provided in an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of a layout image provided by an embodiment of the present disclosure;

[0036] Figure 4 A flowchart of an image generation method provided by an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram of the structure of an image generation model provided by an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of a training device for an image generation model provided in an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of the structure of an image generating device provided by an embodiment of the present disclosure;

[0040] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The terms "first" and "second" in the embodiments of the present disclosure are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0042] First, some technical terms and application scenarios involved in the embodiments of the present disclosure are introduced.

[0043] The image generation process can be understood as a process of automatically generating new images. Generally, the industry uses image generation models for image generation. For example, the image generation model can be a diffusion model based on deep learning.

[0044] Typically, an image may include multiple foreground objects, for example, the foreground objects in the image may include people, objects, stickers, etc. When a user (for example, a business party with image generation requirements) needs to achieve controllable generation of an image, control information for controlling the foreground object may be constructed, and the control information may be input into the image generation model, so that the image generation model generates an image that satisfies the control information according to the control information.

[0045] In some examples, the control information may be text with the control information, for example, the control information may be text for describing the position, shape, color, etc. of the foreground object. In other examples, the control information may be a picture with the control information, for example, the control information may be a contour image, a segmentation image, a joint image, a depth map, etc. of the foreground object.

[0046] In the related art, control information can be used to control the position of the foreground object. However, it is difficult for the above method to achieve accurate control of the position of the foreground object in the generated image by relying on concise control information. Specifically, when the control information is text with control information, the image generation model often finds it difficult to understand the text used to describe the position of the foreground object in the control information (for example, "a is to the left of b", "c is located in the lower right corner of the image", etc.), and it is difficult for the image generation model to combine the text with control information to achieve accurate control of the position of the foreground object in the generated image. When the control information is a picture with control information, the accuracy of the control information is required to be high. The picture with control information needs to accurately and meticulously depict the contour, depth and other information of the foreground object, and the control information with a high degree of accuracy comes from high-quality reference images. In the scenario of batch image generation, it is difficult to obtain accurate and large reference images.

[0047] Therefore, in order to effectively control the position of the foreground object in the image, the related art also proposes a multi-step image generation method, that is, generating a background image and a foreground object separately, and then using separate position control information to combine the foreground object on the background image to generate the final image. However, the above method also has the following problems: the multi-step image generation method has a complicated process and the training process of the image generation model is cumbersome. In addition, since the background image and the foreground object are generated separately, the correlation between the background image and the foreground object is low. In addition, the position control information often limits the position of the foreground object, and the foreground object is placed at the position specified by the position control information, and the diversity is limited. How to make the background image and the foreground object in the image generated by the image generation model have a certain correlation, while following certain layout information, has become a problem that the industry urgently needs to solve.

[0048] In view of this, the present disclosure provides a training method for an image generation model, wherein the image generation model includes an image generation module and a layout control module, wherein the output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of foreground objects in the image. The method first obtains first training data, wherein the first training data includes a first training image and description information of the first training image, wherein the description information of the first training image is used to describe the background image and foreground objects of the first training image, the description information of the first training image is used as an input of the image generation module, and the first training image is used as a label to train the image generation module. After completing the training of the image generation module, second training data is obtained; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground object of the second training image, and the layout image of the second training image is used to indicate the area in the second training image that does not include the foreground object, the description information of the second training image and the layout image of the second training image are used as inputs of the layout control module, the layout control features output by the layout control module are obtained, the description information of the second training image and the layout control features output by the layout control module are used as inputs of the trained image generation module, and the second training image is used as a label to train the layout control module.

[0049] In this method, the image generation module and the layout control module in the image generation model are trained separately in a step-by-step training manner. The image generation module is trained first. Since the description information of the first training image describes both the background image and the foreground object, the one-time generation of the background image and the foreground object in the generated image can be achieved, thereby enhancing the correlation between the background image and the foreground object. After the training of the image generation module is completed, the layout control module is trained in combination with the trained image generation module. Since the layout image of the second training image indicates an area that does not include the foreground object, the area in the generated image where the foreground object cannot be generated can be controlled to achieve controllable layout. In this way, the trained image generation model can comprehensively consider the correlation between the background image and the foreground object in the generated image, while achieving controllable layout and meeting diverse image generation requirements.

[0050] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, the following is an explanation with reference to the accompanying drawings. First, an image generation model is introduced. Figure 1 A structural schematic diagram of an image generation model is shown. In the embodiment of the present disclosure, the image generation model may be a diffusion model, and the image generation model may include an image generation module and a layout control module.

[0051] The layout control module is used to control the layout of the foreground objects in the image. The input of the layout control module includes the initial noise image, description information and layout image, and the output of the layout control module is the layout control feature. The image generation module is used to generate an image, for example, to generate an image based on text, or to generate an image based on text and control information. The input of the image generation module includes the initial noise image, description information and layout control feature, and the output of the image generation module is the target image.

[0052] Among them, the initial noise image can be understood as an image composed of initial noise (for example, a random number). Usually, the initial noise image has the same size as the layout image and the target image. The image generation module can use the initial noise image as a reference and generate the target image based on the initial noise image. In this way, the layout control module can extract layout control features based on the initial noise image in combination with the description information and the layout image to increase the controllability of the target image. The image generation module can generate a target image that meets the description information and the layout control features based on the initial noise image in combination with the description information and the layout control features to achieve image generation.

[0053] It should be noted that the image generation model may include more modules in addition to the image generation module and the layout control module. For example, the image generation model may also include a variational autoencoder module, which includes an encoder and a decoder, wherein the encoder is used to downsample the initial noisy image (for example, compressing the 512×512 initial noisy image into a 16×16 image), and the decoder is used to upsample the image output by the image generation module (for example, restoring the 16×16 image output by the image generation module into a 512×512 target image).

[0054] Next, based on Figure 1 The image generation model shown in the figure explains the training process of the image generation model. Figure 2 The flowchart of a training method of an image generation model shown in FIG. 1 is a flowchart of a training method of an image generation model, and the method specifically includes:

[0055] S201: Acquire first training data.

[0056] In the disclosed embodiment, the image generation model is trained in a multi-step training manner. In the first training step, only the image generation module is trained, that is, only the model parameters of the image generation module are adjusted.

[0057] The first training data can be understood as training data for training the image generation module. The first training data may include a first training image and description information of the first training image, and the description information of the first training image may be used to describe the background image and foreground object of the first training image. In the embodiment of the present disclosure, the description information of the first training image may be described in natural language to represent the natural language content, that is, the description information of the first training image may describe the image content of the first training image in the form of natural language.

[0058] The background image can be understood as an image used as the background in the first training image, and the size of the background image can be the same as the size of the first training image. The foreground object can be understood as an object other than the background image in the first training image, and the foreground object can usually include multiple objects. For example, when the description information of the first training image is "the picture has a beach as the background, and there is a dog playing with a ball on the beach", the beach can be the background image, and the dog and the ball can be the foreground objects.

[0059] That is, considering that the image generation module can generate an image based on text, the first training image and the description information of the first training image are selected as the first training data to train the image generation module in a targeted manner.

[0060] S202: Using the description information of the first training image as input to an image generation module, and using the first training image as a label to train the image generation module.

[0061] In the disclosed embodiment, the training of the image generation model may be supervised training, and the label may be understood as the correct output result in the supervised training.

[0062] In a specific implementation, the description information of the first training image is input into the image generation module. The image generation module gradually performs denoising processing on the basis of the initial noisy image, and injects the semantic features of the description information of the first training image during the denoising process to generate an output image that matches the description information of the first training image.

[0063] Next, based on the output image of the image generation module and the first training image as a label, a loss function value is calculated, which can characterize the degree of difference between the output image of the image generation module and the first training image. The model parameters of the image generation module are adjusted to minimize the loss function value, thereby completing a round of training for the image generation module.

[0064] Repeat the above steps, and perform multiple rounds of training in the first training step. The first training image in each round of training is different. When the difference between the output image of the image generation module and the first training image meets the training end condition, the training of the image generation module is ended. In this way, when the input is the description information of the first training image, the image generation module can output an image similar to the first training image. In addition, since the description information of the first training image describes both the background image and the foreground object, the trained image generation module can generate the background image and the foreground object at the same time.

[0065] S203: After completing the training of the image generation module, obtain second training data.

[0066] Completing the training of the image generation module can be understood as completing the first training step. After completing the first training step, the second training step begins. In the second training step, the trained image generation module is used to train the layout control module, that is, to adjust the model parameters of the layout control module.

[0067] The second training data can be understood as training data for training the layout control module. The second training data may include a second training image, description information of the second training image, and a layout image of the second training image. The description information of the second training image may be used to describe the background image and foreground object of the second training image. The layout image of the second training image may be used to indicate an area in the second training image that does not include a foreground object.

[0068] That is to say, in the disclosed embodiment, the layout image is only used to inform the layout control module that the area where the foreground object cannot be generated is not restricted. In this way, the image generation model can freely generate images without restrictions on the specific position, outline, style, and pattern of the foreground object.

[0069] S204: Using the description information of the second training image and the layout image of the second training image as inputs of a layout control module, and obtaining layout control features output by the layout control module.

[0070] The layout control feature can be understood as a feature related to controlling the area where the foreground object is located. In specific implementation, the description information of the second training image and the layout image of the second training image are input to the layout control module, and the layout control module extracts the layout control feature from the layout image of the second training image based on the initial noise image and the description information of the second training image, and outputs the layout control feature.

[0071] S205: Using the description information of the second training image and the layout control features output by the layout control module as inputs of the trained image generation module, and using the second training image as a label to train the layout control module.

[0072] In the disclosed embodiment, the trained image generation module is used in combination with the layout control features output by the layout control module to generate an output image. In specific implementation, the description information and layout control features of the second training image are input to the image generation module, and the image generation module injects the semantic features and layout control features of the description information of the second training image on the basis of the initial noise image to generate an output image that matches the description information of the second training image and matches the layout image.

[0073] Next, based on the output image of the image generation module and the second training image as a label, a loss function value is calculated, which can characterize the degree of difference between the position of the foreground object in the output image of the image generation module and the position of the foreground object in the second training image. The model parameters of the layout control module are adjusted to minimize the loss function value, thereby completing a round of training process for the layout control module.

[0074] Repeat the above steps, and perform multiple rounds of training in the second training step. The second training image in each round of training is different. When the difference between the output image of the image generation module and the second training image meets the training end condition, the training of the layout control module is ended. In this way, when the input is the layout image of the second training image, the image generation module can output an image that meets the layout image of the second training image. In addition, since the layout image of the second training image only indicates the area where the foreground object cannot be generated, there is no restriction on the specific position and specific layout method of the foreground object, thereby improving the flexibility of the image generation model.

[0075] In this method, the image generation module and the layout control module in the image generation model are trained separately in a step-by-step training manner. The image generation module is trained first. Since the description information of the first training image describes both the background image and the foreground object, the one-time generation of the background image and the foreground object in the generated image can be achieved, thereby enhancing the correlation between the background image and the foreground object. After the training of the image generation module is completed, the layout control module is trained in combination with the trained image generation module. Since the layout image of the second training image indicates an area that does not include the foreground object, the area in the generated image where the foreground object cannot be generated can be controlled to achieve controllable layout. In this way, the trained image generation model can comprehensively consider the correlation between the background image and the foreground object in the generated image, while achieving controllable layout and meeting diverse image generation requirements.

[0076] The foregoing article introduces the training method of the image generation model provided in the embodiment of the present disclosure. The following article specifically describes the training data in the image generation model.

[0077] The image generation requirements in different business scenarios are often different. For example, the image styles expected to be generated in different business scenarios may be different, and the types of foreground objects may be different. Therefore, in the embodiments of the present disclosure, the first training image may be related to the business scenario, that is, the first training image may be an image that meets the image generation requirements in the business scenario, so that the image generation module learns the image generation requirements in the business scenario and generates an image that meets the business scenario.

[0078] Considering the training effect on the image generation module, the first training image may be a high-quality image. For example, the first training image may be a high-quality image produced by a designer.

[0079] In some possible implementations, in the training of the image generation module, training data of different sizes may be used for training. Specifically, a target training image set is randomly selected from multiple training image sets, a training image is randomly selected from the target training image set, and the randomly selected training image is determined as the first training image.

[0080] Among them, a training image set includes multiple training images with the same aspect ratio, and the aspect ratios of the training images in different training image sets are different. In other words, the training images are cropped into images with multiple aspect ratios, and the training images with the same aspect ratio are aggregated into one training image set. In the multi-round training process for the image generation module, the target training image set is randomly selected, and the training images in the target training image set are randomly selected. In this way, multi-scale training is performed on the image generation module, so that the image generation module has the ability to generate images with different aspect ratios.

[0081] After determining the first training image, description information of the first training image can be determined based on the first training image. The disclosed embodiment does not limit the method for determining the description information of the first training image. For example, the description information of the first training image can be obtained by manual annotation. For another example, the description information of the first training image can also be obtained by analyzing the first training image using an image analysis model.

[0082] Considering that the image generation model in the embodiment of the present disclosure can be applied to template generation scenarios, such as template generation scenarios for video promotion, the second training data can come from template images made by designers. Specifically, a plurality of original images are obtained, wherein the original images include a placeholder area and a text area, and the text in the text area of ​​the plurality of original images is erased, and the plurality of original images after erasure are determined as the second training images.

[0083] The original image can be understood as a template image made by a designer, which can be used in video promotion. The placeholder area in the original image can be used to place video materials, and the text area in the original image can be used to place promotional copy.

[0084] Since the video material has not been placed in the original image, the placeholder area can be an area that does not include the foreground object. By erasing the text in the text area in the original image, the text area is also changed to an area that only includes the background image and does not include the foreground object. In this way, there is a part of the original image that only includes the background image, and then the second training image is determined.

[0085] After determining the second training image, the description information of the second training image and the layout image of the second training image can be determined. In specific implementation, the description information of the second training image can be determined in a manner similar to that of determining the description information of the first training image, which will not be described in detail here.

[0086] Regarding the process of determining the layout image of the second training image, specifically, for each original image in the multiple original images, the following steps are performed: a blank image with the same image size as the original image is generated, and the area in the blank image corresponding to the placeholder area and text area of ​​the original image is determined as an area that does not include a foreground object, so as to obtain the layout image of the second training image corresponding to the original image.

[0087] Since the layout image does not include background images, foreground objects and other content that is not related to the layout information, for each original image, a blank image is first generated, and then the area corresponding to the placeholder area and the text area in the blank image is determined as an area that does not include foreground objects, and the original image is converted into a layout image.

[0088] In some embodiments, the layout image of the second training image may include a first area of ​​a first color and a second area of ​​a non-first color, the first area being an area including a foreground object, and the second area being an area not including a foreground object.

[0089] like Figure 3 As shown, the layout image 30 of the second training image includes a first area 301 of a first color (white) and a second area 302 of a non-first color (gray). The first area 301 is an area including a foreground object in the second training image, and the second area 302 is an area not including a foreground object in the second training image, such as a placeholder area or a text area after erasing text.

[0090] In this way, different areas are distinguished by different colors in the layout image, and the layout control module extracts layout control features by learning different colors in the layout image, thereby realizing layout control.

[0091] Based on the training method of the image generation model provided above, the embodiment of the present disclosure also provides an image generation method. Figure 4 A schematic flow chart of an image generation method is shown, the method specifically comprising:

[0092] S401: Acquire first description information and a first layout image.

[0093] The first description information is used to describe the background image and the foreground object of the desired image, and the first layout image is used to indicate the area of ​​the desired image that does not include the foreground object.

[0094] That is to say, the user (for example, a business party with image generation needs) only needs to provide the first description information and the first layout image. The first description information is used to inform the image generation model of the content that the image is expected to cover, and the first layout image is used to inform the image generation model of the layout position of the image that is expected. There is no need to provide information describing the specific position, shape, color, etc. of the background image or foreground object.

[0095] The disclosed embodiments do not limit the source of the first description information. For example, the user may extract the first description information from existing advertising materials. For another example, in a template generation scenario for video promotion, the user may also extract the first description information from video materials.

[0096] The disclosed embodiment does not limit the source of the first layout image. For example, the first layout image may be derived from a template image created by a designer. By performing template parsing on the template image, multiple layout images are obtained, and the user may select the first layout image from the multiple layout images.

[0097] S402: Inputting the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features.

[0098] S403: Obtain a first image output by the image generation model.

[0099] Among them, the image generation model can be trained based on the second image, the description information of the second image and the second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area in the second image that does not include the foreground object.

[0100] That is to say, by using the descriptive information of the second image and the second layout image as the input of the image generation model and the second image as a label, the image generation model can extract layout control features in combination with the layout image, and then generate an image in combination with the layout control features and the descriptive information, so that it has the ability to generate an image that satisfies the layout image and the descriptive information.

[0101] In this method, by inputting the first description information describing the desired image and the first layout image controlling the area where the foreground object is located, the image generation model can generate the background image and the foreground object at one time, thereby improving the correlation between the background image and the foreground object. Since the first layout image is used as the input of the image generation model, the position of the foreground object in the first image is controllable, which can be applied to different business scenarios such as video promotion and material packaging.

[0102] Considering the quality of the image generated by the image generation model, the embodiment of the present disclosure can also optimize the image. Figure 5 As shown, the image generation model can be trained by the training method of the image generation model provided in the previous text. The image generation model can also include an optimizer module. The output of the image generation module is an input of the optimizer module. The optimizer module is used to perform refined processing on the image.

[0103] During the inference process of the image generation model, the first description information and the first layout image are used as inputs of a layout control module in the image generation model to obtain a first layout control feature output by the layout control module. The first description information and the first layout control feature are used as inputs of an image generation module in the image generation model to obtain an initial image output by the image generation module. The initial image is refined using an optimizer module in the image generation model to obtain a first image output by the optimizer module.

[0104] The core principle of the optimizer module is to optimize the image by performing denoising. In the disclosed embodiment, the denoising performed by the optimizer module can be divided into multiple steps, in each step, the noise in the image is gradually reduced, the image quality is improved, and the image is more delicate.

[0105] In this way, for the initial image output by the image generation module, the optimizer module performs denoising processing on the initial image, so that the optimizer module outputs a first image with better image quality.

[0106] The disclosed embodiments support image optimization using the optimizer module in different ways. In some embodiments, the first description information and the initial image are used as inputs of the optimizer module in the image generation model, so that the optimizer module performs a first denoising process on the initial image to obtain a first image output by the optimizer module.

[0107] In other embodiments, the initial image is denoised, and the first description information and the initial image after denoising are used as inputs to an optimizer module in an image generation model, so that the optimizer module performs a second denoising process on the initial image after denoising to obtain a first image output by the optimizer module.

[0108] That is to say, the optimizer module can directly perform denoising on the initial image output by the image generation module, or it can first perform denoising on the initial image output by the image generation module, and then use the optimizer module to perform denoising on the initial image after denoising. In this way, in the method of directly performing denoising on the initial image output by the image generation module, the optimizer module reduces the noise in the initial image, so that the first image blurs the background picture and highlights the main object more compared with the initial image. In the method of first performing denoising on the initial image output by the image generation module and then performing denoising, more content is added to the initial image through denoising, so that the first image has more image details and richer image information than the initial image. At the same time, since the input of the optimizer module also includes the first description information, the first image is more matched with the first description information than the initial image, and is more in line with the image generation needs of the user.

[0109] Combination of the above Figures 1 to 5 The training method of the image generation model and the image generation method provided in the embodiments of the present disclosure are introduced in detail. The apparatus and device provided in the embodiments of the present disclosure will be introduced in conjunction with the accompanying drawings.

[0110] See also Figure 6 The structure diagram of the training device of the image generation model shown in the figure, the image generation model includes an image generation module and a layout control module, the output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of the foreground object in the image; the device 60 includes:

[0111] An acquisition module 601 is used to acquire first training data; wherein the first training data includes a first training image and description information of the first training image, and the description information of the first training image is used to describe the background image and foreground object of the first training image;

[0112] A training module 602, configured to use the description information of the first training image as an input of the image generation module, and use the first training image as a label to train the image generation module;

[0113] The acquisition module 601 is further used to acquire second training data after completing the training of the image generation module; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground object of the second training image, and the layout image of the second training image is used to indicate an area in the second training image that does not include a foreground object;

[0114] A processing module 603 is used to use the description information of the second training image and the layout image of the second training image as inputs of the layout control module to obtain layout control features output by the layout control module;

[0115] The training module 602 is further configured to use the description information of the second training image and the layout control features output by the layout control module as inputs of the trained image generation module, and to use the second training image as a label to train the layout control module.

[0116] In some possible implementations, the layout image of the second training image includes a first area of ​​a first color and a second area not of the first color, the first area is an area including a foreground object, and the second area is an area not including a foreground object.

[0117] In some possible implementations, the acquisition module 601 is specifically used to:

[0118] Randomly selecting a target training image set from a plurality of training image sets; wherein one of the training image sets includes a plurality of training images with the same aspect ratio, and the aspect ratios of the training images in different training image sets are different;

[0119] Randomly selecting a training image from the target training image set, and determining the randomly selected training image as a first training image;

[0120] Describe information of the first training image according to the first training image.

[0121] In some possible implementations, the acquisition module 601 is specifically used to:

[0122] Acquire multiple original images; wherein the original images include a placeholder area and a text area;

[0123] Erasing the text in the text area in the multiple original images, and determining the multiple original images after erasing as second training images;

[0124] Describing information of the second training image is determined according to the second training image, and a layout image of the second training image is determined according to the second training image.

[0125] In some possible implementations, the acquisition module 601 is specifically used to:

[0126] For each of the plurality of original images, the following steps are performed:

[0127] generating a blank image having the same image size as the original image;

[0128] The area in the blank image corresponding to the placeholder area and the text area of ​​the original image is determined as an area not including a foreground object, and a layout image of the second training image corresponding to the original image is obtained.

[0129] The training device 60 of the image generation model according to the embodiment of the present disclosure may correspond to the method described in the embodiment of the present disclosure, and the above and other operations and / or functions of each module / unit of the training device 60 of the image generation model are respectively to achieve Figure 2 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0130] See also Figure 7 The schematic diagram of the structure of the image generating device shown in FIG. 70 includes:

[0131] The acquisition module 701 is used to acquire first description information and a first layout image; wherein the first description information is used to describe the background image and the foreground object of the desired image, and the first layout image is used to indicate the area of ​​the desired image that does not include the foreground object;

[0132] An input module 702, configured to input the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background image and a foreground object based on the first description information and the layout control features;

[0133] The output module 703 is used to obtain the first image output by the image generation model; wherein the image generation model is trained based on a second image, description information of the second image and a second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area in the second image that does not include the foreground object.

[0134] In some possible implementations, the image generation is obtained by training using the aforementioned image generation model training method, the image generation model also includes an optimizer module, the output of the image generation module is an input of the optimizer module, and the optimizer module is used to perform refined processing on the image.

[0135] In some possible implementations, the input module 702 is specifically used to:

[0136] Using the first description information and the first layout image as inputs of a layout control module in an image generation model, and obtaining a first layout control feature output by the layout control module;

[0137] Using the first description information and the first layout control feature as inputs of an image generation module in the image generation model to obtain an initial image output by the image generation module;

[0138] The initial image is refined using the optimizer module in the image generation model to obtain a first image output by the optimizer module.

[0139] In some possible implementations, the input module 702 is specifically used to:

[0140] using the first description information and the initial image as inputs of an optimizer module in the image generation model, so that the optimizer module performs a first denoising process on the initial image to obtain a first image output by the optimizer module; or

[0141] The initial image is subjected to denoising processing, and the first description information and the initial image after denoising processing are used as inputs of the optimizer module in the image generation model, so that the optimizer module performs a second denoising processing on the initial image after denoising processing to obtain a first image output by the optimizer module.

[0142] The image generating device 70 according to the embodiment of the present disclosure may correspond to executing the method described in the embodiment of the present disclosure, and the above and other operations and / or functions of each module / unit of the image generating device 70 are respectively to implement Figure 4 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0143] The present disclosure also provides an electronic device. The electronic device is specifically used to implement Figure 6 The function of the training device 60 of the image generation model in the embodiment shown, or, to achieve the following Figure 7 The function of the image generating device 70 in the embodiment shown.

[0144] Figure 8A schematic diagram of the structure of an electronic device 800 is provided. Figure 8 As shown, the electronic device 800 includes a bus 801, a processor 802, a communication interface 803 and a memory 804. The processor 802, the memory 804 and the communication interface 803 communicate with each other via the bus 801.

[0145] The bus 801 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0146] The processor 802 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0147] The communication interface 803 is used for communicating with the outside. For example, the communication interface 803 can be used for communicating with a terminal.

[0148] The memory 804 may include a volatile memory, such as a random access memory (RAM). The memory 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0149] The memory 804 stores executable codes, and the processor 802 executes the executable codes to perform the aforementioned image generation model training method or image generation method.

[0150] Specifically, in implementing Figure 6 or Figure 7 In the case of the embodiment shown, and Figure 6 The training device 60 of the image generation model described in the embodiment or Figure 7When each module or unit of the image generating device 70 described in the embodiment is implemented by software, the execution Figure 6 or Figure 7 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 804. The processor 802 executes the program code corresponding to each unit stored in the memory 804 to execute the training method or image generation method of the aforementioned image generation model.

[0151] The present disclosure embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the training method of the image generation model of the training device 60 applied to the image generation model, or execute the image generation method applied to the image generation device 70.

[0152] The embodiment of the present disclosure also provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present disclosure is generated in whole or in part.

[0153] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0154] When the computer program product is executed by a computer, the computer executes any of the aforementioned image generation model training methods or image generation methods. The computer program product may be a software installation package, and when it is necessary to use any of the aforementioned image generation model training methods or image generation methods, the computer program product may be downloaded and executed on a computer.

[0155] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0157] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0158] In the context of the disclosed embodiments, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0160] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0161] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0162] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0163] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for an image generation model, characterized in that: The image generation model comprises an image generation module and a layout control module, wherein the output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of foreground objects in the image; the method comprises: Acquire first training data; wherein the first training data includes a first training image and description information of the first training image, and the description information of the first training image is used to describe the background image and foreground object of the first training image; Using the description information of the first training image as input to the image generation module, and using the first training image as a label to train the image generation module; After completing the training of the image generation module, obtaining second training data; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground objects of the second training image, and the layout image of the second training image is used to indicate an area of ​​the second training image that does not include the foreground object; Using the description information of the second training image and the layout image of the second training image as inputs of the layout control module, and obtaining layout control features output by the layout control module; The description information of the second training image and the layout control features output by the layout control module are used as inputs of the trained image generation module, and the layout control module is trained by using the second training image as a label.

2. The method according to claim 1, characterized in that: The layout image of the second training image includes a first area of ​​a first color and a second area of ​​a color other than the first color, the first area is an area including a foreground object, and the second area is an area not including a foreground object.

3. The method according to claim 1, characterized in that The obtaining of first training data comprises: Randomly selecting a target training image set from a plurality of training image sets; wherein one of the training image sets includes a plurality of training images with the same aspect ratio, and the aspect ratios of the training images in different training image sets are different; Randomly selecting a training image from the target training image set, and determining the randomly selected training image as a first training image; Describe information of the first training image according to the first training image.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the second training data comprises: Acquire multiple original images; wherein the original images include a placeholder area and a text area; Erasing the text in the text area in the multiple original images, and determining the multiple original images after erasing as second training images; Describing information of the second training image is determined according to the second training image, and a layout image of the second training image is determined according to the second training image.

5. The method according to claim 4, characterized in that The step of determining the layout image of the second training image according to the second training image includes: For each of the plurality of original images, the following steps are performed: generating a blank image having the same image size as the original image; The area in the blank image corresponding to the placeholder area and the text area of ​​the original image is determined as an area not including a foreground object, and a layout image of the second training image corresponding to the original image is obtained.

6. An image generation method, characterized in that: The method comprises: Acquire first description information and a first layout image; wherein the first description information is used to describe the background picture and foreground object of the desired image, and the first layout image is used to indicate an area of ​​the desired image that does not include the foreground object; Inputting the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features; Obtain the first image output by the image generation model; wherein the image generation model is trained based on a second image, description information of the second image and a second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area of ​​the second image that does not include the foreground object.

7. The method according to claim 6, characterized in that The image generation model is trained by the image generation model training method described in any one of claims 1 to 5, and the image generation model also includes an optimizer module, the output of the image generation module is an input of the optimizer module, and the optimizer module is used to perform refined processing on the image.

8. The method according to claim 7, characterized in that The step of inputting the first description information and the first layout image into an image generation model so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features, comprises: Using the first description information and the first layout image as inputs of a layout control module in an image generation model, and obtaining a first layout control feature output by the layout control module; Using the first description information and the first layout control feature as inputs of an image generation module in the image generation model to obtain an initial image output by the image generation module; The initial image is refined using the optimizer module in the image generation model to obtain a first image output by the optimizer module.

9. The method according to claim 8, characterized in that The step of performing fine processing on the initial image by using the optimizer module in the image generation model to obtain a first image output by the optimizer module includes: using the first description information and the initial image as inputs of an optimizer module in the image generation model, so that the optimizer module performs a first denoising process on the initial image to obtain a first image output by the optimizer module; or The initial image is subjected to denoising processing, and the first description information and the initial image after denoising processing are used as inputs of the optimizer module in the image generation model, so that the optimizer module performs a second denoising processing on the initial image after denoising processing to obtain a first image output by the optimizer module.

10. A training device for an image generation model, characterized in that: The image generation model comprises an image generation module and a layout control module, wherein the output of the layout control module is an input of the image generation module, the image generation module is used to generate an image, and the layout control module is used to control the layout of foreground objects in the image; the device comprises: An acquisition module, used to acquire first training data; wherein the first training data includes a first training image and description information of the first training image, and the description information of the first training image is used to describe the background image and foreground object of the first training image; a training module, configured to use the description information of the first training image as an input of the image generation module, and to use the first training image as a label to train the image generation module; The acquisition module is further used to acquire second training data after completing the training of the image generation module; wherein the second training data includes a second training image, description information of the second training image, and a layout image of the second training image, the description information of the second training image is used to describe the background image and foreground objects of the second training image, and the layout image of the second training image is used to indicate an area in the second training image that does not include a foreground object; a processing module, configured to use the description information of the second training image and the layout image of the second training image as inputs of the layout control module, and obtain layout control features output by the layout control module; The training module is further used to use the description information of the second training image and the layout control features output by the layout control module as inputs of the trained image generation module, and to use the second training image as a label to train the layout control module.

11. An image generating device, characterized in that: The device comprises: An acquisition module, configured to acquire first description information and a first layout image; wherein the first description information is used to describe the background image and the foreground object of the desired image, and the first layout image is used to indicate an area of ​​the desired image that does not include the foreground object; an input module, configured to input the first description information and the first layout image into an image generation model, so that the image generation model extracts layout control features based on the first layout image, and generates a first image including a background picture and a foreground object based on the first description information and the layout control features; An output module is used to obtain the first image output by the image generation model; wherein the image generation model is trained based on a second image, description information of the second image and a second layout image, the description information of the second image is used to describe the background picture and foreground object of the second image, and the second layout image is used to indicate the area of ​​the second image that does not include the foreground object.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 5 or claims 6 to 9.

13. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 5 or claims 6 to 9.

14. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method of any one of claims 1 to 5 or claims 6 to 9.

Images