Image generation and model training method, device and equipment based on large model

Abstract

The invention provides an image generation and model training method, device and equipment based on a large model, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to AIGC content generation and other scenes based on artificial intelligence. The image generation method based on the large model comprises the steps of performing feature extraction on a foreground image corresponding to foreground content in an original image to obtain foreground features; performing image-text processing on the reference image to obtain global features and text features; the reference image is obtained based on a target style of background content to be generated; and generating a target image comprising the foreground content and the background content by adopting a large model based on the foreground features, the global features and the text features.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence, and specifically to a large-model-based image generation method, device, equipment, medium, and product. Background Art

[0002] Artificial Intelligence Generated Content (AIGC) refers to a technology that generates relevant content with appropriate generalization capabilities through learning and identifying existing data based on artificial intelligence technologies such as generative adversarial networks and large pre-trained models. Summary of the invention

[0003] The present disclosure provides a large model-based image generation method, device, equipment, medium and product.

[0004] According to one aspect of the present disclosure, a large model-based image generation method is provided, comprising: extracting features of a foreground image corresponding to foreground content in an original image to obtain foreground features; performing image-text processing on a reference image to obtain global features and text features; the reference image is obtained based on a target style of background content to be generated; and using a large model to generate a target image including the foreground content and the background content based on the foreground features, the global features and the text features.

[0005] According to another aspect of the present disclosure, an image generation model training method is provided, including: performing feature extraction on a foreground image corresponding to foreground content in an original image sample to obtain foreground features; performing image and text processing on a reference image sample to obtain global features and text features; the reference image sample is acquired based on a target style of a background content sample; using an image generation model to generate a predicted image based on the foreground features, the global features and the text features; constructing a loss function based on the predicted image; and adjusting model parameters of the image generation model based on the loss function.

[0006] According to another aspect of the present disclosure, there is provided an image generation device based on a large model, comprising: a first extraction module, used to perform feature extraction on a foreground image corresponding to the foreground content in an original image to obtain foreground features; a second extraction module, used to perform image and text processing on a reference image to obtain global features and text features; the reference image is obtained based on a target style of background content to be generated; a generation module, used to generate a target image including the foreground content and the background content based on the foreground features, the global features and the text features using a large model.

[0007] According to another aspect of the present disclosure, there is provided an image generation model training device, comprising: a first extraction module, used to perform feature extraction on a foreground image corresponding to foreground content in an original image sample to obtain foreground features; a second extraction module, used to perform image and text processing on a reference image sample to obtain global features and text features; the reference image sample is acquired based on a target style of a background content sample; a prediction module, used to generate a predicted image based on the foreground features, the global features and the text features using an image generation model; a construction module, used to construct a loss function based on the predicted image; and an adjustment module, used to adjust model parameters of the image generation model based on the loss function.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0011] According to the embodiments of the present disclosure, an image with a target style background can be efficiently obtained.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of an overall system for implementing an embodiment of the present disclosure;

[0016] Figure 3 is an overall schematic diagram of an image generation process provided according to an embodiment of the present disclosure;

[0017] Figure 4 is a structural schematic diagram of an image generation model provided according to an embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0021] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0022] Fig. 9 It is a schematic diagram of an electronic device used to implement the image generation method or image generation model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Regarding the background in the image, in the related art, professional designers usually generate the required background through professional software. However, this manual design method has the problems of low efficiency and high cost.

[0025] In order to efficiently generate an image with a desired style background, the present disclosure provides the following embodiments.

[0026] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure. This embodiment provides an image generation method based on a large model, such as Figure 1 As shown, the method includes:

[0027] 101. Perform feature extraction on the foreground image corresponding to the foreground content in the original image to obtain foreground features.

[0028] 102. Perform image and text processing on the reference image to obtain global features and text features; the reference image is obtained based on the target style of the background content to be generated.

[0029] 103. Using a large model, based on the foreground features, the global features and the text features, generate a target image including the foreground content and the background content.

[0030] The image can be divided into a foreground part and a background part. The foreground part refers to the main part of the image, for example, in a face scene, the foreground part can be a person, and in a product scene, the foreground part is the product part; the background part refers to the part behind the foreground part, such as the curtain, scenery, etc. behind the main part.

[0031] The original image refers to the image to be processed, specifically, the foreground portion is retained and the background portion is replaced.

[0032] Reference images refer to images that serve as style references. Style refers to the overall visual characteristics and artistic expression of an image, which includes the combined effects of multiple visual elements such as the image's color matching, texture, and composition.

[0033] The original image and the reference image may be input by the user.

[0034] For the original image, after the original image is obtained, foreground segmentation can be performed on the original image to obtain a foreground image corresponding to the foreground part.

[0035] Foreground segmentation refers to the process of distinguishing the foreground and background parts of an image. Through foreground segmentation, an image of the foreground part can be obtained while ignoring the irrelevant background part. That is, the foreground image contains the foreground part of the original image, while the background part can be replaced with a preset pixel value.

[0036] Foreground segmentation can adopt various related technologies, such as threshold-based segmentation, edge-based segmentation, and pre-trained deep learning models, so as to obtain a foreground image by performing foreground segmentation on the original image.

[0037] After obtaining the foreground image, a pre-trained image feature extraction network can be used to extract features from the foreground image to obtain foreground features.

[0038] For the reference image, after obtaining the reference image, the reference image is processed by image and text to obtain global features and text features.

[0039] Specifically, the style of the background content to be generated may be referred to as a target style. After the target style is determined, an image having the target style may be obtained as a reference image, and then subsequent processing may be performed on the reference image.

[0040] Image and text processing refers to the processing related to images and texts.

[0041] The image correlation processing may include: extracting features from the reference image, and the obtained image features are called global features. Specifically, a pre-trained image feature extraction network may be used to extract features from the reference image to obtain global features. The feature extraction networks used for the reference image and the foreground image may be the same or different.

[0042] The text-related processing may include: extracting text corresponding to the reference image, performing feature extraction on the text, and obtaining text features. Specifically, a pre-trained text extraction model may be used to perform text extraction on the reference image to obtain the corresponding text, and then a pre-trained text feature extraction network may be used to perform feature extraction on the text to obtain text features.

[0043] After obtaining the foreground features, global features and text features, a large model can be used to process these features to obtain the target image.

[0044] The large model refers to the Large Language Model (LLM). LLM is a hot technology in the AI ​​field in recent years. LLM is a natural language processing model based on deep learning. It has a huge number of parameters and a complex structure, which enables it to process and understand a large amount of natural language data. Through the process of pre-training and fine-tuning, the LLM model can perform well in a variety of natural language processing tasks, including but not limited to text generation, language understanding, machine translation, etc.

[0045] The target image is the final generated image, which contains the foreground part of the original image and the background part with the target style of the reference image.

[0046] Specifically, foreground features, global features, and text features can be input into the large model (LLM), and the output is the target image.

[0047] In this embodiment, image generation is performed based on the original image and the reference image. The generated target image has the foreground portion of the original image, and the background portion is consistent with the style of the reference image. Compared with the manual design method, it can improve efficiency and reduce costs, thereby efficiently generating images with the desired style background. Moreover, processing based on a large model can utilize the excellent performance of the large model to further improve the image generation effect.

[0048] Figure 2 It is a schematic diagram of an overall system for implementing an embodiment of the present disclosure.

[0049] like Figure 2 As shown, the overall system includes: a user terminal 201 and a server 202.

[0050] The user terminal 201 may include: a personal computer (PC), a laptop, a mobile phone or other mobile devices, and the server 202 may be a local server or a cloud server, and may be a single server or a cluster server.

[0051] A client is deployed on the user terminal 201 , and a server is deployed on the server 202 .

[0052] The user can send the original image and the reference image to the server through the client; the server generates the image based on the original image and the reference image to obtain the target image, the foreground part of the target image is the foreground image of the original image, and the style of the background image of the target image is the style of the reference image. After the server obtains the target image, it can send the target image to the client and display it to the user through the client.

[0053] The server can generate images based on a pre-trained image generation model.

[0054] Figure 3 It is an overall schematic diagram of the image generation process provided according to an embodiment of the present disclosure.

[0055] like Figure 3 As shown, the user can input an original image 301 and a reference image 302. Taking a commodity scene as an example, the foreground portion of the original image is the commodity. The style of the reference image is the style of the background portion required by the user.

[0056] After the server obtains the original image, the original image can be input into the foreground segmenter, which performs foreground segmentation on the original image, and the output is the foreground image 303. The foreground image is input into the pre-trained foreground feature extractor, which extracts features from the foreground image, and the output is the foreground feature.

[0057] After the server obtains the reference image, it can input the reference image into the image feature extractor. The image feature extractor extracts features from the reference image, and the output is the global feature.

[0058] After the server obtains the reference image, it can also input the reference image into the text extractor, which extracts text from the reference image and outputs the corresponding text, such as "a magnificent landscape photo of mountains and rivers at sunset". After that, the text is input into the text feature extractor, which extracts features from the text and outputs text features.

[0059] The foreground segmenter, foreground feature extractor, image feature extractor, text extractor, and text feature extractor mentioned above can all be pre-trained deep neural network models.

[0060] After obtaining the foreground features, global features and text features mentioned above, these features are input into the image generation model.

[0061] Taking the diffusion model as an example, the model input includes noise. The noise can be randomly generated by the server, and its dimension is consistent with the dimension of the foreground feature, the global feature and the text feature, and can be a vector. Therefore, the noise can be represented by the noise feature 304.

[0062] After randomly generating noise features, the noise features are input into the image generation model.

[0063] The image generation model processes the foreground features, noise features, text features and global features, and outputs a target image 305. The foreground portion of the target image is the foreground portion of the original image, and the background image of the target image has the target style of the reference image.

[0064] The image generation model may specifically adopt a large model, for example, the diffusion model mentioned above is a large model (LLM).

[0065] For the large model, the large model performs related operations according to prompt information. In this embodiment, the prompt information specifically includes text corresponding to the reference image.

[0066] Figure 4 It is a structural diagram of an image generation model provided according to an embodiment of the present disclosure.

[0067] The input end of the image generation model can include a fusion module for fusing noise features and foreground features, such as Figure 4 As shown, the fusion module may specifically be an addition module, that is, the noise feature and the foreground feature are added to obtain the fusion feature.

[0068] like Figure 4 As shown, the image generation model includes: a self-attention network 401 and a cross-attention network, and the cross-attention network includes: a first cross-attention network 402 and a second cross-attention network 403.

[0069] After obtaining the fused features, the fused features are input into the self-attention network 401, and the self-attention network 401 performs self-attention processing on the fused features, and the output is the features after self-attention processing.

[0070] The features after self-attention processing and the text features are input into the first cross-attention network 402. The first cross-attention network 402 performs cross-attention processing on the text features and the features after self-attention processing, and the output is the first cross-attention processed features.

[0071] The first cross-attention processed features and the global features are input into the second cross-attention network 403, and the second cross-attention network 403 performs cross-attention processing on the global features and the first cross-attention processed features, and the output is the second cross-attention processed features.

[0072] After obtaining the second cross-attention processed feature, it can also be added to the first cross-attention processed feature, and the added feature can be called the target feature.

[0073] The above-mentioned cross-attention processing takes the example of processing text features first and then global features. It can also be processing global features first and then text features. For example, a first cross-attention processing is performed on the features after self-attention processing and the global features, and then a second cross-attention processing is performed on the features after the first cross-attention processing and the text features.

[0074] The above example takes the target feature obtained based on the first cross-attention processed feature and the second cross-attention processed feature as an example. The second cross-attention processed feature can also be directly used as the target feature.

[0075] After obtaining the target feature, the target feature can be converted into an image. For example, the image generation model can also include a decoding network (not shown), the decoding network includes, for example, a fully connected network, whose input is the target feature and whose output is the target image.

[0076] In this way, after the image generation model processes the input noise features, foreground features, text features, and global features, the output is the target image.

[0077] In combination with the above application scenarios, the present disclosure also provides the following embodiments.

[0078] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure, and this embodiment provides an image generation method. Figure 5 As shown, the method includes:

[0079] 501. Perform foreground segmentation on an original image to obtain a foreground image corresponding to the foreground content in the original image.

[0080] 502. Perform feature extraction on the foreground image to obtain foreground features.

[0081] For example, combined with Figure 3 , a pre-trained foreground segmentor can be used to perform foreground segmentation on the original image to obtain a foreground image, and then a pre-trained foreground feature extractor can be used to perform feature extraction to obtain foreground features.

[0082] In this embodiment, the foreground segmentation is first performed on the original image, and then the feature extraction is performed on the obtained foreground image to obtain the foreground feature, which can improve the accuracy of the foreground feature and further improve the accuracy of the generated image.

[0083] 503. Perform feature extraction on the reference image to obtain global features; the reference image is obtained based on the target style of the background content to be generated.

[0084] 504. Perform text extraction on the reference image to obtain text corresponding to the reference image, and perform feature extraction on the text to obtain text features.

[0085] For example, combined with Figure 3 , a pre-trained image feature extractor can be used to extract features from a reference image to obtain global features; a pre-trained text extractor can be used to extract text from a reference image to obtain corresponding text, and then a pre-trained text feature extractor can be used to extract features from the text to obtain text features.

[0086] In this embodiment, by extracting the global features of the reference image and the text features of the text corresponding to the reference image, since the global features are image features, the image features and text features can be combined for subsequent processing to improve the comprehensiveness of the information and thereby improve the image generation effect.

[0087] 505. Using a large model, based on the foreground features, the global features, and the text features, generate a target image including the foreground content and the background content.

[0088] Specifically, a fusion feature is obtained based on the foreground feature; self-attention processing is performed on the fusion feature to obtain a self-attention processed feature; cross-attention processing is performed on the self-attention processed feature, the global feature and the text feature to obtain a target feature; and the target feature is converted into the target image.

[0089] In this embodiment, by combining self-attention and cross-attention, the feature representation capability can be improved, thereby improving the quality and effect of the target image.

[0090] In some embodiments, the foreground feature and the pre-generated noise feature may be added to obtain the fused feature.

[0091] For example, refer to Figure 4 , the foreground features and noise features can be added to obtain the fused features.

[0092] In this embodiment, the fusion feature is obtained by adding the foreground feature and the noise feature, so that the fusion feature can be obtained simply and efficiently.

[0093] In some embodiments, a first cross-attention processing is performed on the self-attention processed features and the text features to obtain first cross-attention processed features; a second cross-attention processing is performed on the first cross-attention processed features and the global features to obtain second cross-attention processed features; and based on the second cross-attention processed features, the target features are obtained.

[0094] For example, refer to Figure 4 After obtaining the fused features, first perform self-attention processing on the fused features, then perform the first cross-attention processing on the features after self-attention processing and the text features, and then perform the second cross-attention processing on the features after the first cross-attention processing and the global features to obtain the features after the second cross-attention processing.

[0095] In this embodiment, by first performing cross-attention processing on text features and then performing cross-attention processing on global features, the processing effect can be improved, thereby improving the target image effect.

[0096] In some embodiments, the first cross-attention processed feature and the second cross-attention processed feature may be added to obtain the target feature.

[0097] For example, refer to Figure 4 , obtain the second cross-attention processed feature, add it to the first cross-attention processed feature to obtain the target feature.

[0098] In this embodiment, the target feature is obtained by adding the first cross-attention processed feature and the second cross-attention processed feature, and the target feature can be obtained based on the residual calculation method to further improve the accuracy of the target feature and thus improve the target image effect.

[0099] Figure 6 is a schematic diagram according to the third embodiment of the present disclosure, which provides an image generation model training method, such as Figure 6 As shown, the method includes:

[0100] 601. Perform feature extraction on a foreground image corresponding to the foreground content in the original image sample to obtain a foreground feature.

[0101] 602. Perform image and text processing on the reference image sample to obtain global features and text features; the reference image sample is obtained based on the target style of the background content sample.

[0102] 603. Use an image generation model to generate a predicted image based on the foreground feature, the global feature, and the text feature.

[0103] 604. Construct a loss function based on the predicted image.

[0104] 605. Adjust model parameters of the image generation model based on the loss function.

[0105] Among them, original image samples, reference image samples and background content samples can be obtained from existing data sets.

[0106] After obtaining the original image samples and the reference image samples, the predicted image can be obtained by similarly processing the original image and the reference image.

[0107] After obtaining the predicted image, a loss function can be constructed based on the predicted image.

[0108] For example, a true value image may also be obtained, which may be obtained by manual annotation, etc. The foreground content of the true value image is the foreground content in the original image sample, and the background content of the true value image is the above-mentioned background content sample.

[0109] After obtaining the true value image, a loss function can be constructed based on the predicted image and the true value image, such as a loss function that characterizes the error between the predicted image and the true value image.

[0110] The above loss function is constructed based on the predicted image and the true value image. The loss function can also be constructed in other unsupervised or self-supervised ways.

[0111] After the loss function is constructed, the model parameters can be adjusted based on the loss function to minimize the loss function. For example, the model parameters can be adjusted by back propagation (BP) until a preset condition is met, such as reaching a preset number of times or the model converges.

[0112] The model that meets the preset conditions is used as the final image generation model, and the final image generation model can be used in the above-mentioned reasoning stage, that is, image generation is performed according to the image generation model.

[0113] Specifically, in the inference stage, an image generation model can be used to process the original image and reference image input by the user to obtain a target image.

[0114] In addition, the training process can be divided into a pre-training phase and a fine-tuning phase.

[0115] In the pre-training stage, general domain samples may be used for training, that is, original image samples and reference image samples may include samples from multiple domains.

[0116] In the fine-tuning stage, samples of specific fields can be used for training, that is, the original image samples and the reference image samples are samples of specific fields, and the specific fields are the fields required by users, such as the e-commerce field, or the transportation field, etc.

[0117] In addition, the training process is to adjust the model parameters of the image generation model. Combined with the above application process, the overall process may also involve other models, such as foreground segmenter, foreground feature extractor, image feature extractor, text extractor, and text feature extractor. When training the image generation model, the model parameters of these other models can remain unchanged.

[0118] In this embodiment, an image generation model is trained based on original image samples and reference image samples. Afterwards, the image generation model can be used to generate a target image, which has a foreground portion of the original image and a background portion that is consistent with the style of the reference image. This allows images with a desired style and background to be generated efficiently.

[0119] In some embodiments, the image generation model includes: a self-attention network and a cross-attention network;

[0120] The adopting the image generation model to process the foreground feature, the global feature and the text feature to obtain a predicted image includes:

[0121] Acquire fusion features based on the foreground features;

[0122] Using the self-attention network, performing self-attention processing on the fused features to obtain self-attention processed features;

[0123] Using the cross attention network, cross attention processing is performed on the self-attention processed features, the global features and the text features to obtain target features;

[0124] The target features are converted into the target image.

[0125] In this embodiment, by combining self-attention and cross-attention, the feature representation capability can be improved, thereby improving the quality and effect of the target image.

[0126] In some embodiments, the cross-attention network includes: a first cross-attention network and a second cross-attention network;

[0127] The cross attention network is used to perform cross attention processing on the self-attention processed features, the global features and the text features to obtain target features, including:

[0128] Using the first cross attention network, performing a first cross attention process on the self-attention processed features and the text features to obtain first cross attention processed features;

[0129] Using the second cross attention network, performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature;

[0130] Based on the second cross-attention processed features, the target features are obtained.

[0131] In this embodiment, by first performing cross-attention processing on text features and then performing cross-attention processing on global features, the processing effect can be improved, thereby improving the target image effect.

[0132] Figure 7 It is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides an image generation device based on a large model. The device 700 includes: a first extraction module 701, a second extraction module 702 and a generation module 703.

[0133] The first extraction module 701 is used to extract features from the foreground image corresponding to the foreground content in the original image to obtain foreground features; the second extraction module 702 is used to perform image and text processing on the reference image to obtain global features and text features; the reference image is obtained based on the target style of the background content to be generated; the generation module 703 is used to use a large model to generate a target image including the foreground content and the background content based on the foreground features, the global features and the text features.

[0134] In this embodiment, image generation is performed based on the original image and the reference image. The generated target image has the foreground portion of the original image, and the background portion is consistent with the style of the reference image. Compared with the manual design method, it can improve efficiency and reduce costs, thereby efficiently generating images with the desired style background. Moreover, processing based on a large model can utilize the excellent performance of the large model to further improve the image generation effect.

[0135] In some embodiments, the generating module 703 is further used to:

[0136] Acquire fusion features based on the foreground features;

[0137] Performing self-attention processing on the fused features to obtain self-attention processed features;

[0138] Performing cross attention processing on the self-attention processed features, the global features, and the text features to obtain target features;

[0139] The target features are converted into the target image.

[0140] In this embodiment, by combining self-attention and cross-attention, the feature representation capability can be improved, thereby improving the quality and effect of the target image.

[0141] In some embodiments, the generating module 703 is further used to:

[0142] Performing a first cross attention process on the self-attention processed feature and the text feature to obtain a first cross attention processed feature;

[0143] Performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature;

[0144] Based on the second cross-attention processed features, the target features are obtained.

[0145] In this embodiment, by first performing cross-attention processing on text features and then performing cross-attention processing on global features, the processing effect can be improved, thereby improving the target image effect.

[0146] In some embodiments, the generating module 703 is further used to:

[0147] The first cross-attention processed feature and the second cross-attention processed feature are added to obtain the target feature.

[0148] In this embodiment, the target feature is obtained by adding the first cross-attention processed feature and the second cross-attention processed feature, and the target feature can be obtained based on the residual calculation method to further improve the accuracy of the target feature and thus improve the target image effect.

[0149] In some embodiments, the generating module 703 is further used to:

[0150] The foreground feature and the pre-generated noise feature are added to obtain the fused feature.

[0151] In this embodiment, the fusion feature is obtained by adding the foreground feature and the noise feature, so that the fusion feature can be obtained simply and efficiently.

[0152] In some embodiments, the second extraction module 702 is further configured to:

[0153] Performing feature extraction on the reference image to obtain the global feature;

[0154] Performing text extraction on the reference image to obtain text corresponding to the reference image, and performing feature extraction on the text to obtain the text feature.

[0155] In this embodiment, by extracting the global features of the reference image and the text features of the text corresponding to the reference image, since the global features are image features, the image features and text features can be combined for subsequent processing to improve the comprehensiveness of the information and thereby improve the image generation effect.

[0156] In some embodiments, the first extraction module 703 is further used to:

[0157] Performing foreground segmentation on the original image to obtain a foreground image corresponding to the foreground content;

[0158] Feature extraction is performed on the foreground image to obtain the foreground feature.

[0159] In this embodiment, the foreground segmentation is first performed on the original image, and then the feature extraction is performed on the obtained foreground image to obtain the foreground feature, which can improve the accuracy of the foreground feature and further improve the accuracy of the generated image.

[0160] Figure 8 It is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides an image generation model training device, and the device 800 includes: a first extraction module 801, a second extraction module 802, a prediction module 803, a construction module 804 and an adjustment module 805.

[0161] The first extraction module 801 is used to extract features of the foreground image corresponding to the foreground content in the original image sample to obtain foreground features; the second extraction module 802 is used to perform image and text processing on the reference image sample to obtain global features and text features; the reference image sample is obtained based on the target style of the background content sample; the prediction module 803 is used to use the image generation model to generate a predicted image based on the foreground features, the global features and the text features; the construction module 804 is used to construct a loss function based on the predicted image; the adjustment module 805 is used to adjust the model parameters of the image generation model based on the loss function.

[0162] In this embodiment, an image generation model is trained based on original image samples and reference image samples. Afterwards, the image generation model can be used to generate a target image, which has a foreground portion of the original image and a background portion that is consistent with the style of the reference image. This allows images with a desired style and background to be generated efficiently.

[0163] In some embodiments, the image generation model includes: a self-attention network and a cross-attention network;

[0164] The prediction module 803 is further used for:

[0165] Acquire fusion features based on the foreground features;

[0166] Using the self-attention network, performing self-attention processing on the fused features to obtain self-attention processed features;

[0167] Using the cross attention network, cross attention processing is performed on the self-attention processed features, the global features and the text features to obtain target features;

[0168] The target features are converted into the target image.

[0169] In this embodiment, by combining self-attention and cross-attention, the feature representation capability can be improved, thereby improving the quality and effect of the target image.

[0170] In some embodiments, the cross-attention network includes: a first cross-attention network and a second cross-attention network;

[0171] The prediction module 803 is further used for:

[0172] Using the first cross attention network, performing a first cross attention process on the self-attention processed features and the text features to obtain first cross attention processed features;

[0173] Using the second cross attention network, performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature;

[0174] Based on the second cross-attention processed features, the target features are obtained.

[0175] In this embodiment, by first performing cross-attention processing on text features and then performing cross-attention processing on global features, the processing effect can be improved, thereby improving the target image effect.

[0176] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0177] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0178] It is understandable that unless there is any special limitation on the sequence of steps in the process, it means that the timing relationship between these steps is not limited.

[0179] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0180] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0181] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0182] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 909 is also connected to the bus 904.

[0183] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0184] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as an image generation method or an image generation model training method. For example, in some embodiments, the image generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the image generation method or the image generation model training method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the image generation method or the image generation model training method in any other suitable manner (eg, by means of firmware).

[0185] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable task processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0190] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an image based on a large model, comprising: Extracting features of a foreground image corresponding to the foreground content in the original image to obtain foreground features; Perform image and text processing on the reference image to obtain global features and text features; The reference image is obtained based on a target style of the background content to be generated; A large model is adopted to generate a target image including the foreground content and the background content based on the foreground feature, the global feature and the text feature.

2. The method according to claim 1, wherein: The step of acquiring a target image based on the foreground feature, the global feature and the text feature includes: Acquire fusion features based on the foreground features; Performing self-attention processing on the fused features to obtain self-attention processed features; Performing cross attention processing on the self-attention processed features, the global features, and the text features to obtain target features; The target features are converted into the target image.

3. The method according to claim 2, wherein: The performing cross attention processing on the self-attention processed feature, the global feature and the text feature to obtain the target feature includes: Performing a first cross attention process on the self-attention processed feature and the text feature to obtain a first cross attention processed feature; Performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature; Based on the second cross-attention processed features, the target features are obtained.

4. The method according to claim 3, wherein: The acquiring the target feature based on the second cross-attention processed feature includes: The first cross-attention processed feature and the second cross-attention processed feature are added to obtain the target feature.

5. The method according to claim 2, wherein: The acquiring of fusion features based on the foreground features includes: The foreground feature and the pre-generated noise feature are added to obtain the fused feature.

6. The method according to claim 1, wherein: The image-text processing of the reference image with the target style to obtain global features and text features includes: Performing feature extraction on the reference image to obtain the global feature; Performing text extraction on the reference image to obtain text corresponding to the reference image, and performing feature extraction on the text to obtain the text feature.

7. The method according to claim 1, wherein: The step of extracting features of a foreground image corresponding to the foreground content in the original image to obtain foreground features includes: Performing foreground segmentation on the original image to obtain a foreground image corresponding to the foreground content; Feature extraction is performed on the foreground image to obtain the foreground feature.

8. A method for training an image generation model, comprising: Extracting features of a foreground image corresponding to the foreground content in the original image sample to obtain foreground features; Perform image and text processing on the reference image samples to obtain global features and text features; The reference image sample is obtained based on the target style of the background content sample; Using an image generation model, based on the foreground feature, the global feature and the text feature, a predicted image is generated; Constructing a loss function based on the predicted image; Model parameters of the image generation model are adjusted based on the loss function.

9. The method according to claim 8, wherein: The image generation model includes: a self-attention network and a cross-attention network; The adopting the image generation model to process the foreground feature, the global feature and the text feature to obtain a predicted image includes: Acquire fusion features based on the foreground features; Using the self-attention network, performing self-attention processing on the fused features to obtain self-attention processed features; Using the cross attention network, cross attention processing is performed on the self-attention processed features, the global features and the text features to obtain target features; The target features are converted into the target image.

10. The method according to claim 9, wherein: The cross attention network includes: a first cross attention network and a second cross attention network; The cross attention network is used to perform cross attention processing on the self-attention processed features, the global features and the text features to obtain target features, including: Using the first cross attention network, performing a first cross attention process on the self-attention processed features and the text features to obtain first cross attention processed features; Using the second cross attention network, performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature; Based on the second cross-attention processed features, the target features are obtained.

11. An image generation device based on a large model, comprising: A first extraction module is used to extract features of a foreground image corresponding to the foreground content in the original image to obtain foreground features; The second extraction module is used to perform image and text processing on the reference image to obtain global features and text features; The reference image is obtained based on a target style of the background content to be generated; A generation module is used to use a large model to generate a target image including the foreground content and the background content based on the foreground feature, the global feature and the text feature.

12. The device according to claim 11, wherein The generating module is further used for: Acquire fusion features based on the foreground features; Performing self-attention processing on the fused features to obtain self-attention processed features; Performing cross attention processing on the self-attention processed features, the global features, and the text features to obtain target features; The target features are converted into the target image.

13. The device according to claim 12, wherein: The generating module is further used for: Performing a first cross attention process on the self-attention processed feature and the text feature to obtain a first cross attention processed feature; Performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature; Based on the second cross-attention processed features, the target features are obtained.

14. The device according to claim 13, wherein: The generating module is further used for: The first cross-attention processed feature and the second cross-attention processed feature are added to obtain the target feature.

15. The device according to claim 12, wherein: The generating module is further used for: The foreground feature and the pre-generated noise feature are added to obtain the fused feature.

16. The device according to claim 11, wherein The second extraction module is further used for: Performing feature extraction on the reference image to obtain the global feature; Performing text extraction on the reference image to obtain text corresponding to the reference image, and performing feature extraction on the text to obtain the text feature.

17. The device according to claim 11, wherein: The first extraction module is further used for: Performing foreground segmentation on the original image to obtain a foreground image corresponding to the foreground content; Feature extraction is performed on the foreground image to obtain the foreground feature.

18. An image generation model training device, comprising: A first extraction module is used to extract features of a foreground image corresponding to the foreground content in the original image sample to obtain a foreground feature; The second extraction module is used to perform image and text processing on the reference image sample to obtain global features and text features; The reference image sample is obtained based on the target style of the background content sample; A prediction module, configured to generate a predicted image based on the foreground feature, the global feature and the text feature by using an image generation model; A construction module, used for constructing a loss function based on the predicted image; An adjustment module is used to adjust the model parameters of the image generation model based on the loss function.

19. The device according to claim 18, wherein: The image generation model includes: a self-attention network and a cross-attention network; The prediction module is further used to: Acquire fusion features based on the foreground features; Using the self-attention network, performing self-attention processing on the fused features to obtain self-attention processed features; Using the cross attention network, cross attention processing is performed on the self-attention processed features, the global features and the text features to obtain target features; The target features are converted into the target image.

20. The device according to claim 18, wherein The cross attention network includes: a first cross attention network and a second cross attention network; The prediction module is further used to: Using the first cross attention network, performing a first cross attention process on the self-attention processed features and the text features to obtain first cross attention processed features; Using the second cross attention network, performing a second cross attention processing on the first cross attention processed feature and the global feature to obtain a second cross attention processed feature; Based on the second cross-attention processed features, the target features are obtained.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images