Training method, related device and medium of image generation model
By generating and utilizing denoising control information, background fine-tuning control information and object fine-tuning control information, the image generation model is trained, and the problem of incoordination between objects and background is solved, and the accuracy of image generation is improved.
Patent Information
- Application Number
- CN202411081525.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-08
AI Technical Summary
In business scenarios such as video production and virtual display, it is difficult to collect sufficient training data, resulting in inconsistent objects and backgrounds in the target image generated by the model, affecting accuracy.
By acquiring multiple graphic and text sample pairs, denoising control information, background fine-tuning control information and object fine-tuning control information are generated, and the sample reference image is denoised using the image generation model to denoising, and predicted image features are trained, and images are trained based on these features Generate models to improve object-background coordination.
The accuracy of image generation model training and replacement processing are improved, so that the objects in the generated target image are more coordinated and consistent with the background.
Smart Images

Figure CN118747726B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of big data technology, and in particular to a training method, related device and medium for an image generation model. Background Art
[0002] At present, in various business scenarios such as video production and virtual display, it is often necessary to replace background images and object images to create personalized images. For example, when creating a personalized image of person A, the relevant technology often uses a neural network model to cut out the character image of person A, embeds the cutout character image into a selected background image B in the form of a texture, and performs edge repair on the generated texture image so that person A can be displayed in the specified background image B.
[0003] However, in actual model training, it is often difficult to collect enough training data that meets expectations, which will affect the model training effect and cause inconsistencies between the object (foreground) and the background in the target image generated by the model (for example, it is impossible to make the object and the background in the target image consistent in terms of lighting, filters, etc.), resulting in low accuracy of the target image generated by the model. Summary of the invention
[0004] The embodiments of the present disclosure provide a training method, related device and medium for an image generation model, which can improve the coordination between an object and a background in image processing for object replacement in a background image, thereby improving the accuracy of the replacement process.
[0005] According to one aspect of the present disclosure, a training method for an image generation model is provided, the training method comprising:
[0006] Acquire a plurality of image-text sample pairs, wherein each of the image-text sample pairs comprises a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to an expected result of replacing a reference object in the sample background image with the sample object;
[0007] Generate denoising control information based on the first description information, the second description information, and the sample image description information;
[0008] generating background fine-tuning control information based on the sample background image, and generating object fine-tuning control information based on the sample object image;
[0009] Based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, the sample latent space features corresponding to the sample reference image are denoised by the image generation model to obtain predicted image features;
[0010] Based on the predicted image features of the multiple image-text sample pairs and the sample reference image, the image generation model is trained to obtain a trained image generation model, and the trained image generation model is used to generate an image based on a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information.
[0011] According to one aspect of the present disclosure, there is provided an image generation method, the method comprising:
[0012] Acquire a target object image of a target object, a target background image, object description information of the target object image, background description information of the target background image, and target image description information, wherein the target background image includes a reference object to be replaced by the target object in the target object image, and the target image description information is used to describe the replacement from the reference object to the target object;
[0013] Generate denoising prompt information based on the background description information, the object description information, and the target image description information;
[0014] generating background fine-tuning prompt information based on the target background image, and generating object fine-tuning prompt information based on the target object image;
[0015] Based on a preset noise image, the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information, image generation is performed through a trained image generation model to obtain a target image, wherein the trained image generation model is generated according to the above-mentioned image generation model training method; and the target image is used to indicate the result of replacing the reference object in the target background image with the target object of the target object image.
[0016] According to one aspect of the present disclosure, a training device for an image generation model is provided, the training device comprising:
[0017] A first acquisition unit is used to acquire a plurality of image-text sample pairs, wherein each of the image-text sample pairs includes a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to an expected result of replacing a reference object in the sample background image with the sample object;
[0018] A first generating unit, configured to generate denoising control information based on the first description information, the second description information, and the sample image description information;
[0019] a second generating unit, configured to generate background fine-tuning control information based on the sample background image, and generate object fine-tuning control information based on the sample object image;
[0020] a denoising unit, configured to perform denoising processing on the sample latent space features corresponding to the sample reference image through the image generation model based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, so as to obtain predicted image features;
[0021] A training unit is used to train the image generation model based on the predicted image features of the plurality of image-text sample pairs and the sample reference image to obtain a trained image generation model, wherein the trained image generation model is used to generate an image based on a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information.
[0022] Optionally, the image generation model includes a denoising network; the denoising network includes an upsampling attention network and a downsampling attention network; the background fine-tuning control information includes first background fine-tuning information and second background fine-tuning information; the object fine-tuning control information includes first object fine-tuning information and second object fine-tuning information;
[0023] The denoising unit comprises:
[0024] A downsampling subunit, configured to perform downsampling processing on the sample latent space features through the downsampling attention network based on the first object fine-tuning information, the first background fine-tuning information, and the denoising control information, to obtain a sample downsampling result;
[0025] An upsampling subunit is used to upsample the sample downsampling result through the upsampling attention network based on the second object fine-tuning information, the second background fine-tuning information, and the denoising control information to obtain the predicted image feature.
[0026] Optionally, the first background fine-tuning information includes a first background query feature, a first background key feature and a first background value feature; the first object fine-tuning information includes a first object query feature, a first object key feature and a first object value feature; the denoising control information includes a reference image description embedding feature, a sample image description embedding feature and a background image description embedding feature;
[0027] The down sampling subunit comprises:
[0028] A first attention module is used to perform a first attention calculation through the downsampling attention network based on the sample latent space feature, the first background query feature, the first background key feature, and the background image description embedding feature to obtain a background image attention result;
[0029] A second attention module is used to perform a second attention calculation through the downsampling attention network based on the sample latent space feature, the first object query feature, the first object key feature, and the sample image description embedding feature to obtain a sample object image attention result;
[0030] A replacement module, used for performing feature replacement on the background image attention result based on the sample object image attention result to obtain a replaced background image attention result;
[0031] A third attention module is used to perform a third attention calculation through the downsampling attention network based on the replaced background image attention result, the first background value feature, the first object value feature and the reference image description embedding feature to obtain the sample downsampling result.
[0032] Optionally, the first attention module is used to:
[0033] Performing weighted processing on the sample latent space feature and the first background query feature to obtain a first weighted image feature;
[0034] Performing weighted processing on the background image description embedding feature and the first background key feature to obtain a second weighted image feature;
[0035] Perform matrix multiplication on the first weighted image feature and the second weighted image feature to obtain the background image attention result.
[0036] Optionally, the replacement module is used to:
[0037] Determine a first word number of the reference object based on the first description information, and determine a second word number of the sample object based on the second description information;
[0038] determining a word weight between the reference object and the sample object based on the first word number and the second word number;
[0039] Based on the word weights, the reference object features in the background image attention result are replaced with the sample object features in the sample object image attention result to obtain the replaced background image attention result.
[0040] Optionally, the third attention module is used to:
[0041] Performing weighted processing on the first background value feature, the first object value feature and the reference image description embedding feature to obtain a sample weighted image feature;
[0042] The sample weighted image features and the replaced background image attention results are matrix multiplied to obtain the sample downsampling result.
[0043] Optionally, the first generating unit is used for:
[0044] Performing word embedding processing on the first description information to obtain background image description embedding features;
[0045] Performing word embedding processing on the second description information to obtain sample image description embedding features;
[0046] Performing word embedding processing on the sample image description information to obtain a reference image description embedding feature;
[0047] The reference image description embedding feature, the sample image description embedding feature and the background image description embedding feature are integrated into the denoising control information.
[0048] Optionally, the second generating unit is used for:
[0049] activating a preset fine-tuning network based on the first description information;
[0050] Performing feature extraction on the sample background image based on the activated fine-tuning network to obtain a first background query feature, a first background key feature, and a first background value feature;
[0051] The first background query feature, the first background key feature and the first background value feature are integrated into the background fine-tuning control information.
[0052] Optionally, the second generating unit is used for:
[0053] activating a preset fine-tuning network based on the second description information;
[0054] Performing feature extraction on the sample object image based on the activated fine-tuning network to obtain a first object query feature, a first object key feature, and a first object value feature;
[0055] The first object query feature, the first object key feature and the first object value feature are integrated into the object fine-tuning control information.
[0056] Optionally, the sample latent space feature corresponding to the sample reference image is generated in the following manner:
[0057] Performing encoding processing on the sample reference image to obtain a sample image encoding feature;
[0058] Based on a preset time step, the sample image encoding features are diffused through the diffusion network of the image generation model to obtain the sample latent space features.
[0059] Optionally, the sample reference image is generated by:
[0060] Acquire an original reference image, wherein the original reference image is used to indicate an expected result of replacing a reference object in the sample background image with a sample object;
[0061] Generate a random number that obeys a Gaussian distribution based on a predetermined random number generation model;
[0062] For each pixel point in the original reference image, the random number is added to the pixel value of the pixel point to obtain the sample reference image.
[0063] Optionally, the training unit comprises;
[0064] An acquisition module, used for acquiring reference noise in the sample reference image for each of the image-text sample pairs, and performing noise prediction based on the predicted image features to obtain predicted noise;
[0065] A calculation module, configured to calculate a sub-loss function based on a comparison between the reference noise and the predicted noise;
[0066] A determination module, configured to determine a total loss function based on the sub-loss function of each of the image-text sample pairs;
[0067] A training module is used to train the image generation model based on the total loss function.
[0068] Optionally, the prediction noise is obtained by prediction of multiple prediction time steps;
[0069] The calculation module is used for:
[0070] Performing difference calculation on the reference noise and the predicted noise to obtain a noise difference;
[0071] A regularization term is calculated on the noise difference to obtain the sub-loss function.
[0072] Optionally, the sample object image is generated by:
[0073] Acquire a sample image of the sample object;
[0074] Performing image segmentation on the sample image based on a preset object segmentation model to obtain a sample segmented image having the sample object;
[0075] Perform image enhancement on the sample segmented image to obtain the sample object image.
[0076] According to one aspect of the present disclosure, there is provided an image generating device, the image generating device comprising:
[0077] a second acquisition unit, configured to acquire a target object image of a target object, a target background image, object description information of the target object image, background description information of the target background image, and target image description information, wherein the target background image includes a reference object to be replaced by the target object in the target object image, and the target image description information is used to describe the replacement from the reference object to the target object;
[0078] A third generating unit, configured to generate denoising prompt information based on the background description information, the object description information, and the target image description information;
[0079] a fourth generating unit, configured to generate background fine-tuning prompt information based on the target background image, and generate object fine-tuning prompt information based on the target object image;
[0080] An image generation unit is used to generate an image based on a preset noise image, the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information through a trained image generation model to obtain a target image, wherein the trained image generation model is generated according to the above-mentioned image generation model training method; and the target image is used to indicate the result of replacing the reference object in the target background image with the target object of the target object image.
[0081] Optionally, the trained image generation model includes a diffusion network, a denoising network, and a decoding network;
[0082] The image generating unit is used for:
[0083] Based on the diffusion network, the encoded image features of the preset noise image are diffused to obtain target latent space features;
[0084] Based on the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information, denoising the target latent space features through the denoising network to obtain a target denoising result;
[0085] The target denoising result is feature decoded based on the decoding network to obtain the target image.
[0086] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned image generation model training method or image generation method when executing the computer program.
[0087] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the image generation model or the image generation method as described above is implemented.
[0088] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program, and the computer program is read and executed by a processor of an electronic device, so that the electronic device executes the training method of the image generation model or the image generation method as described above.
[0089] In the disclosed embodiment, denoising control information is generated based on the first description information of the sample background image, the second description information of the sample object image, and the sample image description information of the sample reference image. Since the first description information focuses on describing the background from the textual aspect, the second description information focuses on describing the replaced object from the textual aspect, and the sample image description information focuses on describing the overall image after object replacement from the textual aspect, the denoising control information comprehensively expresses various aspects of the object replacement process from the textual perspective. The background fine-tuning control information focuses on the background information obtained from the content reflected by the image itself, and the object fine-tuning control information focuses on the object information obtained from the content reflected by the image itself. Therefore, the background fine-tuning control information, the object fine-tuning control information, and the denoising control information are combined to perform denoising on the sample latent space features corresponding to the sample reference image. Such denoising takes into account a relatively comprehensive range of factors, fully considers the coordination between the object and the background in the image processing for object replacement, improves the accuracy of the image generation model training, and also improves the accuracy of the replacement process.
[0090] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0092] Figure 1 is a system architecture diagram of a training method for an image generation model and a system for applying the image generation method according to an embodiment of the present disclosure;
[0093] Figure 2A-2D A schematic diagram showing an application of the image generation method according to an embodiment of the present disclosure in an object display scenario;
[0094] Figure 3 is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure;
[0095] Figure 4 is a flowchart of determining a sample object image according to an embodiment of the present disclosure;
[0096] Figure 5 is a flow chart of determining a sample reference image according to an embodiment of the present disclosure;
[0097] Figure 6 is a schematic diagram of an implementation process of determining a sample reference image according to an embodiment of the present disclosure;
[0098] Figure 7 is a flow chart of generating denoising control information according to an embodiment of the present disclosure;
[0099] Figure 8 is a schematic diagram of an implementation process of generating denoising control information according to an embodiment of the present disclosure;
[0100] Fig. 9 is a flow chart of generating background fine-tuning control information according to an embodiment of the present disclosure;
[0101] Fig.10 is a flow chart of generating object fine-tuning control information according to an embodiment of the present disclosure;
[0102] Fig.11 is a flow chart of generating sample latent space features according to an embodiment of the present disclosure;
[0103] Fig.12is a flow chart of generating predicted image features according to one embodiment of the present disclosure;
[0104] Figure 13A-13B is a schematic diagram of an overall implementation process of generating predicted image features through a denoising network according to an embodiment of the present disclosure;
[0105] Fig.14 is a flow chart of generating sample downsampling results according to one embodiment of the present disclosure;
[0106] Fig.15 is a flowchart of generating a replaced background image attention result according to an embodiment of the present disclosure;
[0107] Fig.16 is a schematic diagram of an implementation process of generating a sample downsampling result according to an embodiment of the present disclosure;
[0108] Fig.17 is a flowchart of a training image generation model according to one embodiment of the present disclosure;
[0109] Fig.18 is a flowchart of determining a sub-loss function according to an embodiment of the present disclosure;
[0110] Fig.19 is a schematic diagram of implementation details of a training image generation model according to an embodiment of the present disclosure;
[0111] Fig. 20 is a flowchart of an image generating method according to an embodiment of the present disclosure;
[0112] Fig.21 is a flowchart of generating a target image according to one embodiment of the present disclosure;
[0113] Fig. 22 is a simplified schematic diagram of an implementation of an image generation method according to an embodiment of the present disclosure;
[0114] Fig.23 is a schematic diagram of implementation details of an image generation method according to an embodiment of the present disclosure;
[0115] Fig.24 is a module diagram of a training device for an image generation model according to an embodiment of the present disclosure;
[0116] Fig.25 is a module diagram of an image generating device according to an embodiment of the present disclosure;
[0117] Fig.26 is a terminal structure diagram of a method for training an image generation model according to an embodiment of the present disclosure;
[0118] Fig. 27It is a server structure diagram of a method for training an image generation model according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0119] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.
[0120] The following describes the system architecture and scenarios for application of the embodiments of the present disclosure.
[0121] Figure 1 The system architecture diagram of the training method of the image generation model and the image generation method according to the embodiment of the present disclosure includes an object terminal 140, the Internet 130, a gateway 120, an image processing server 110, and an image database 150.
[0122] The object terminal 140 includes various forms such as desktop computers, laptop computers, PDAs (personal digital assistants), tablet computers, mobile phones, vehicle-mounted terminals, home theater terminals, smart TVs, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. The object terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data. The object terminal 140 includes an image processing system, which is used to receive the background image, object image, description information of the background image, and description information of the object image selected by the object, and submit the background image, object image, description information of the background image, and description information of the object image to the image processing server, so that the image processing server generates a target image with the target object in the object image as the foreground and the background image as the background.
[0123] The image processing server 110 refers to a computer system that can provide certain services to the object terminal 140. Compared with the ordinary object terminal 140, the image processing server 110 has higher requirements in terms of stability, security, performance, etc. The image processing server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), or a cloud server, etc. The image processing server 110 includes various types of services, wherein the implementation of each service of the image processing server 110 is often associated with some intermediate databases or storage media, etc. The image processing server 110 is used to generate a target image with a target object in the object image as the foreground and a background image as the background according to the background image, object image, description information of the background image and description information of the object image submitted by the object using a trained image generation model, and the image database 150 is used to store various images such as the target image, object image, and background image, wherein the image database 150 can be set separately or integrated in the image processing server 110 or other electronic devices.
[0124] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even have completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the object terminal 140 to the image processing server 110 must be sent to the corresponding server through the gateway 120. The message sent by the image processing server 110 to the object terminal 140 must also be sent to the corresponding object terminal 140 through the gateway 120.
[0125] The embodiments of the present disclosure can be applied in various scenarios, such as Figure 2A-2D The item display scene shown, etc.
[0126] like Figure 2AAs shown, when the subject wants to use the same background to display multiple objects, the subject will log in to the image processing system on the subject terminal and enter the image synthesis process. At this time, a prompt field "Please select the background image, upload the target object image, and enter the relevant image description" will be displayed on the page, and an editing area for selecting the background image, an editing area for selecting the target object image, an editing area for entering the image description, an editing area for entering the object image description, and an editing area for entering the background image description will be provided. Based on this, the subject selects "background image 5" in the editing area for selecting the background image; the subject selects "D:\object image of object A\image 1" in the editing area for selecting the target object image; the subject enters "replace the plastic bottle in background image 5 with a glass bottle" in the editing area for entering the object image description, enters "a glass bottle" in the editing area for entering the object image description, enters "there is a plastic bottle on the table in front of the window sill" in the editing area for entering the background image description, and clicks the "OK" button.
[0127] like Figure 2B As shown, after the object clicks the "OK" button, a prompt window will be displayed on the page, wherein the prompt window has a prompt field "Background control information, object control information, denoising control information, and random noise map for image synthesis are being generated, please wait patiently..." to indicate to the object the generation process of various control information before image synthesis.
[0128] like Figure 2C As shown, it is a schematic diagram of the image processing system performing image synthesis. In the page, a prompt field "The image generation model is being called, and image synthesis is being performed based on background control information, object control information, denoising control information, and random noise map. Please wait patiently..." is displayed in the form of a prompt window to show the image synthesis process of the image generation model to the object.
[0129] like Figure 2D As shown, when the image generation model is executed, the page will display a prompt field "The plastic bottle in background image 5 has been replaced with a glass bottle. In addition, the generated image has been saved to the path: M:\synthetic image folder\object display", and a comparison diagram of background image 5 and the generated target image will be displayed. Among them, background image 5 shows a plastic bottle placed on a table, and the target image shows a glass bottle placed on a table. Through the image generation model in the image processing system, the plastic bottle in background image 5 is replaced with a glass bottle while keeping the background unchanged.
[0130] It should be noted that, in addition to being applicable to the above-mentioned object display scenarios, the image generation method of the disclosed embodiment can also be applied to various application scenarios such as video editing scenarios in self-media and personalized image production scenarios in social media.
[0131] The following is a general description of the training method of the image generation model of the embodiment of the present disclosure.
[0132] According to one embodiment of the present disclosure, a method for training an image generation model is provided.
[0133] The training method of this image generation model is generally used in business scenarios where a target object (person, animal, object, etc.) needs to be embedded into a background image and the coordination between the embedded target object and the background is required to be high, such as Figure 2A-2D The object display scene shown, etc. The present disclosure provides a solution for jointly training a model based on image description, background image and background description, object image and object description, which can improve the coordination between the object and the background in the image processing of replacing the object in the background image, thereby improving the accuracy of the replacement process.
[0134] like Figure 3 As shown, the training method of the image generation model according to an embodiment of the present disclosure can be performed by an electronic device, which can be Figure 1 The image processing server or object terminal shown. The training method of the image generation model according to one embodiment of the present disclosure may include:
[0135] Step 310: obtaining a plurality of image-text sample pairs;
[0136] Step 320: Generate denoising control information based on the first description information, the second description information, and the sample image description information;
[0137] Step 330: Generate background fine-tuning control information based on the sample background image, and generate object fine-tuning control information based on the sample object image;
[0138] Step 340: Based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, the sample latent space features corresponding to the sample reference image are denoised by the image generation model to obtain predicted image features;
[0139] Step 350: Based on the predicted image features of the plurality of image-text sample pairs and the sample reference images, an image generation model is trained to obtain a trained image generation model.
[0140] Steps 310 - 350 are described in detail below.
[0141] In step 310, a plurality of image-text sample pairs are obtained.
[0142] In the disclosed embodiment, a text-image sample pair is used as training data, wherein each text-image sample pair includes a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image.
[0143] The sample background image refers to an image to which the sample object is to be added. The sample background image is often a landscape image, a scene image, etc. containing a series of things.
[0144] The first description information of the sample background image is used to describe the image scene, image content, etc. of the sample background image. For example, for a sample background image representing a park, the first description information is "the scene of the sample background image is a park".
[0145] The sample object image of the sample object refers to image data containing the sample object. The sample object image can often reflect object characteristics such as object posture, object movement, and object appearance of the sample object.
[0146] The second description information of the sample object image is used to describe the object category, object characteristics, etc. of the sample object in the sample object image. For example, for a sample object image representing a white puppy, the second description information is "the object category of the sample object is a dog".
[0147] The sample reference image is obtained by adding noise to the expected result of replacing the reference object in the sample background image with the sample object.
[0148] The sample image description information of the sample reference image is used to indicate the conditional constraints on the denoising process.
[0149] For example, the sample image description information may be "replace the kitten in the park with a puppy". In this case, the sample background image is an image with a park as the scene and a kitten in it. The sample object image is an image with a puppy in it.
[0150] In step 320, denoising control information is generated based on the first description information, the second description information, and the sample image description information.
[0151] The denoising control information is used as a text condition constraint to assist the image generation model in performing image denoising during image generation, so that the image denoising effect meets the actual requirements.
[0152] To save space, the specific process of generating denoising control information based on the first description information, the second description information, and the sample image description information in the embodiment of the present disclosure will be described in detail below and will not be repeated here.
[0153] In step 330, background fine-tuning control information is generated based on the sample background image, and object fine-tuning control information is generated based on the sample object image.
[0154] The background fine-tuning control information is used to guide and control the image generation model's attention to and learning of background feature information during the denoising process, and is used to indicate the degree of fine-tuning of background features during the denoising process.
[0155] The object fine-tuning control information is used to guide and control the image generation model's attention to and learning of object feature information during the denoising process, and is used to indicate the degree of fine-tuning of object features during the denoising process.
[0156] To save space, the specific process of generating background fine-tuning control information based on the sample background image and the specific process of generating object fine-tuning control information based on the sample object image in the embodiment of the present disclosure will be described in detail below and will not be repeated here.
[0157] In step 340, based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, the sample latent space features corresponding to the sample reference image are denoised by the image generation model to obtain predicted image features.
[0158] The image generation model of the embodiment of the present disclosure is a neural network model constructed based on a stable diffusion (SD) model. The image generation model of the embodiment of the present disclosure often takes a background image, an object image, a description of a process of replacing an object in the background image with an object in the object image, a text description of the background image, and a text description of the object image as input, and outputs a composite image with the scene of the background image as the background and the object in the object image as the foreground. The image generation model can embed an object into a background image and can meet the image generation requirements in a variety of scenarios.
[0159] The sample latent space feature corresponding to the sample reference image refers to the representation of the pure noise image without image features corresponding to the sample reference image. The sample latent space feature can be in vector form or matrix form without limitation.
[0160] The predicted image features refer to the features predicted by the image generation model under the constraints of background fine-tuning control information, object fine-tuning control information, and denoising control information, which can represent the image content of the sample reference image in the latent vector space.
[0161] To save space, the specific process of denoising the sample latent space features corresponding to the sample reference image through the image generation model in the embodiment of the present disclosure will be described in detail below and will not be repeated here.
[0162] In step 350, an image generation model is trained based on the predicted image features of a plurality of image-text sample pairs and sample reference images to obtain a trained image generation model.
[0163] The trained image generation model is used to generate an image based on the target object image, the target background image, the object description information of the target object image, the background description information of the target background image, and the target image description information.
[0164] To save space, the specific process of predicting image features based on multiple image-text sample pairs and training the image generation model of sample reference images in the embodiment of the present disclosure will be described in detail below and will not be repeated here.
[0165] Through the above steps 310-350, in the embodiment of the present disclosure, based on the first description information of the sample background image, the second description information of the sample object image, and the sample image description information of the sample reference image, the denoising control information is generated. Since the first description information focuses on describing the background from the text aspect, the second description information focuses on describing the replaced object from the text aspect, and the sample image description information focuses on describing the overall image after the object replacement from the text aspect, the denoising control information comprehensively expresses various aspects of the object replacement process from the text perspective. The background fine-tuning control information focuses on the background information obtained from the content reflected by the image itself, and the object fine-tuning control information focuses on the object information obtained from the content reflected by the image itself. Therefore, the background fine-tuning control information, the object fine-tuning control information, and the denoising control information are combined to perform denoising on the sample latent space features corresponding to the sample reference image. Such denoising processing considers relatively comprehensive factors. When training the model, it fully considers that the model focuses on learning the coordination between the object and the background in the image processing for object replacement, thereby improving the accuracy of the image generation model training and the accuracy of the replacement processing.
[0166] The above is a general description of steps 310 to 350. The specific implementation of steps 310, 320, 330, 340 and 350 will be described in detail below.
[0167] Step 310 is described in detail below.
[0168] In step 310, a plurality of image-text sample pairs are obtained, wherein each image-text sample pair includes a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to the expected result of replacing a reference object in the sample background image with the sample object.
[0169] Please refer to Figure 4 In one embodiment, the sample object image of the sample object is determined by:
[0170] Step 410: Obtain a sample image of a sample object;
[0171] Step 420: performing image segmentation on the sample image based on a preset object segmentation model to obtain a sample segmented image having the sample object;
[0172] Step 430: Perform image enhancement on the sample segmentation image to obtain a sample object image.
[0173] Steps 410 - 430 are described in detail below.
[0174] In step 410, a sample image is an image containing a sample object.
[0175] In the specific implementation of this embodiment, with authorization, sample images of multiple sample images can be extracted from an existing image database, and video data with sample objects can be extracted from an existing video database, and the video data can be segmented into video frames, and each video frame with sample objects can be used as a sample image.
[0176] In step 420, the sample segmentation image refers to a local image having a sample object segmented from the sample image, and the sample segmentation image is a part of the sample image.
[0177] The object segmentation model can be a lightweight semantic segmentation model BiSeNet-v2 or PP_LiteSeg.
[0178] Taking the semantic segmentation model PP_LiteSeg as an example, the object segmentation model includes an encoding module, a pyramid pooling module, a decoding module and an attention fusion module. Specifically, first, the sample image is input into the encoding module of the object segmentation model, and the sample image is multi-scale encoded by the encoding module to obtain the first sample image feature whose image size is one-fourth of the sample image, the second sample image feature whose image size is one-eighth of the sample image, the third sample image feature whose image size is one-sixteenth of the sample image, and the fourth sample image feature whose image size is one-thirty-second of the sample image. Then, the fourth sample image feature is feature pooled by the pyramid pooling module to obtain the sample image pooling feature. Further, the sample image pooling feature and the third sample image feature are feature fused by the attention fusion module to obtain the first image fusion feature. Then, the first image fusion feature and the second sample image feature are feature fused by the attention fusion module to obtain the second image fusion feature. Further, the second image fusion feature is decoded by the decoding module to obtain a sample segmented image with a sample object.
[0179] In step 430, the image enhancement of the sample segmented image includes but is not limited to contrast enhancement, brightness enhancement, sharpening, and noise reduction of the sample segmented image. Specifically, first, when the sample segmented image is enhanced, a linear stretching or logarithmic transformation method is used to expand the pixel value distribution of the sample segmented image so that the light and dark areas of the sample image segmented image are more distinct, so as to achieve contrast enhancement of the sample segmented image. Then, after contrast enhancement, the brightness values of all pixels of the sample segmented image are adjusted to improve the overall brightness of the sample segmented image. Further, after brightness adjustment, a Gaussian filter or a median filter is used to smooth the sample segmented image and reduce noise in the sample segmented image. Finally, after noise reduction, the sample segmented image is edge enhanced by a Laplace operator to achieve sharpening of the sample segmented image and obtain a sample object image.
[0180] The advantage of this embodiment is that the sample image of the sample object is segmented, and a local image with the sample object is segmented from the sample image, which can effectively eliminate the interference of irrelevant image information and improve the image quality of the sample segmented image used for training. Furthermore, multiple image enhancement processes such as contrast enhancement, brightness enhancement, sharpening, and noise reduction are adopted for the sample segmented image, which can further improve the image quality of the sample object image.
[0181] Please refer to Figure 5 In one embodiment, the sample reference image is determined by:
[0182] Step 510: Acquire an original reference image;
[0183] Step 520: Generate a random number that obeys a Gaussian distribution based on a predetermined random number generation model;
[0184] Step 530: For each pixel in the original reference image, add a random number to the pixel value of the pixel to obtain a sample reference image.
[0185] Steps 510 - 530 are described in detail below.
[0186] In step 510, the original reference image is used to indicate the expected result of replacing the reference object in the sample background image with the sample object.
[0187] In the specific implementation of this embodiment, the reference object of the sample background image can be replaced with the sample object using image editing software, so that the obtained replaced image is regarded as the expected result of replacing the reference object in the sample background image with the sample object. The image editing software can be Adobe Photoshop or other software.
[0188] In step 520, the predetermined random number generation model refers to a random number generator, and a random number is a randomly generated number, wherein the random number generated by the embodiment of the present disclosure and obeying the Gaussian distribution may be repeated.
[0189] When this embodiment is implemented, the predetermined random number generation model has a library function (numpy.random.normal() function). Specifically, a random number is generated by the library function of the predetermined random number generation model according to the mean value of 0 and the standard deviation of 1 that the Gaussian distribution is to follow. The random number obeys the Gaussian distribution. The random number obeying the Gaussian distribution can be expressed in the form of a random noise image, and the image size of the random noise image is the same as the image size of the original reference image.
[0190] In step 530, for each pixel in the original reference image, the pixel value of the pixel in the original reference image is first determined, and the noise value of the pixel in the random noise image corresponding to the random number is determined. Then, the pixel value and the noise value of the pixel are added to obtain the noisy pixel value of the pixel. Finally, the sample reference image is obtained according to the noisy pixel value of each pixel.
[0191] like Figure 6As shown, it is a schematic diagram of the specific implementation process of superimposing random noise on the original reference image. The original reference image is an 8×8 image with 64 pixels. The random noise map corresponding to the random number obeying the Gaussian distribution is also an 8×8 image with 64 pixels. Specifically, for each pixel of the original reference image, the pixel value of each pixel is first determined, wherein the pixel value of each pixel includes 1, 2, 3, 4, 5 or 6. Then, for each pixel, the noise value of each pixel in the random noise map is determined, wherein the noise value includes 0, 1, 2, 3, 4, 5 or 6. Further, for each pixel, the pixel value and the noise value are added to obtain the noisy pixel value of each pixel. For example, for each pixel in the first row of the original reference image, the pixel value of the first pixel is 1, the noise value is 5, and the noisy pixel value is 1+5=6. The pixel value of the second pixel is 1, the noise value is 1, and the noisy pixel value is 1+1=2. The pixel value of the third pixel is 1, and the noise value is 4, so the noisy pixel value is 1+4=5; and so on, until the noisy pixel value of the last pixel in the last row is determined, and a sample reference image is generated.
[0192] The benefit of this embodiment is that random noise conforming to the Gaussian distribution is added to the expected result of replacing the reference object in the sample background image with the sample object, and environmental noise is introduced into the idealized reference image, so that the final expected result is more in line with the actual situation, thereby improving the authenticity and accuracy of the sample reference image and making the sample reference image more referenceable.
[0193] Step 320 is described in detail below.
[0194] In step 320, denoising control information is generated based on the first description information, the second description information, and the sample image description information.
[0195] Please refer to Figure 7 In one embodiment, step 320 specifically includes but is not limited to the following steps 710-740:
[0196] Step 710: Perform word embedding processing on the first description information to obtain background image description embedding features;
[0197] Step 720: Perform word embedding processing on the second description information to obtain sample image description embedding features;
[0198] Step 730: Perform word embedding processing on the sample image description information to obtain a reference image description embedding feature;
[0199] Step 740: Integrate the reference image description embedding features, the sample image description embedding features and the background image description embedding features into denoising control information.
[0200] Steps 710-740 are described in detail below.
[0201] In step 710, the background image description embedding feature is used to perform image scene description on the sample background image in the form of vector features.
[0202] When this embodiment is implemented specifically, word embedding processing may be performed on the first description information based on a preset text encoder to obtain background image description embedding features.
[0203] In the embodiment of the present disclosure, the text encoder includes a tokenizer, an embedding layer, and a text attention calculation module (text transformer).
[0204] Specifically, first, the first description information is input into a word segmenter, and the first description information is segmented based on spaces, punctuation marks or separators based on the word segmenter to obtain multiple words. Furthermore, the stop words in the multiple words are removed to obtain multiple description words, and the index corresponding to each description word is found in a preset index table, and the description word is represented by the index. Among them, the preset index table is used to indicate the correspondence between words and indexes. Then, the index of each description word is input into the embedding layer for word embedding processing to obtain the description word embedding features corresponding to each description word. Finally, the description word embedding features of each description word are input into the text attention calculation module for attention calculation, and the text attention calculation module outputs the background image description embedding features corresponding to the first description information.
[0205] In step 720, the sample image description embedded features are used to describe the object category of the sample object image in the form of vector features.
[0206] When this embodiment is implemented, the specific process of step 720 is similar to the specific process of the above step 710. To save space, it will not be repeated.
[0207] In step 730, the reference image description embedded features are used to describe the foreground and background of the sample reference image in the form of vector features.
[0208] When this embodiment is implemented, the specific process of step 730 is similar to the specific process of the above step 710. To save space, it will not be repeated.
[0209] In step 740, the reference image description embedding features, the sample image description embedding features and the background image description embedding features are included in the same set, and all feature information in the set is determined as denoising control information.
[0210] like Figure 8As shown in FIG. 1 , it is a specific example of generating a benchmark image description embedding feature based on sample image description information. Specifically, the input image instance is an alarm clock image. In order to improve the learning ability of the model, the sample image description information of the input image instance is set to “A Photo of S * ”, and use the specific symbol “S * " is used to represent it. Based on this, the sample image description information is first input into the word segmenter of the text encoder, and the sample image description information is segmented by the word segmenter to obtain multiple words, and the index corresponding to each word is found in the preset index table. Among them, the index corresponding to the word "A" is 508, the index corresponding to the word "Photo" is 701, the index corresponding to the word "of" is 73, and the index corresponding to the word "S * "The corresponding index is *. Further, the embedding layer performs word embedding processing on the index of each word, and maps the index corresponding to each word from the numerical space to the vector space according to the preset feature word table (the feature word table is used to indicate the vector features corresponding to each index), and obtains the description word embedding feature corresponding to each word, among which the description word embedding feature corresponding to the word "A" is " "The word "Photo" corresponding to the description word embedding feature is" "The word "of" corresponds to the descriptive word embedding feature" "; the word "S * "The corresponding description word embedding feature is" Furthermore, the description word embedding features of each word are input into the text attention calculation module for attention calculation, and the reference image description embedding features are output through the text attention calculation module. . In addition, in the disclosed embodiment, the sample image description embedding features and the background image description embedding features are obtained in this way, and the reference image description embedding features, the sample image description embedding features and the background image description embedding features are used together as conditional constraints for the denoising process of the image generator, so that the predicted image instance output by the image generator fits the content represented by the sample image description information. In addition, the predicted image instance is obtained by performing three diffusion processes (three noise addition operations) on the input image instance to obtain a noisy image instance, and the image generator performs three denoising processes on the noisy image instance based on the reference image description embedding features, the sample image description embedding features and the background image description embedding features.
[0211] The advantage of this embodiment is that a preset text encoder is used to perform word embedding processing on the first description information, the second description information, and the sample image description information respectively, to obtain the background image description embedding features corresponding to the first description information, the sample image description embedding features corresponding to the second description information, and the reference image description embedding features corresponding to the sample image description information. This realizes the vectorization of the description information that describes the background, the target object, and the overall image after object replacement from a textual perspective, and aggregates the three types of feature information that describe the background, the target object, and the overall image after object replacement into a whole, which together serve as conditional constraints for denoising processing, thereby improving the comprehensiveness and content effectiveness of the denoising control information.
[0212] Step 330 is described in detail below.
[0213] In step 330, background fine-tuning control information is generated based on the sample background image, and object fine-tuning control information is generated based on the sample object image.
[0214] Please refer to Fig. 9 In one embodiment, the specific process of generating background fine-tuning control information includes but is not limited to the following steps 910-930:
[0215] Step 910: activating a preset fine-tuning network based on the first description information;
[0216] Step 920: extract features of the sample background image based on the activated fine-tuning network to obtain a first background query feature, a first background key feature, and a first background value feature;
[0217] Step 930: Integrate the first background query feature, the first background key feature, and the first background value feature into background fine-tuning control information.
[0218] Steps 910 - 930 are described in detail below.
[0219] In step 910, the preset fine-tuning network is a neural network generated based on the fine-tuning technology (LoRA technology) of the deep learning model.
[0220] When this embodiment is implemented, when it involves feature extraction of multiple different description information, different description information is often applicable to different fine-tuning networks. Based on this, first, based on the first description information, a preset candidate fine-tuning network for feature extraction of the first description information is screened out from multiple preset candidate fine-tuning networks. Then, the screened preset candidate fine-tuning network is activated as the selected fine-tuning network, so as to realize feature extraction of the first description information using the activated fine-tuning network.
[0221] In step 920, the first background query feature is used to indicate image feature information of the sample background image on the query channel.
[0222] The first background key feature is used to indicate image feature information of the sample background image on the key channel.
[0223] The first background value feature is used to indicate image feature information of the sample background image on the value channel.
[0224] In the specific implementation of this embodiment, first, the sample background image is input into the activated fine-tuning network. Then, the activated fine-tuning network linearly projects the sample background image on different channels to obtain the key vector of the sample background image on the key channel, the value vector of the sample background image on the value channel, and the query vector of the sample background image on the query channel. Finally, the key vector on the key channel is determined as the first background key feature (LoRA K1), the value vector on the value channel is determined as the first background value feature (LoRA V1), and the query vector on the query channel is determined as the first background query feature (LoRA Q1).
[0225] In step 930 , the first background value feature (LoRA V1 ), the first background key feature (LoRA K1 ) and the first background query feature (LoRA Q1 ) are integrated into a whole to obtain background fine-tuning control information (LoRA Q1K1V1 ).
[0226] The benefit of this embodiment is that the fine-tuning network is used to extract features of the sample background image in multiple channel dimensions to obtain the first background query feature, the first background key feature and the first background value feature. Therefore, based on this series of background features, the background information can be obtained from the content reflected by the image itself, and multi-dimensional background fine-tuning can be achieved in the denoising process, thereby improving the comprehensiveness of the background fine-tuning control information and thus improving the accuracy of the background fine-tuning in the denoising process.
[0227] Please refer to Fig.10 In one embodiment, the specific process of generating object fine-tuning control information includes but is not limited to the following steps 1010-1030:
[0228] Step 1010: activating a preset fine-tuning network based on the first description information;
[0229] Step 1020: extract features of the sample object image based on the activated fine-tuning network to obtain a first object query feature, a first object key feature, and a first object value feature;
[0230] Step 1030: Integrate the first object query feature, the first object key feature, and the first object value feature into object fine-tuning control information.
[0231] Steps 1010-1030 are described in detail below.
[0232] The first object query feature is used to indicate image feature information of the sample object image on the query channel.
[0233] The first object key feature is used to indicate image feature information of the sample object image on the key channel.
[0234] The first object value feature is used to indicate image feature information of the sample object image on the value channel.
[0235] When this embodiment is implemented, the specific process of steps 1010-1030 is similar to the specific process of steps 910-930, and will not be described in detail to save space.
[0236] The benefit of this embodiment is that the fine-tuning network is used to extract features of the sample object image in multiple channel dimensions to obtain the first object query feature, the first object key feature and the first object value feature. Therefore, based on this series of object features, the object information can be obtained from the content reflected by the image itself, and multi-dimensional object fine-tuning can be achieved in the denoising process, thereby improving the comprehensiveness of the object fine-tuning control information and thus improving the accuracy of object fine-tuning in the denoising process.
[0237] It should be noted that, since the denoising network of the disclosed embodiment often contains an upsampling attention network and a downsampling attention network, in order to achieve background fine-tuning and object fine-tuning of the entire denoising process, the background fine-tuning control information often has multiple sets of background LoRA parameters, each set of background LoRA parameters contains a background value feature, a background key feature, and a background query feature. Similarly, the object fine-tuning control information often has multiple sets of object LoRA parameters, each set of object LoRA parameters contains an object value feature, an object key feature, and an object query feature. Among them, the network parameters of the fine-tuning network used to generate different groups of background LoRA parameters (or object LoRA parameters) are different.
[0238] For example, when both the upsampling attention network and the downsampling attention network are composed of one attention module, the background fine-tuning control information has two sets of background LoRA parameters, which are respectively the object LoRA parameters for the downsampling attention network (also referred to as the first background fine-tuning information) composed of the first background value feature, the first background key feature and a first background query feature; the object LoRA parameters for the upsampling attention network (also referred to as the second background fine-tuning information) are composed of the second background value feature, the second background key feature and a second background query feature. Similarly, the object fine-tuning control information has two sets of object LoRA parameters, which are respectively the object LoRA parameters for the downsampling attention network (also referred to as the first object fine-tuning information) composed of the first object value feature, the first object key feature and a first object query feature; the object LoRA parameters for the upsampling attention network (also referred to as the second object fine-tuning information) are composed of the second object value feature, the second object key feature and a second object query feature.
[0239] Furthermore, when both the upsampling attention network and the downsampling attention network are composed of multiple attention modules, the number of background LoRA parameters in the background fine-tuning control information and the number of object LoRA parameters in the object fine-tuning control information are the same as the number of modules of the attention modules, so that each attention module is fine-tuned using a set of background LoRA parameters and a set of object LoRA parameters, thereby improving the fine-tuning accuracy and comprehensiveness of fine-tuning in the denoising process.
[0240] The sample latent space features for generating sample reference images are described in detail below.
[0241] Please refer to Fig.11 In one embodiment, the specific process of generating the sample latent space features of the sample reference image includes but is not limited to the following steps 1110-1120:
[0242] Step 1110, encoding the sample reference image to obtain a sample image encoding feature;
[0243] Step 1120: Based on a preset time step, diffusion processing is performed on the sample image encoding features through the diffusion network of the image generation model to obtain the sample latent space features.
[0244] Steps 1110 - 1120 are described in detail below.
[0245] In step 1110 , the sample image encoding feature is used to indicate the image feature of the sample reference image in the vector space (latent space).
[0246] When this embodiment is implemented, first, the preset image encoder is called with authorization. Then, the sample reference image is input into the image encoder, and the sample reference image is encoded based on the image encoder to achieve the mapping of the sample reference image from the pixel space to the vector space (latent space), thereby obtaining the sample image encoding feature.
[0247] In step 1120, the preset time step is used to indicate the number of times the sample image coding feature of the sample reference image is denoised.
[0248] In an embodiment of the present disclosure, the image generation model includes a diffusion network.
[0249] The diffusion network is used to implement the gradual noise addition processing of the sample image encoding features until the input sample reference image features approach pure noise.
[0250] In the specific implementation of this embodiment, according to the preset time step, noise is added to the sample image coding feature through the diffusion network one by one, so that the sample image coding feature gradually loses its original features. When the number of noise additions reaches the preset time step, the sample image coding feature will become a latent space representation without any features. At this time, the obtained latent space representation is determined as the sample latent space feature.
[0251] For example, the preset time step is set to T, where T is a positive integer, and the sample image encoding features are denoised T times using a diffusion network, and the result after T times of denoising is used as the sample latent space feature.
[0252] The benefit of this embodiment is that the sample reference image is mapped from the pixel space to the latent space, so that the sample reference image becomes a sample image coding feature that meets the input requirements of the image generation model, which can improve the usability of the sample reference image. Furthermore, the sample image coding feature is denoised multiple times using the diffusion network of the image generation model, which realizes the pure noise processing of the sample image coding feature, and can effectively strip the effective image features in the sample image coding feature, so that the model can restore the image features according to various control information in the subsequent denoising process, which is conducive to improving the model's processing accuracy for image processing tasks to be replaced with objects.
[0253] Step 340 is described in detail below.
[0254] In step 340, based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, the sample latent space features corresponding to the sample reference image are denoised by the image generation model to obtain predicted image features.
[0255] In the disclosed embodiment, the image generation model includes not only the above-mentioned diffusion network, but also a denoising network, which is used to gradually denoise the sample latent space features output by the diffusion network until the predicted image features corresponding to the sample reference image are generated.
[0256] In the disclosed embodiment, the denoising network may be a U-type network structure (U-net network), and the denoising network includes an upsampling attention network and a downsampling attention network.
[0257] The upsampling attention network is used to downsample the sample latent space feature vector to obtain more low-dimensional features.
[0258] The downsampling attention network is used to upsample the output results of the upsampling attention network and restore the output results to denoised image features with the same feature dimension as the sample latent space feature vector.
[0259] In the embodiment of the present disclosure, the background fine-tuning control information includes first background fine-tuning information and second background fine-tuning information. The object fine-tuning control information includes first object fine-tuning information and second object fine-tuning information.
[0260] The first background fine-tuning information is used to guide the background fine-tuning in the downsampling attention process of the denoising network.
[0261] The second background fine-tuning information is used to guide the background fine-tuning in the upsampling attention process of the denoising network.
[0262] The first object fine-tuning information is used to guide the object fine-tuning in the downsampling attention process of the denoising network.
[0263] The second object fine-tuning information is used to guide the object fine-tuning in the upsampling attention process of the denoising network.
[0264] Please refer to Fig.12 In one embodiment, step 340 specifically includes but is not limited to the following steps 1210-1220:
[0265] Step 1210: based on the first object fine-tuning information, the first background fine-tuning information, and the denoising control information, down-sample the sample latent space features through a down-sampling attention network to obtain a sample down-sampling result;
[0266] Step 1220: Based on the second object fine-tuning information, the second background fine-tuning information, and the denoising control information, upsample the sample downsampling result through the upsampling attention network to obtain predicted image features.
[0267] Steps 1210 - 1220 are described in detail below.
[0268] In step 1210, the sample downsampling result is used to indicate the image features generated by denoising the sample latent space features by downsampling.
[0269] When this embodiment is implemented, in order to improve the learning effect of the model on the relationship between background features and object features during the downsampling process, a cross-attention mechanism can be introduced in the downsampling process. Specifically, first, based on the background image description embedded features in the denoising control information and the first background fine-tuning information, a cross-attention calculation is performed on the sample latent space features to achieve background fine-tuning of the sample latent space features. At the same time, based on the sample image description embedded features in the denoising control information and the first object fine-tuning information, a cross-attention calculation is performed on the sample latent space features to achieve object fine-tuning of the sample latent space features. Then, based on the reference image description embedded features in the denoising control information, the sample latent space features that have been fine-tuned for the object and the sample latent space features that have been fine-tuned for the background are fused with attention to obtain the sample downsampling result.
[0270] In order to save space, the specific process of downsampling the sample latent space features through the downsampling attention network in the embodiment of the present disclosure to obtain the sample downsampling result will be described in detail below. No further description will be given here.
[0271] In step 1220, the specific process of step 1220 is similar to the specific process of step 1210. The difference is that step 1210 is for downsampling of sample latent space features; while step 1220 is for upsampling of sample downsampling results, and the specific processes of the two are symmetrical. In addition, the feature dimension of the predicted image feature obtained in step 1220 is the same as the feature dimension of the sample latent space feature input in step 1220.
[0272] like Fig.13AAs shown, it is a specific schematic diagram of the denoising network performing a denoising operation (when the predetermined time step is T=1) according to the object fine-tuning control information, the background fine-tuning control information, and the denoising control information. Specifically, the denoising network includes a downsampling attention network and an upsampling attention network, wherein the downsampling attention network includes an attention downsampling module 1 having an attention downsampling layer and a residual block structure, and an attention downsampling module 2 having an attention downsampling layer and a residual block structure. The upsampling attention network includes an attention downsampling module 1 having an attention upsampling layer and a residual block structure, and an attention upsampling module 2 having an attention upsampling layer and a residual block structure. Since both the upsampling attention network and the downsampling attention network have two attention modules, the background fine-tuning control information has four sets of background LoRA parameters (first background fine-tuning information 1, first background fine-tuning information 2, second background fine-tuning information 1, and second background fine-tuning information 2), and the object fine-tuning control information has four sets of object LoRA parameters (first object fine-tuning information 1, first object fine-tuning information 2, second object fine-tuning information 1, and second object fine-tuning information 2). Based on this, first, the sample latent space features and denoising control information are input into the denoising network. Then, the attention downsampling layer in the attention downsampling module 1 performs attention calculation on the sample latent space features based on the denoising control information, the first background fine-tuning information 1 and the first object fine-tuning information 1 to obtain the first attention calculation result, and the residual block structure in the attention downsampling module 1 performs residual processing on the first attention calculation result to obtain the first downsampling result. Further, the attention downsampling layer in the attention downsampling module 2 performs attention calculation on the first downsampling result based on the denoising control information, the first background fine-tuning information 2 and the first object fine-tuning information 2 to obtain the second attention calculation result, and the residual block structure in the attention downsampling module 2 performs residual processing on the second attention calculation result to obtain the sample downsampling result. Then, the spatial transformation structure performs spatial transformation on the sample downsampling result to obtain the sample downsampling result after spatial transformation. Further, the attention upsampling layer in the attention upsampling module 2 performs attention calculation on the sample downsampling result after spatial transformation based on the denoising control information, the second background fine-tuning information 1, and the second object fine-tuning information 1 to obtain a third attention calculation result, and the residual block structure in the attention downsampling module 2 performs residual processing on the third attention calculation result to obtain a first upsampling result. Finally, the attention upsampling layer in the attention upsampling module 1 performs attention calculation on the first upsampling result based on the denoising control information, the second background fine-tuning information 2, and the second object fine-tuning information 2 to obtain a fourth attention calculation result, and the residual block structure in the attention upsampling module 2 performs residual processing on the fourth attention calculation result to obtain a predicted image feature.
[0273] like Fig. 13BAs shown in FIG. 1 , it is the specific process of diffusion processing and denoising processing of the image generation model in model training. Specifically, before the sample reference image is input into the image generation model, the sample reference image is first encoded into a sample image encoding feature. Then, the sample image encoding feature is subjected to T noise addition operations through the diffusion network, and the sample image encoding feature is converted into an image feature close to pure noise, and a sample latent space feature is obtained, wherein the sample latent space feature can no longer reflect the image information possessed by the sample reference image. Furthermore, through the denoising network, based on the given denoising control information, background fine-tuning control information and object fine-tuning control information, the sample latent space feature is subjected to T denoising operations to eliminate the noise carried on the sample latent space feature, and the sample latent space feature is restored to the form of an image feature to obtain a predicted image feature. Among them, the predicted image feature can reflect the image information possessed by the sample reference image to a certain extent.
[0274] The benefit of this embodiment is that the denoising process is divided into a downsampling stage and an upsampling stage, and different background fine-tuning information and different object fine-tuning information are relied on in the upsampling stage and the downsampling stage, which can improve the effectiveness of fine-tuning. Specifically, in the downsampling stage, the downsampling attention network performs feature downsampling according to the first object fine-tuning information, the first background fine-tuning information, and the denoising control information, and in the upsampling stage, the upsampling attention network performs feature upsampling according to the second object fine-tuning information, the second background fine-tuning information, and the denoising control information. In this way, in both the downsampling stage and the upsampling stage, the three aspects of control information, the background fine-tuning information, the object fine-tuning information and the denoising control information, are combined for guidance and constraints, which can achieve accurate fine-tuning and flexible control of image denoising. This method enables the model to perform image denoising under various conditions, improves the image denoising ability of the model, and is conducive to the model's learning of the coordination of the background and foreground, thereby improving the replacement processing accuracy and image generation accuracy of the model.
[0275] The following is a detailed description of the situation in which the downsampling attention network in the denoising network of the image generation model has an attention downsampling module.
[0276] In the disclosed embodiment, the first background fine-tuning information includes a first background query feature, a first background key feature, and a first background value feature. The first object fine-tuning information includes a first object query feature, a first object key feature, and a first object value feature. The denoising control information includes a reference image description embedding feature, a sample image description embedding feature, and a background image description embedding feature.
[0277] Please refer to Fig.14 In one embodiment, step 1210 specifically includes but is not limited to the following steps 1410-1440:
[0278] Step 1410: Based on the sample latent space feature, the first background query feature, the first background key feature, and the background image description embedding feature, a first attention calculation is performed through a downsampling attention network to obtain a background image attention result;
[0279] Step 1420, based on the sample latent space feature, the first object query feature, the first object key feature, and the sample image description embedding feature, a second attention calculation is performed through the downsampling attention network to obtain a sample object image attention result;
[0280] Step 1430, performing feature replacement on the background image attention result based on the sample object image attention result to obtain the replaced background image attention result;
[0281] Step 1440: Based on the replaced background image attention result, the first background value feature, the first object value feature and the reference image description embedding feature, a third attention calculation is performed through the downsampling attention network to obtain a sample downsampling result.
[0282] Steps 1410-1440 are described in detail below.
[0283] In step 1410, the background image attention result is used to indicate the result of fine-tuning the background features of the sample latent space features according to the first background fine-tuning information and the first description information.
[0284] In the specific implementation of this embodiment, step 1410 specifically includes but is not limited to the following steps:
[0285] Performing weighted processing on the sample latent space feature and the first background query feature to obtain a first weighted image feature;
[0286] Performing weighted processing on the background image description embedding feature and the first background key feature to obtain a second weighted image feature;
[0287] The first weighted image feature and the second weighted image feature are matrix multiplied to obtain the background image attention result.
[0288] Specifically, first, based on the preset first weight and second weight, the sample latent space feature and the first background query feature are weighted, and the product of the first weight and the sample latent space feature is added to the product of the second weight and the first background query feature to obtain the first weighted image feature. Next, based on the preset third weight and fourth weight, the background image description embedded feature and the first background key feature are weighted, and the third weight and the background image description embedded feature are added to the product of the fourth weight and the first background key feature to obtain the second weighted image feature. Finally, the first weighted image feature and the second weighted image feature are matrix multiplied, and each element in the first weighted image feature is multiplied with each element of the second weighted image feature to obtain the background image attention result.
[0289] It should be noted that the sum of the first weight and the second weight, and the sum of the third weight and the fourth weight may be 1 or may not be 1, without limitation.
[0290] In step 1420, the sample object image attention result is used to indicate the result of fine-tuning the object features of the sample latent space features according to the first object fine-tuning information and the second description information.
[0291] When this embodiment is implemented, the specific process of step 1420 is similar to the specific process of step 1410. The difference is that step 1410 is an attention calculation for background features, while step 1420 is an attention calculation for sample objects. The objects to be calculated for attention are different, but the processing methods are basically the same. To save space, it will not be repeated.
[0292] In step 1430, the replaced background image attention result is used to indicate the result of adjusting the reference object features in the sample latent space features to the target object features.
[0293] In the specific implementation of this embodiment, first, the sample object features corresponding to the sample object are determined in the sample object image attention result, and the reference object features corresponding to the reference object are determined in the background image attention result. Then, the reference object features in the background image attention result are replaced with the sample object features, and the background image attention result after feature replacement is determined as the replaced background image attention result.
[0294] In step 1440, step 1440 specifically includes but is not limited to the following steps:
[0295] Performing weighted processing on the first background value feature, the first object value feature and the reference image description embedding feature to obtain a sample weighted image feature;
[0296] The sample weighted image features and the replaced background image attention results are matrix multiplied to obtain the sample downsampling result.
[0297] Specifically, first, based on the preset fifth weight, sixth weight and seventh weight, the first background value feature, the first object value feature and the reference image description embedding feature are weighted, and the product of the fifth weight and the first background value feature, the product of the sixth weight and the first object value feature, and the product of the seventh weight and the reference image description embedding feature are added to obtain the sample weighted image feature. Then, the sample weighted image feature and the replaced background image attention result are matrix multiplied, and each element in the sample weighted image feature is multiplied with each element of the replaced background image attention result to obtain the sample downsampling result.
[0298] It should be noted that the sum of the fifth weight, the sixth weight and the seventh weight may be 1 or may not be 1, without limitation.
[0299] The benefit of this embodiment is that when the cross-attention mechanism is introduced in the downsampling process, the cross-attention mechanism is improved and optimized. Specifically, when the background of the sample latent space feature is fine-tuned based on the background image description embedding feature corresponding to the first description information, the image features of the background image in the key channel and the query channel (the first background key feature and the first background query feature) are integrated. At the same time, when the object of the sample latent space feature is fine-tuned based on the object image description embedding feature corresponding to the second description information, the image features of the sample object image in the key channel and the query channel (the first object key feature and the first object query feature) are integrated. This method realizes object fine-tuning and background fine-tuning in details, which can improve the accuracy of downsampling and make the object features and background features of the sample downsampling results have better coordination. In addition, this embodiment also introduces a feature replacement method in downsampling, which adjusts the reference object features in the sample latent space features to the target object features, which can enhance the accuracy of image feature expression.
[0300] In order to reduce the adverse effects of the object replacement process on background features, the embodiments of the present disclosure provide a feature replacement scheme based on balanced weights, which can maintain good weight consistency between the generated predicted image features and the various features of the sample reference image.
[0301] Please refer to Fig.15 In one embodiment, step 1430 specifically includes but is not limited to the following steps 1510-1530:
[0302] Step 1510: determining a first word number of a reference object based on the first description information, and determining a second word number of a sample object based on the second description information;
[0303] Step 1520: Determine the word weight between the reference object and the sample object based on the first word number and the second word number;
[0304] Step 1530: Based on the word weights, replace the reference object features in the background image attention result with the sample object features in the sample object image attention result to obtain the replaced background image attention result.
[0305] Steps 1510-1530 are described in detail below.
[0306] In step 1510, the first word number is used to indicate the number of words used to describe the reference object in the first description information.
[0307] The second word number is used to indicate the number of words used to describe the sample object in the second description information.
[0308] For example, when the first description information is "cat lying in grass", the reference object is "cat", and the first word number (token length) is 1; when the second description information is "Dolly Cat lying in grass", the sample object is "Dolly Cat", and the second word number (token length) is 2.
[0309] In the specific implementation of this embodiment, first, the first description information is subjected to word extraction to obtain words used to describe the reference object, and the first word number of the reference object is determined according to the total number of words extracted. Next, the second description information is subjected to word extraction to obtain words used to describe the sample object, and the second word number of the sample object is determined according to the total number of words extracted.
[0310] In step 1520, the word weight is used to indicate the weight ratio of each word used to describe the sample object in the second description information.
[0311] In the specific implementation of this embodiment, first, the first word number is divided by the second word number to obtain a first proportion. Then, the weight of the word used to describe the reference object in the first description information is multiplied by the first proportion to obtain the word weight between the reference object and the sample object.
[0312] For example, when the weight of each word in the first description information is 1, the weight of the reference object, "cat" is 1. At this time, since the ratio of the number of first words to the number of second words is 1:2, the word weight is 0.5, and the weight of the sample object "Dolly Cat" becomes 1×0.5=0.5.
[0313] In step 1530, the sample object feature refers to a part of the feature elements describing the sample object in the sample object image attention result.
[0314] The reference object feature refers to a part of the feature elements describing the reference object in the background image attention result.
[0315] In the specific implementation of this embodiment, first, the sample object features in the sample object image attention result are weighted according to the word weight to obtain the weighted sample object features. Then, the reference object features in the background image attention result are replaced with the weighted sample object features, and the background image attention result after feature replacement is determined as the replaced background image attention result.
[0316] For example, when the reference object (cat) is replaced by the sample object (Dolly Cat), the sample object features generated by each word of the sample object (Dolly Cat) in the sample object image attention result are weighted by 0.5, so that the sum of the weights of the features generated by the words used to describe the sample object in the background image attention result after replacement is 1, taking into account the word length.
[0317] The advantage of this embodiment is that it adopts a feature replacement method based on balanced weights. A word weight is calculated according to the number of first words of the reference object in the first description information and the number of second words of the sample object in the second description information. When the reference object features in the background image attention result are replaced by the sample object features in the sample object image attention result, the weight ratio of the sample object features is adjusted according to the word weight, so that the word weights of each feature before and after the feature replacement are balanced, thereby maintaining a good weight consistency between the generated predicted image features and the various features of the sample reference image, reducing the impact of object replacement on background features, improving the coordination between the object and background of the generated image, and thereby improving the image generation accuracy.
[0318] like Fig.16As shown, it is a schematic diagram of the implementation process of generating sample downsampling results by a downsampling attention network with one attention module. Specifically, when the downsampling attention network has only one attention module, the first background fine-tuning information includes the first background query feature (LoRA Q1), the first background key feature (LoRA K1) and the first background value feature (LoRA V1); the first object fine-tuning information includes the first object query feature (LoRA Q2), the first object key feature (LoRAK2) and the first object value feature (LoRA V2). Based on this, firstly, the first background query feature (LoRA Q1) and the sample latent space feature as the query component are weighted to obtain the first weighted image feature, and the first background key feature (LoRAK1) and the background image description embedding feature as the key component are weighted to obtain the second weighted image feature, and then the first weighted image feature and the second weighted image feature are matrix multiplied to obtain the background image attention result. Next, the first object query feature (LoRA Q2) and the sample latent space feature as the query component are weighted to obtain the third weighted image feature, and the first object key feature (LoRA K2) and the sample image description embedding feature as the key component are weighted to obtain the fourth weighted image feature, and then the third weighted image feature and the fourth weighted image feature are matrix multiplied to obtain the sample image attention result. Furthermore, based on the target object feature in the sample image attention result and the reference object feature in the background image attention result, a feature replacement operation is performed under the premise of adding the word weight w to obtain the replaced background image attention result. Finally, the first background value feature (LoRA V1), the first object value feature (LoRA V2) and the reference image description embedding feature as the value component are weighted to obtain the sample weighted image feature, and then the sample weighted image feature and the replaced background image attention result are matrix multiplied to obtain the sample downsampling result.
[0319] It should be noted that when there are multiple attention downsampling modules in the downsampling attention network, each attention downsampling module can also perform downsampling processing according to the above steps 1410-1440. Among them, the input of the first attention downsampling module is the sample latent space feature. For other attention downsampling modules except the first attention downsampling module, the output of the previous attention downsampling module is used as the input of the next attention downsampling module, and the output of the last attention downsampling module is used as the sample downsampling result.
[0320] In addition, in the embodiments of the present disclosure, the upsampling attention network in the denoising network has one attention upsampling module, or has multiple attention upsampling modules, which is similar to the various situations of the downsampling attention network described above. To save space, no further description is given.
[0321] Step 350 is described in detail below.
[0322] In step 350, an image generation model is trained based on the predicted image features of a plurality of image-text sample pairs and sample reference images.
[0323] Please refer to Fig.17 In one embodiment, step 350 specifically includes but is not limited to the following steps 1710-1740:
[0324] Step 1710: for each image-text sample pair, obtain the reference noise in the sample reference image, and perform noise prediction based on the predicted image features to obtain predicted noise;
[0325] Step 1720: Calculate a sub-loss function based on the comparison between the reference noise and the predicted noise;
[0326] Step 1730: Determine a total loss function based on the sub-loss function of each image-text sample pair;
[0327] Step 1740: Train the image generation model based on the total loss function.
[0328] Steps 1710-1740 are described in detail below.
[0329] In step 1710, reference noise is used to indicate the degree to which noise is added to the expected result.
[0330] Prediction noise is obtained through predictions of multiple prediction time steps. Prediction noise is used to indicate the noise contained in the denoised result (predicted image features) produced by the image generation model at the last prediction time step.
[0331] The prediction time step is used to indicate the noise adding frequency and the denoising frequency of the sample image encoding features corresponding to the sample reference image of the input image generation model.
[0332] In the specific implementation of this embodiment, for each image-text sample pair, noise extraction is performed on the sample reference image to obtain reference noise in the sample reference image. The reference noise is different from the random noise added in step 530. Figure 1 To.
[0333] Furthermore, since the image generation model will denoise the denoising result (denoised image) generated in the previous prediction time step again at each prediction time step, and generate predicted image features after denoising processing for multiple prediction time steps, at this time, it is only necessary to extract noise from the predicted image features to obtain the predicted noise.
[0334] It should be noted that the noise contained in the denoising results of different prediction time steps is different.
[0335] In step 1720 , a sub-loss function is used to indicate the degree of difference between the reference noise and the predicted noise of a single image-text sample pair.
[0336] In the specific implementation of this embodiment, the noise difference calculation is performed on the reference noise and the predicted noise to obtain a noise difference result. Finally, the sub-loss function is calculated according to the noise difference result.
[0337] In step 1730, the total loss function is used to indicate the overall difference between the reference noise and the predicted noise of all image-text sample pairs. The smaller the total loss function, the smaller the overall difference between the reference noise and the predicted noise of all image-text sample pairs, and the higher the image generation accuracy of the image generation model.
[0338] When this embodiment is implemented, the sub-loss functions of each image-text sample pair are averaged to obtain the total loss function. Specifically, first, the total number of image-text sample pairs is determined. Then, the sub-loss functions of all image-text sample pairs are added to obtain the sum of the sub-loss functions. Finally, the sub-loss functions are divided by the total number of image-text sample pairs to obtain the total loss function.
[0339] In step 1740, the model parameters of the image generation model are adjusted with the training goal of minimizing the total loss function, and the above steps 310-350 are repeated to implement iterative training of the image generation model. The model parameters that minimize the total loss function are used as the final model parameters, and the image generation model with the final model parameters is used as the trained image generation model.
[0340] The benefit of this embodiment is that, based on a supervised learning approach, a sub-loss function for each image-text sample pair is determined according to the noise difference between the reference noise in the sample baseline image of each image-text sample pair and the predicted noise in the predicted image features, and a total loss function is constructed based on multiple sub-loss functions. This approach can train the image generation model by minimizing the difference between the reference noise and the noise prediction results, which is beneficial to improving the model training effect, and thereby improving the model's prediction accuracy for the generated image.
[0341] In the disclosed embodiment, the predicted noise is obtained through prediction of multiple prediction time steps.
[0342] The prediction time step is used to indicate the noise adding frequency and the denoising frequency of the sample image encoding features corresponding to the sample reference image of the input image generation model.
[0343] Please refer to Fig.18 In one embodiment, step 1720 specifically includes but is not limited to the following steps 1810-1820:
[0344] Step 1810: Calculate the difference between the reference noise and the predicted noise to obtain a noise difference;
[0345] Step 1820: Calculate the regularization term on the noise difference to obtain a sub-loss function.
[0346] Steps 1810-1820 are described in detail below.
[0347] In step 1810, the noise difference is used to indicate the difference in values between the reference noise and the predicted noise of each pixel point on the sample reference image.
[0348] In the specific implementation of this embodiment, first, for each pixel point on the sample reference image, the reference noise and the predicted noise of the pixel point are subtracted (the predicted noise is subtracted from the reference noise) to obtain the noise difference of each pixel point. Then, the noise differences of multiple pixel points are integrated to obtain the noise difference of the image-text sample pair.
[0349] In step 1820, first, for each image-text sample pair, the noise difference is regularized to obtain a regularization term calculation result, which is used to indicate the regularization difference between the reference noise and the predicted noise of the image-text sample pair. Next, the regularization term calculation result is converted into the form of a conditional loss function, and the converted conditional loss function is determined as a sub-loss function. Among them, the sub-loss function of the embodiment of the present disclosure can be expressed as shown in formula (1):
[0350] Formula (1);
[0351] in, is the sub-loss function; is the reference noise, Indicates that the reference noise follows a standard normal distribution (mean 0, variance 1). t represents the prediction time step, which is used to gradually update the sample latent space features during the image generation process. represents the latent space features of the sample at the prediction time step t, is the result of adding noise to the sample image encoding feature z at the prediction time step t. Both y and c refer to control information (including denoising control information, background fine-tuning control information, and object fine-tuning control information), which are used to conditionally constrain the image generation process. x is the sample reference image used for training. To indicate the encoder The expectation for the input x. Refers to the conditional constraints , at the prediction time step t, the latent space features of the sample Prediction noise after denoising. is the result of the regularization calculation.
[0352] The benefit of this embodiment is that by calculating the regularization term loss between the reference noise contained in the sample baseline image of the image-text sample pair and the predicted noise prediction contained in the predicted image features, the regularization term calculation results of each image-text sample pair are obtained, and the regularization term calculation results (L2 loss) are used as a sub-loss function to train the image generation model, which can better adjust the image generation process of the image generation model and make the generation process more controllable. At the same time, conditional information (denoising control information, background fine-tuning control information and object fine-tuning control information) is added to the sub-loss function, which can make the generated predicted image features closely related to the given conditions, thereby improving the degree to which the model pays attention to the coordination between the object and the background in the image processing of replacing the object in the background image, thereby improving the accuracy of the model in the replacement processing, so that the finally generated target image has better image quality.
[0353] like Fig.19 , which is an overall flow chart of the model training of the embodiment of the present disclosure. Specifically, the sample background image, the sample object image, the expected effect image, the image description information of the expected effect image, the first description information of the sample background image, and the second description information of the sample object image are used as training data for the model training.
[0354] First, a reference noise is generated using a random number i, and the reference noise is added to the expected effect image to obtain a sample reference image. The specific process is similar to the above steps 510-530.
[0355] Next, the encoder is used to encode the sample reference image to obtain the sample encoded image feature Z, and the sample encoded image feature Z is diffused through the diffusion network. The sample encoded image feature Z is denoised T times using the diffusion network to obtain the sample latent space feature Z T, and its specific process is similar to the above steps 1110-1120. Further, the image description information of the expected effect image, the first description information of the sample background image, and the second description information of the sample object image are used as text condition constraints and converted into a text form that meets the input requirements of the text encoder τ, and the image description information of the expected effect image, the first description information of the sample background image, and the second description information of the sample object image are input to the text encoder, and the text encoder outputs the reference image description embedding feature, the background image description embedding feature, and the sample image description embedding feature, and its specific process is similar to the above steps 710-740. Furthermore, the sample background image and the sample object image are respectively input into the fine-tuning network LoRA to obtain background fine-tuning control information and object fine-tuning control information, wherein the background fine-tuning control information includes the first background feature QKV1-A1*, the second background feature QKV2-A1*, the third background feature QKV3-A1* and the fourth background feature QKV4-A1*; the object fine-tuning control information includes the first object feature QKV1-A2*, the second object feature QKV2-A2*, the third object feature QKV3-A2* and the fourth object feature QKV4-A2*, and the specific process is similar to the above steps 910-930 and steps 1010-1030.
[0356] Next, the denoising network performs the latent space feature Z of the sample T When denoising, firstly, based on the first background feature QKV1-A1*, the second background feature QKV2-A1*, the first object feature QKV1-A2*, the second object feature QKV2-A2*, the reference image description embedding feature, the background image description embedding feature and the sample image description embedding feature, the sample latent space feature Z is denoised by each attention downsampling module of the downsampling attention network of the denoising network. T Down-sampling is performed to obtain the sample down-sampling result. Then, based on the third background feature QKV3-A1*, the fourth background feature QKV4-A1*, the third object feature QKV3-A2*, the fourth object feature QKV4-A2*, the reference image description embedding feature, the background image description embedding feature and the sample image description embedding feature, the sample down-sampling result is up-sampled through each attention up-sampling module of the up-sampling attention network of the denoising network to obtain the denoising result Z after one denoising operation. T-1' . Further, the denoising network is used to denoise the result Z at the prediction time step T T-1' According to the above process, denoising is performed. After T-1 denoising times, the predicted image feature Z is obtained. ’ , and its specific process is similar to the above steps 1210-1220.
[0357] Finally, based on the predicted image feature Z ’Noise prediction is performed to obtain predicted noise, a loss function LOSS is constructed based on the reference noise and the predicted noise, and the image generation model is iteratively trained based on the loss function LOSS until the image generation model meets the training requirements. The specific process is similar to the above steps 1710-1740. To save space, it will not be repeated.
[0358] The following is a detailed description of an image generation method according to an embodiment of the present disclosure.
[0359] According to an embodiment of the present disclosure, an image generating method is provided.
[0360] This image generation method is generally used in business scenarios where a target object (person, animal, object, etc.) needs to be embedded into a background image and the coordination between the embedded target object and the background is required to be high, such as Figure 2A-2D The object display scene shown, etc. The disclosed embodiment provides a solution for image generation through an image generation model based on image description, background image and background description, object image and object description, which can improve the coordination between the object and the background in the image processing of object replacement in the background image, thereby improving the accuracy of the replacement process.
[0361] like Fig. 20 As shown, the image generation method according to an embodiment of the present disclosure may be executed by an electronic device, which may be Figure 1 The image processing server or the target terminal shown. An image generation method according to an embodiment of the present disclosure may include:
[0362] Step 2010: acquiring a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information of the target object;
[0363] Step 2020: generating denoising prompt information based on the background description information, the object description information, and the target image description information;
[0364] Step 2030: Generate background fine-tuning prompt information based on the target background image, and generate object fine-tuning prompt information based on the target object image;
[0365] Step 2040: Based on the preset noise image, denoising prompt information, object fine-tuning prompt information and background fine-tuning prompt information, image generation is performed through the trained image generation model to obtain a target image.
[0366] Steps 2010-2040 are described in detail below.
[0367] In step 2010, a target object image of a target object, a target background image, object description information of the target object image, background description information of the target background image, and target image description information are acquired.
[0368] The target object image of the target object refers to a series of images that can reflect the object characteristics of the target object.
[0369] The target background image refers to a background image to which the target object is to be added, wherein the target background image contains a reference object to be replaced by the target object in the target object image.
[0370] The background description information of the target background image is used to describe the image scene and background content of the target background image.
[0371] The object description information of the target object image is used to describe the object category, object characteristics, etc. of the target object in the target object image.
[0372] The target description information is used to describe the replacement from the reference object to the target object.
[0373] When this embodiment is implemented, the specific process of step 2010 is similar to the specific process of the above step 310. To save space, it will not be repeated.
[0374] In step 2020, denoising prompt information is generated based on the background description information, the object description information, and the target image description information.
[0375] The denoising hint information is used as a text condition constraint to assist the image generation model in performing image denoising during image generation, so as to improve the accuracy of image denoising and make the image denoising effect meet the actual requirements.
[0376] In the specific implementation of this embodiment, step 2020 is similar to the above steps 710-720, and will not be described in detail to save space.
[0377] In step 2030, background fine-tuning prompt information is generated based on the target background image, and object fine-tuning prompt information is generated based on the target object image.
[0378] Background fine-tuning hints are used to guide the image generation model to fine-tune background features during the denoising process.
[0379] Object fine-tuning cues are used to guide the image generation model to fine-tune object features during the denoising process.
[0380] In the specific implementation of this embodiment, the specific process of generating background fine-tuning prompt information based on the target background image is similar to the specific process of the above steps 910-930. The specific process of generating object fine-tuning prompt information based on the target object image is similar to the specific process of the above steps 1010-1030. To save space, it will not be repeated.
[0381] In step 2040, based on the preset noise image, denoising prompt information, object fine-tuning prompt information and background fine-tuning prompt information, image generation is performed through the trained image generation model to obtain a target image.
[0382] The preset noise image is a noise image generated based on random numbers and obeying Gaussian distribution, and its generation method is similar to the above step 520. To save space, it will not be described in detail.
[0383] The trained image generation model is generated according to the training method of the image generation model in the above-mentioned embodiment.
[0384] The target image is used to indicate a result of replacing a reference object in a target background image with a target object of the target object image.
[0385] To save space, the specific process of obtaining the target image by performing image generation through the image generation model based on the preset noise image, denoising prompt information, object fine-tuning prompt information and background fine-tuning prompt information in the embodiment of the present disclosure will be described in detail below. No further description will be given here.
[0386] Through steps 2010-2040, in the embodiment of the present disclosure, based on the background description information of the target background image, the object description information of the target object image, and the target image description information of the image to be generated, denoising prompt information is generated. Since the background description information focuses on describing the background from the text aspect, the object description information focuses on describing the target object from the text aspect, and the target image description information focuses on describing the overall image after object replacement from the text aspect, the denoising prompt information comprehensively expresses various aspects of the object replacement process from the text perspective. The background fine-tuning prompt information focuses on the background information obtained from the content reflected by the image itself, and the object fine-tuning prompt information focuses on the object information obtained from the content reflected by the image itself. Therefore, the background fine-tuning prompt information, the object fine-tuning prompt information, and the denoising prompt information are combined to perform denoising on the target latent space features corresponding to the preset noise image. Such denoising processing considers relatively comprehensive factors, fully considers the coordination between the object and the background in the image processing for object replacement, and can improve the feature quality of the generated target image features. Finally, the target image features are feature decoded using a decoding network to obtain the target image to be generated, which can make the foreground and background in the target image have better coordination.
[0387] In the disclosed embodiment, the trained image generation model includes a diffusion network, a denoising network, and a decoding network.
[0388] The decoding network is used to transform the image features in the target denoising result from the latent vector space to the pixel space.
[0389] Please refer to Fig.21 In one embodiment, step 2040 specifically includes but is not limited to the following steps 2110-2130:
[0390] Step 2110: Diffusion processing is performed on the encoded image features of the preset noise image based on the diffusion network to obtain target latent space features;
[0391] Step 2120: Based on the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information, the target latent space features are denoised by a denoising network to obtain a target denoising result;
[0392] Step 2130: Feature decoding is performed on the target denoising result based on the decoding network to obtain a target image.
[0393] Steps 2110-2130 are described in detail below.
[0394] The encoded image features of the preset noise image are used to indicate image features of the preset noise image in a vector space (latent space).
[0395] The target latent space features are used to indicate the result of adding noise to the encoded image features of the preset noise image at a fixed time step through the diffusion network.
[0396] The target denoising result is used to indicate the image features that meet the constraint requirements of the target image description information after the target latent space features are denoised by the denoising network at a fixed time step.
[0397] In the specific implementation of this embodiment, the specific process of step 2110 is similar to the above steps 1110-1120, and the specific process of step 2120 is similar to the above steps 1210-1220. To save space, they are not repeated here.
[0398] In step 2130, the target denoising result is input into the decoding network, and the image features in the target denoising result are mapped from the latent vector space back to the original pixel space through the decoding network to generate a noise-free target image that is consistent with the target description information.
[0399] The benefit of this embodiment is that the diffusion network is first used to perform multiple denoising on the encoded image features of the preset noise image to obtain the target latent space features. Furthermore, the denoising network is used to perform multiple denoising on the target latent space features to achieve the prediction of the source image. In this process, the predicted image features are fine-tuned using denoising prompt information, background fine-tuning prompt information, and object fine-tuning prompt information to achieve feature information adjustment at the image angle and text angle, so that the target object and the background in the final predicted target denoising result are more coordinated. Finally, the decoding network is used to decode the target denoising result to obtain the predicted target image, which can improve the coordination between the object and the background in the image processing for object replacement.
[0400] like Fig. 22 As shown, it is a schematic diagram of a specific application module of the image generation method of the embodiment of the present disclosure. Specifically, first, for the background image, a fine-tuning generation model (fine-tuning network) is used to extract features to obtain background fine-tuning prompt information, and its specific process is similar to the above steps 910-930; for the foreground target image containing the target object, a fine-tuning generation model (fine-tuning network) is used to extract features to obtain object fine-tuning prompt information, and its specific process is similar to the above steps 1010-1030. Further, the background fine-tuning prompt information, the object fine-tuning prompt information, and the generally generated target image description information are input into the image generation model, and the target latent space features corresponding to the preset noise image are diffused and denoised by the image generation model, and the fine-tuning feature replacement operation is introduced in the denoising process to obtain the target image features. Finally, the target image features are converted into a target image by using a decoding network for output, thereby generating a target image with the background image as the background and the target object as the foreground, and its specific process is similar to the above steps 2110-2140. In order to save space, it will not be repeated.
[0401] like Fig.23 , is an overall flow chart of image generation based on the image generation model of the embodiment of the present disclosure. Specifically, the target background image, the target object image, the target image description information corresponding to the image to be generated, the background description information of the target background image, and the object description information of the target object image are taken as input.
[0402] First, a random noise graph is generated using a random number i, and the specific process is similar to the above step 520.
[0403] Next, the random noise image is encoded by the encoder to obtain the noise-encoded image feature Z, and the noise-encoded image feature Z is diffused by the diffusion network. The noise-encoded image feature Z is denoised T times by the diffusion network to obtain the target latent space feature Z T, and its specific process is similar to the above steps 1110-1120. Further, the target image description information, background description information and object description information are used as text condition constraints, converted into a text form that meets the input requirements of the text encoder τ, and the target image description information, background description information and object description information are input to the text encoder, and the text encoder outputs the denoising prompt information, and its specific process is similar to the above steps 710-740. Furthermore, the target background image and the target object image are respectively input into the fine-tuning network LoRA to obtain background fine-tuning control information and object fine-tuning control information, wherein the background fine-tuning prompt information includes the first background feature QKV1-A1*, the second background feature QKV2-A1*, the third background feature QKV3-A1* and the fourth background feature QKV4-A1*; the object fine-tuning prompt information includes the first object feature QKV1-A2*, the second object feature QKV2-A2*, the third object feature QKV3-A2* and the fourth object feature QKV4-A2*, and the specific process is similar to the above steps 910-930 and steps 1010-1030.
[0404] Next, the denoising network is used to quantize the target latent space feature Z T When denoising, firstly, based on the first background feature QKV1-A1*, the second background feature QKV2-A1*, the first object feature QKV1-A2*, the second object feature QKV2-A2*, and the denoising prompt information, the target latent space feature Z is denoised by each attention downsampling module of the downsampling attention network of the denoising network. T Down-sampling is performed to obtain the target down-sampling result. Then, based on the third background feature QKV3-A1*, the fourth background feature QKV4-A1*, the third object feature QKV3-A2*, the fourth object feature QKV4-A2*, the reference image description embedding feature, and the denoising prompt information, the target down-sampling result is up-sampled through each attention up-sampling module of the up-sampling attention network of the denoising network to obtain the denoising result Z after one denoising operation. T-1' . Further, the denoising network is used to denoise the result Z at the prediction time step T T-1' According to the above process, denoising is performed. After T-1 denoising times, the target image feature Z is obtained. ’ Finally, based on the decoding network, the target image feature Z ’ Feature decoding is performed to obtain the target image I, and the specific process is similar to the above steps 2110-2140. To save space, it will not be repeated.
[0405] The following describes the apparatus and devices of the embodiments of the present disclosure.
[0406] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.
[0407] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present application is obtained.
[0408] Fig.24 A schematic diagram of the structure of a training device 2400 for an image generation model provided in an embodiment of the present disclosure. The training device 2400 for an image generation model includes:
[0409] A first acquisition unit 2410 is used to acquire a plurality of image-text sample pairs, wherein each image-text sample pair includes a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to an expected result of replacing a reference object in the sample background image with the sample object;
[0410] A first generating unit 2420, configured to generate denoising control information based on the first description information, the second description information, and the sample image description information;
[0411] A second generating unit 2430, configured to generate background fine-tuning control information based on the sample background image, and generate object fine-tuning control information based on the sample object image;
[0412] A denoising unit 2440 is used to perform denoising processing on the sample latent space features corresponding to the sample reference image through the image generation model based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information to obtain predicted image features;
[0413] The training unit 2450 is used to train the image generation model based on the predicted image features of the plurality of image-text sample pairs and the sample reference image to obtain a trained image generation model. The trained image generation model is used to generate an image according to the target object image, the target background image, the object description information of the target object image, the background description information of the target background image, and the target image description information.
[0414] Optionally, the image generation model includes a denoising network; the denoising network includes an upsampling attention network and a downsampling attention network; the background fine-tuning control information includes first background fine-tuning information and second background fine-tuning information; the object fine-tuning control information includes first object fine-tuning information and second object fine-tuning information;
[0415] The denoising unit 2440 includes:
[0416] A downsampling subunit (not shown), configured to perform downsampling processing on the sample latent space features through a downsampling attention network based on the first object fine-tuning information, the first background fine-tuning information, and the denoising control information, to obtain a sample downsampling result;
[0417] The upsampling subunit (not shown) is used to upsample the sample downsampling result through the upsampling attention network based on the second object fine-tuning information, the second background fine-tuning information, and the denoising control information to obtain the predicted image features.
[0418] Optionally, the first background fine-tuning information includes a first background query feature, a first background key feature, and a first background value feature; the first object fine-tuning information includes a first object query feature, a first object key feature, and a first object value feature; the denoising control information includes a reference image description embedding feature, a sample image description embedding feature, and a background image description embedding feature;
[0419] The downsampling subunit (not shown) includes:
[0420] A first attention module (not shown), configured to perform a first attention calculation through a downsampling attention network based on the sample latent space feature, the first background query feature, the first background key feature, and the background image description embedding feature, to obtain a background image attention result;
[0421] A second attention module (not shown), configured to perform a second attention calculation through a downsampling attention network based on the sample latent space feature, the first object query feature, the first object key feature, and the sample image description embedding feature, to obtain a sample object image attention result;
[0422] A replacement module (not shown) is used to perform feature replacement on the background image attention result based on the sample object image attention result to obtain a replaced background image attention result;
[0423] The third attention module (not shown) is used to perform a third attention calculation through a downsampling attention network based on the replaced background image attention result, the first background value feature, the first object value feature and the reference image description embedding feature to obtain a sample downsampling result.
[0424] Optionally, a first attention module (not shown) is used to:
[0425] Performing weighted processing on the sample latent space feature and the first background query feature to obtain a first weighted image feature;
[0426] Performing weighted processing on the background image description embedding feature and the first background key feature to obtain a second weighted image feature;
[0427] The first weighted image feature and the second weighted image feature are matrix multiplied to obtain the background image attention result.
[0428] Optionally, a replacement module (not shown) is used to:
[0429] Determining a first word number of a reference object based on the first description information, and determining a second word number of a sample object based on the second description information;
[0430] determining a word weight between the reference object and the sample object based on the first word number and the second word number;
[0431] Based on the word weights, the reference object features in the background image attention result are replaced with the sample object features in the sample object image attention result to obtain the replaced background image attention result.
[0432] Optionally, a third attention module (not shown) is used to:
[0433] Performing weighted processing on the first background value feature, the first object value feature and the reference image description embedding feature to obtain a sample weighted image feature;
[0434] The sample weighted image features and the replaced background image attention results are matrix multiplied to obtain the sample downsampling result.
[0435] Optionally, the first generating unit 2420 is used to:
[0436] Performing word embedding processing on the first description information to obtain background image description embedding features;
[0437] Performing word embedding processing on the second description information to obtain sample image description embedding features;
[0438] Perform word embedding processing on the sample image description information to obtain the benchmark image description embedding features;
[0439] The benchmark image description embedding features, sample image description embedding features and background image description embedding features are integrated into denoising control information.
[0440] Optionally, the second generating unit 2430 is used to:
[0441] activating a preset fine-tuning network based on the first description information;
[0442] Extracting features of the sample background image based on the activated fine-tuning network to obtain a first background query feature, a first background key feature, and a first background value feature;
[0443] The first background query feature, the first background key feature and the first background value feature are integrated into background fine-tuning control information.
[0444] Optionally, the second generating unit 2430 is used to:
[0445] activating a preset fine-tuning network based on the second description information;
[0446] Extracting features of the sample object image based on the activated fine-tuned network to obtain a first object query feature, a first object key feature, and a first object value feature;
[0447] The first object query feature, the first object key feature and the first object value feature are integrated into object fine-tuning control information.
[0448] Optionally, the sample latent space features corresponding to the sample reference image are generated by:
[0449] Performing encoding processing on the sample reference image to obtain encoding features of the sample image;
[0450] Based on the preset time step, the sample image encoding features are diffused through the diffusion network of the image generation model to obtain the sample latent space features.
[0451] Optionally, a sample reference image is generated by:
[0452] Acquire an original reference image, wherein the original reference image is used to indicate an expected result of replacing a reference object in a sample background image with a sample object;
[0453] Generate a random number that obeys a Gaussian distribution based on a predetermined random number generation model;
[0454] For each pixel in the original reference image, a random number is added to the pixel value of the pixel to obtain a sample reference image.
[0455] Optionally, the training unit 2450 includes;
[0456] An acquisition module (not shown) is used to acquire the reference noise in the sample reference image for each image-text sample pair, and perform noise prediction based on the predicted image features to obtain predicted noise;
[0457] A calculation module (not shown), configured to calculate a sub-loss function based on a comparison between the reference noise and the predicted noise;
[0458] A determination module (not shown) is used to determine a total loss function based on the sub-loss function of each image-text sample pair;
[0459] A training module (not shown) is used to train the image generation model based on the total loss function.
[0460] Optionally, the prediction noise is obtained by prediction over multiple prediction time steps;
[0461] The computing module (not shown) is used to:
[0462] Calculate the difference between the reference noise and the predicted noise to obtain the noise difference;
[0463] The regularization term is calculated on the noise difference to obtain the sub-loss function.
[0464] Optionally, a sample object image is generated by:
[0465] Get a sample image of a sample object;
[0466] Performing image segmentation on the sample image based on a preset object segmentation model to obtain a sample segmentation image having the sample object;
[0467] Perform image enhancement on the sample segmentation image to obtain a sample object image.
[0468] Fig.25 The following is a schematic diagram of the structure of an image generating device 2500 provided in an embodiment of the present disclosure. The image generating device 2500 includes:
[0469] A second acquisition unit 2510 is configured to acquire a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information of the target object, wherein the target background image includes a reference object to be replaced by the target object in the target object image, and the target image description information is used to describe the replacement from the reference object to the target object;
[0470] The third generating unit 2520 is used to generate denoising prompt information based on the background description information, the object description information, and the target image description information;
[0471] The fourth generating unit 2530 is used to generate background fine-tuning prompt information based on the target background image, and to generate object fine-tuning prompt information based on the target object image;
[0472] The image generation unit 2540 is used to generate an image based on a preset noise image, denoising prompt information, object fine-tuning prompt information and background fine-tuning prompt information through a trained image generation model to obtain a target image, wherein the trained image generation model is generated according to the above-mentioned image generation model training method; the target image is used to indicate the result of replacing a reference object in a target background image with a target object of the target object image.
[0473] Optionally, the trained image generation model includes a diffusion network, a denoising network, and a decoding network;
[0474] The image generation unit 2540 is used to:
[0475] Based on the diffusion network, the encoded image features of the preset noise image are diffused to obtain the target latent space features;
[0476] Based on the denoising hint information, the object fine-tuning hint information and the background fine-tuning hint information, the target latent space features are denoised through the denoising network to obtain the target denoising result;
[0477] The target denoising result is feature decoded based on the decoding network to obtain the target image.
[0478] Reference Fig.26 , Fig.26 A structural block diagram of a terminal for implementing the training method of the image generation model or the image generation method of the embodiment of the present disclosure, which terminal may be Figure 1The terminal includes: a radio frequency (RF) circuit 2610, a memory 2615, an input unit 2630, a display unit 2640, a sensor 2650, an audio circuit 2660, a wireless fidelity (WiFi) module 2670, a processor 2680, and a power supply 2690. Those skilled in the art will understand that Fig.26 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0479] The RF circuit 2610 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 2680 for processing; in addition, the designed uplink data is sent to the base station.
[0480] The memory 2615 may be used to store software programs and modules. The processor 2680 executes various functional applications and data processing of the target terminal by running the software programs and modules stored in the memory 2615 .
[0481] The input unit 2630 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the target terminal. Specifically, the input unit 2630 may include a touch panel 2631 and other input devices 2632.
[0482] The display unit 2640 may be used to display input information or provided information and various menus of the target terminal. The display unit 2640 may include a display panel 2641.
[0483] The audio circuit 2660, the speaker 2661, and the microphone 2662 may provide an audio interface.
[0484] In this embodiment, the processor 2680 included in the terminal can execute the image generation model training method or image generation method of the previous embodiment.
[0485] Fig. 27 A structural block diagram of a server for implementing the training method of the image generation model or the image generation method of the embodiment of the present disclosure. The server may be Figure 1The image processing server shown. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2722 (for example, one or more processors) and memory 2732, one or more storage media 2730 (for example, one or more mass storage devices) storing application programs 2742 or data 2744. Among them, the memory 2732 and the storage medium 2730 can be temporary storage or permanent storage. The program stored in the storage medium 2730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 2722 can be configured to communicate with the storage medium 2730 to execute a series of instruction operations in the storage medium 2730 on the server.
[0486] The server may also include one or more power supplies 2726, one or more wired or wireless network interfaces 2750, one or more input and output interfaces 2758, and / or one or more operating systems 2741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0487] The central processor 2722 in the server can be used to execute the training method or image generation method of the image generation model of the embodiment of the present disclosure.
[0488] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the image generation model training method or image generation method of the aforementioned embodiments.
[0489] The present disclosure also provides a computer program product, which includes a computer program. A processor of an electronic device reads and executes the computer program, so that the electronic device executes the training method or image generation method for implementing the above-mentioned image generation model.
[0490] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0491] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0492] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.
[0493] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0494] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0495] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0496] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0497] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.
[0498] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.
Claims
1. A training method for an image generation model, characterized in that: The training method comprises: Acquire a plurality of image-text sample pairs, wherein each of the image-text sample pairs comprises a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to an expected result of replacing a reference object in the sample background image with the sample object; Generate denoising control information based on the first description information, the second description information, and the sample image description information; generating background fine-tuning control information based on the sample background image, and generating object fine-tuning control information based on the sample object image; Based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, the sample latent space features corresponding to the sample reference image are denoised by the image generation model to obtain predicted image features; Based on the predicted image features of the multiple image-text sample pairs and the sample reference image, the image generation model is trained to obtain a trained image generation model, and the trained image generation model is used to generate an image based on a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information.
2. The training method of the image generation model according to claim 1, characterized in that: The image generation model includes a denoising network; the denoising network includes an upsampling attention network and a downsampling attention network; The background fine-tuning control information includes first background fine-tuning information and second background fine-tuning information; The object fine-tuning control information includes first object fine-tuning information and second object fine-tuning information; The method of performing denoising processing on the sample latent space features corresponding to the sample reference image through the image generation model based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information to obtain predicted image features includes: Based on the first object fine-tuning information, the first background fine-tuning information, and the denoising control information, down-sampling the sample latent space features through the down-sampling attention network to obtain a sample down-sampling result; Based on the second object fine-tuning information, the second background fine-tuning information, and the denoising control information, the sample downsampling result is upsampled by the upsampling attention network to obtain the predicted image feature.
3. The training method of the image generation model according to claim 2, characterized in that: The first background fine-tuning information includes a first background query feature, a first background key feature, and a first background value feature; the first object fine-tuning information includes a first object query feature, a first object key feature, and a first object value feature; the denoising control information includes a reference image description embedding feature, a sample image description embedding feature, and a background image description embedding feature; The downsampling process of the sample latent space feature is performed by the downsampling attention network based on the first object fine-tuning information, the first background fine-tuning information, and the denoising control information to obtain a sample downsampling result, including: Based on the sample latent space feature, the first background query feature, the first background key feature, and the background image description embedding feature, performing a first attention calculation through the downsampling attention network to obtain a background image attention result; Based on the sample latent space feature, the first object query feature, the first object key feature, and the sample image description embedding feature, performing a second attention calculation through the downsampling attention network to obtain a sample object image attention result; Perform feature replacement on the background image attention result based on the sample object image attention result to obtain a replaced background image attention result; Based on the replaced background image attention result, the first background value feature, the first object value feature and the reference image description embedding feature, a third attention calculation is performed through the downsampling attention network to obtain the sample downsampling result.
4. The training method of the image generation model according to claim 3, characterized in that: The performing feature replacement on the background image attention result based on the sample object image attention result to obtain the replaced background image attention result includes: Determine a first word number of the reference object based on the first description information, and determine a second word number of the sample object based on the second description information; determining a word weight between the reference object and the sample object based on the first word number and the second word number; Based on the word weights, the reference object features in the background image attention result are replaced with the sample object features in the sample object image attention result to obtain the replaced background image attention result.
5. The training method of the image generation model according to claim 3, characterized in that: The method of performing a first attention calculation based on the sample latent space feature, the first background query feature, the first background key feature, and the background image description embedding feature through the downsampling attention network to obtain a background image attention result includes: Performing weighted processing on the sample latent space feature and the first background query feature to obtain a first weighted image feature; Performing weighted processing on the background image description embedding feature and the first background key feature to obtain a second weighted image feature; Matrix multiplication is performed on the first weighted image feature and the second weighted image feature to obtain the background image attention result.
6. The method for training an image generation model according to claim 3, characterized in that: The method of performing a third attention calculation based on the replaced background image attention result, the first background value feature, the first object value feature and the reference image description embedding feature through the downsampling attention network to obtain the sample downsampling result includes: Performing weighted processing on the first background value feature, the first object value feature and the reference image description embedding feature to obtain a sample weighted image feature; The sample weighted image features and the replaced background image attention results are matrix multiplied to obtain the sample downsampling result.
7. The method for training an image generation model according to claim 1, characterized in that: The generating denoising control information based on the first description information, the second description information, and the sample image description information includes: Performing word embedding processing on the first description information to obtain background image description embedding features; Performing word embedding processing on the second description information to obtain sample image description embedding features; Performing word embedding processing on the sample image description information to obtain a reference image description embedding feature; The reference image description embedding feature, the sample image description embedding feature and the background image description embedding feature are integrated into the denoising control information.
8. The method for training an image generation model according to claim 1, characterized in that: The generating background fine-tuning control information based on the sample background image includes: activating a preset fine-tuning network based on the first description information; Performing feature extraction on the sample background image based on the activated fine-tuning network to obtain a first background query feature, a first background key feature, and a first background value feature; The first background query feature, the first background key feature and the first background value feature are integrated into the background fine-tuning control information.
9. The method for training an image generation model according to claim 1, characterized in that: The sample latent space features corresponding to the sample reference image are generated in the following way: Performing encoding processing on the sample reference image to obtain a sample image encoding feature; Based on a preset time step, the sample image encoding features are diffused through the diffusion network of the image generation model to obtain the sample latent space features.
10. The method for training an image generation model according to claim 1, characterized in that: The sample reference image is generated by: Acquire an original reference image, wherein the original reference image is used to indicate an expected result of replacing a reference object in the sample background image with a sample object; Generate a random number that obeys a Gaussian distribution based on a predetermined random number generation model; For each pixel point in the original reference image, the random number is added to the pixel value of the pixel point to obtain the sample reference image.
11. The method for training an image generation model according to claim 1, characterized in that: The step of training the image generation model based on the predicted image features of the plurality of image-text sample pairs and the sample reference image comprises: For each of the image-text sample pairs, obtain reference noise in the sample reference image, and perform noise prediction based on the predicted image features to obtain predicted noise; Calculating a sub-loss function based on a comparison between the reference noise and the predicted noise; Determine a total loss function based on the sub-loss function of each of the image-text sample pairs; Based on the total loss function, the image generation model is trained.
12. The method for training an image generation model according to claim 11, characterized in that: The prediction noise is obtained by prediction of multiple prediction time steps; The calculating a sub-loss function based on the comparison between the reference noise and the predicted noise includes: Performing difference calculation on the reference noise and the predicted noise to obtain a noise difference; A regularization term is calculated on the noise difference to obtain the sub-loss function.
13. The method for training an image generation model according to any one of claims 1 to 12, characterized in that: The sample object image is generated in the following manner: Acquire a sample image of the sample object; Performing image segmentation on the sample image based on a preset object segmentation model to obtain a sample segmented image having the sample object; Perform image enhancement on the sample segmented image to obtain the sample object image.
14. An image generation method, characterized in that: The image generation method comprises: Acquire a target object image of a target object, a target background image, object description information of the target object image, background description information of the target background image, and target image description information, wherein the target background image includes a reference object to be replaced by the target object in the target object image, and the target image description information is used to describe the replacement from the reference object to the target object; Generate denoising prompt information based on the background description information, the object description information, and the target image description information; generating background fine-tuning prompt information based on the target background image, and generating object fine-tuning prompt information based on the target object image; Based on a preset noise image, the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information, image generation is performed through a trained image generation model to obtain a target image, wherein the trained image generation model is generated according to the training method of an image generation model according to any one of claims 1 to 13; and the target image is used to indicate the result of replacing the reference object in the target background image with the target object of the target object image.
15. The image generation method according to claim 14, characterized in that: The trained image generation model includes a diffusion network, a denoising network, and a decoding network; The step of generating an image based on the preset noise image, the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information by using a trained image generation model to obtain a target image includes: Based on the diffusion network, the encoded image features of the preset noise image are diffused to obtain target latent space features; Based on the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information, denoising the target latent space features through the denoising network to obtain a target denoising result; The target denoising result is feature decoded based on the decoding network to obtain the target image.
16. A training device for an image generation model, characterized in that: The training device of the image generation model comprises: A first acquisition unit is used to acquire a plurality of image-text sample pairs, wherein each of the image-text sample pairs includes a sample reference image, a sample background image, a sample object image of a sample object, first description information of the sample background image, second description information of the sample object image, and sample image description information of the sample reference image, wherein the sample reference image is obtained by adding noise to an expected result of replacing a reference object in the sample background image with the sample object; A first generating unit, configured to generate denoising control information based on the first description information, the second description information, and the sample image description information; a second generating unit, configured to generate background fine-tuning control information based on the sample background image, and generate object fine-tuning control information based on the sample object image; a denoising unit, configured to perform denoising processing on the sample latent space features corresponding to the sample reference image through the image generation model based on the background fine-tuning control information, the object fine-tuning control information, and the denoising control information, so as to obtain predicted image features; A training unit is used to train the image generation model based on the predicted image features of the plurality of image-text sample pairs and the sample reference image to obtain a trained image generation model, wherein the trained image generation model is used to generate an image based on a target object image, a target background image, object description information of the target object image, background description information of the target background image, and target image description information.
17. An image generating device, characterized in that: The image generating device comprises: a second acquisition unit, configured to acquire a target object image of a target object, a target background image, object description information of the target object image, background description information of the target background image, and target image description information, wherein the target background image includes a reference object to be replaced by the target object in the target object image, and the target image description information is used to describe the replacement from the reference object to the target object; A third generating unit, configured to generate denoising prompt information based on the background description information, the object description information, and the target image description information; a fourth generating unit, configured to generate background fine-tuning prompt information based on the target background image, and generate object fine-tuning prompt information based on the target object image; An image generation unit is used to generate an image based on a preset noise image, the denoising prompt information, the object fine-tuning prompt information and the background fine-tuning prompt information through a trained image generation model to obtain a target image, wherein the trained image generation model is generated according to the training method of the image generation model described in any one of claims 1 to 13; and the target image is used to indicate the result of replacing the reference object in the target background image with the target object of the target object image.
18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the training method of the image generation model according to any one of claims 1 to 13 or the image generation method according to any one of claims 14 to 15.
19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the training method of the image generation model according to any one of claims 1 to 13 or the image generation method according to any one of claims 14 to 15.
20. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of an electronic device, so that the electronic device executes the training method of the image generation model according to any one of claims 1 to 13 or the image generation method according to any one of claims 14 to 15.
