An object background synthesis method and system based on latent space feature fusion

Abstract

The present application relates to the technical field of image fusion. The present application provides an object background synthesis method based on latent space feature fusion, which comprises the following steps: obtaining a first image and a first text feature vector; processing the first image by using an initial feature extraction network to obtain first image edge features; processing the first image edge features to obtain a first feature vector and a second feature vector; fusing the first feature vector with random noise to obtain a third feature vector; inputting the first text feature vector, the second feature vector and the third feature vector into a U-Net network of an initial consistency model or an initial diffusion model to obtain a target image; comparing the target image with the first image, and updating the initial feature extraction network and the initial consistency model or the initial diffusion model according to a comparison result to obtain a target feature extraction network and a target consistency model or a target diffusion model.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method and system for synthesizing object backgrounds based on latent space feature fusion. Background Technology

[0002] Text-based image generation, text-based image modification, image-based image generation, and image-based local modification techniques have been widely adopted. These techniques are characterized by strong generalization and high-quality generation, and therefore have attracted widespread attention and research.

[0003] While these technologies can achieve image content synthesis and editing, they still have limitations in processing image details. Especially in the field of object background synthesis, existing technologies have two significant drawbacks: first, it is difficult to perceive the original content during large-scale image content completion; second, the completed content often suffers from blurred edges. Summary of the Invention

[0004] This invention provides a method for synthesizing object backgrounds based on latent space feature fusion, which can solve the problem of blurred edges in existing object background synthesis.

[0005] In a first aspect, the present invention provides a method for object background synthesis based on latent space feature fusion, comprising:

[0006] S100, obtain a first image and a first text feature vector, wherein the first image represents an image randomly obtained from the database, and the first text feature vector represents a vector converted from the first text input by the user;

[0007] S200, the first image is processed using an initial feature extraction network to obtain the edge features of the first image;

[0008] S300, the edge features of the first image are processed to obtain a first feature vector and a second feature vector, wherein the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image;

[0009] S400, the first feature vector is fused with random noise to obtain the third feature vector;

[0010] S500, input the first text feature vector, the second feature vector and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image;

[0011] S600, compare the target image with the first image, and update the initial feature extraction network and the initial consistency model or the initial diffusion model according to the comparison result to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0012] In some implementable methods, the step of obtaining the first image and the first text feature vector includes:

[0013] S101, Randomly retrieve an image from the database to obtain the first image;

[0014] S102, Obtain the first text input by the user;

[0015] S103, using the CLIP model, process the first text to obtain the feature vector of the first text.

[0016] In some feasible embodiments, the step of extracting edge features from the first image to obtain the edge features of the first image includes:

[0017] S201, use an edge detection algorithm to perform edge processing on the first image to obtain the edge image of the first image;

[0018] S202, the edge image is extracted using the initial feature extraction network to obtain the edge features of the first image.

[0019] In some feasible embodiments, the step of processing the edge features of the first image to obtain a first feature vector and a second feature vector includes:

[0020] Obtain the latent space, map the edge features of the first image to the latent space, and obtain the first feature vector with the same size as the latent space;

[0021] Based on the first text feature vector, the edge features of the first image are processed to obtain a second feature vector with the same size as the first text feature vector.

[0022] In some feasible embodiments, the step of fusing the first feature vector with random noise to obtain the third feature vector includes:

[0023] S401, Obtain the random noise;

[0024] S402, the first feature vector is fused with the random noise to obtain the third feature vector, wherein the third feature vector represents a feature vector with the same size as the first text feature vector.

[0025] In some feasible implementations, the step of inputting the first text feature vector, the second feature vector, and the third feature vector into a U-Net network of an initial consistency model or an initial diffusion model to obtain the target image includes:

[0026] S501, input the first text feature vector, the second feature vector and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model, and assign weight values ​​to the first text feature vector, the second feature vector and the third feature vector;

[0027] S502, the target feature vector is obtained by processing the first text feature vector, the second feature vector and the third feature vector according to the weight values ​​assigned to them.

[0028] S503, the target image is obtained using the target feature vector.

[0029] In some feasible methods, the step of comparing the target image and the first image, and updating the initial feature extraction network and the initial consistency model or the initial diffusion model based on the comparison result to obtain the target feature extraction network and the target consistency model or the target diffusion model includes:

[0030] S601, using the target image and the first image, calculate the loss function to obtain the difference value;

[0031] S602, according to the preset gradient descent optimization algorithm, update the parameter weights of the initial feature extraction network and the U-Net network of the initial consistency model or the initial diffusion model, and repeat steps S100 to S500 until the preset maximum number of iterations is reached or the loss function converges, to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0032] A second aspect of this application provides an object background synthesis system based on latent space feature fusion, applied to the aforementioned object background synthesis method based on latent space feature fusion, the system comprising:

[0033] The acquisition module is used to acquire a first image and a first text feature vector, wherein the first image represents an image randomly acquired from a database, and the first text feature vector represents a vector converted from the first text input by the user;

[0034] The feature extraction module is used to process the first image using an initial feature extraction network to obtain the edge features of the first image;

[0035] The feature extraction module is also used to process the edge features of the first image to obtain a first feature vector and a second feature vector, wherein the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image;

[0036] The fusion module is used to fuse the first feature vector with random noise to obtain a third feature vector;

[0037] The input module is used to input the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image;

[0038] The result module is used to compare the target image and the first image, and update the initial feature extraction network and the initial consistency model or the initial diffusion model according to the comparison result, so as to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0039] In a third aspect, this application provides a computer-readable storage medium storing computer instructions for causing the computer to perform the aforementioned object background synthesis method based on latent space feature fusion.

[0040] In a fourth aspect, this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the aforementioned object background synthesis method based on latent space feature fusion.

[0041] Beneficial effects of this invention:

[0042] This application discloses an object background synthesis method based on latent space feature fusion. First, a first image and a first text feature vector are obtained. The first image represents an image randomly obtained from a database, and the first text feature vector represents a vector converted from user-input first text. Second, the first image is processed using an initial feature extraction network to obtain edge features. Then, the edge features are processed to obtain a first feature vector and a second feature vector, both representing feature vectors of the first image's edge features. Next, the first feature vector is fused with random noise to obtain a third feature vector. The first text feature vector, the second feature vector, and the third feature vector are input into a U-Net network of an initial consistency model or an initial diffusion model to obtain a target image. Finally, the target image and the first image are compared. Based on the comparison result, the initial feature extraction network and the initial consistency model or the initial diffusion model are updated to obtain a target feature extraction network and a target consistency model or a target diffusion model. By using edge features and random noise to form a feature vector, and fusing this feature vector with the text feature vector and the edge feature vector, the processing capability for local details is enhanced while ensuring that the consistency model or diffusion model perceives global content. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a flowchart of an object background synthesis method based on latent space feature fusion according to the present invention;

[0045] Figure 2 This is a flowchart illustrating the overall process of an object background synthesis method based on latent space feature fusion according to the present invention.

[0046] Figure 3 This is a flowchart illustrating the user steps of an object background synthesis method based on latent space feature fusion according to the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The following explanations of some of the names used in this application are provided to help you understand this application:

[0049] A diffusion model is a type of generative model that simulates the process of matter diffusing from a region of high concentration to a region of low concentration.

[0050] A consistency model is a type of generative model that completes the entire diffusion model in one step under the guidance of ordinary differential equations.

[0051] CLIP, or Contrastive Language-Image Pretraining, is a model used for cross-modal image and text understanding tasks. It achieves semantic alignment between images and text by encoding them into a shared vector space.

[0052] The U-Net model, or Convolutional Networks for Biomedical Image Segmentation model, is a neural network model used for image segmentation. It utilizes "skip connections" to connect the encoder and decoder at the same levels, allowing the network to simultaneously utilize both low-level and high-level feature information. This "skip connection" enables the decoder to leverage higher-level features from the encoder, thereby better recovering the detailed information of the original image.

[0053] Edge detection algorithms are techniques used in digital image processing to identify the edges of objects in an image through analysis and processing. An edge refers to a region in an image where there is a sharp change in grayscale value or color, representing the contour and texture information of an object. The goal of edge detection algorithms is to extract the edges in an image and represent them as a set of discrete pixels or curves.

[0054] Feature extraction networks (MEGs) are a type of neural network in deep learning, primarily used to extract important features from input images or data for subsequent tasks such as classification, detection, and segmentation. MEGs typically consist of multiple convolutional and pooling layers. By performing multiple convolution and pooling operations on the input, they gradually reduce the spatial size and extract increasingly abstract, high-level features. MEGs contain several parameters, which typically include the weights and biases of the convolutional layers.

[0055] With the development of deep generative models based on consistency and diffusion models, techniques such as text-to-image generation, text-to-image modification, image-to-image generation, and local image modification have been widely adopted. These techniques are characterized by strong generalization and high-quality generation, but they also have drawbacks, especially in handling image details, particularly in the field of object-background synthesis. Specifically, they suffer from two main problems: firstly, difficulty in perceiving the original content during large-scale image completion; and secondly, often resulting in blurred edges after completion.

[0056] For example, in scenarios such as e-commerce product image generation and photo background replacement, after taking a picture of the product, when it is necessary to generate or replace the background of the product, the resulting image quality is often poor because the edges of the product are blurred.

[0057] In view of this, this application proposes an object background synthesis method based on latent space feature fusion. Within the framework of existing consistency and diffusion models, image content features and image edge features are aligned into the latent space, enabling the consistency and diffusion models to perceive the overall image content while accurately preserving key edge information, thereby generating higher quality object background images.

[0058] like Figure 1 and Figure 2 As shown, the first aspect of this application provides a method for object background synthesis based on latent space feature fusion, comprising:

[0059] S100, Obtain the first image and the first text feature vector.

[0060] Here, the first image represents an image randomly obtained from the database, and the first text feature vector represents a vector transformed from the first text input by the user.

[0061] Specifically, obtaining the feature vectors of the first image and the first text includes steps S101 to S103.

[0062] S101, randomly retrieve an image from the database to obtain the first image.

[0063] S102, Obtain the first text input by the user.

[0064] S103. Using the CLIP model, the first text is processed to obtain the feature vector of the first text.

[0065] This process can involve pre-establishing a database, or creating a database that stores several images. When an image is needed, it is randomly selected from the database. To ensure the final generated image meets the user's expectations, or is aligned with the user's input, the user needs to input text, and the image is generated based on this text.

[0066] After obtaining the image and the first text, the CLIP model is used to process the first text, thereby obtaining the first text feature vector corresponding to the first text.

[0067] S200: The first image is processed using the initial feature extraction network to obtain the edge features of the first image.

[0068] In the aforementioned steps, a first image is obtained. This image is random and is used for object background synthesis based on latent space feature fusion. Before training, feature extraction is performed on the first image to obtain its edge features.

[0069] Specifically, obtaining the first image edge features includes steps S201 and S202.

[0070] S201, use an edge detection algorithm to perform edge processing on the first image to obtain the edge image of the first image.

[0071] S202, use the initial feature extraction network to extract features from the edge image to obtain the first image edge features.

[0072] To obtain the edge features of the first image, the edges of the first image must first be processed. Specifically, an edge detection algorithm can be used to process the edges of the first image, thereby obtaining the edge image of the first image. Next, feature extraction is performed on the edge image. Feature extraction can be implemented using a feature extraction network. That is, the edge image is taken as an RGB image and input into the initial feature extraction network. After the initial feature extraction network processes the image, the edge features of the first image are obtained.

[0073] It should be noted that other edge detection algorithms can be selected as needed, such as the Canny operator for extracting image edges. This application does not limit this selection; any algorithm that can process the image to obtain the edge image is acceptable.

[0074] S300, process the edge features of the first image to obtain the first feature vector and the second feature vector.

[0075] Wherein, the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image.

[0076] Specifically, obtaining the first feature vector and the second feature vector includes steps S301 and S302.

[0077] S301, Obtain the latent space, map the edge features of the first image to the latent space, and obtain a first feature vector with the same size as the latent space.

[0078] S302, based on the first text feature vector, process the edge features of the first image to obtain a second feature vector with the same size as the first text feature vector.

[0079] Wherein, the first feature vector represents a feature vector with the same size as the latent space, that is, the first feature vector is formed according to the size of the latent space, and the first image edge features mapped to the latent space are aligned with the features of the first image; the second feature vector represents a feature vector with the same size as the first text feature vector, that is, the second feature vector is formed according to the first text feature vector.

[0080] Setting the magnitudes of the first and second feature vectors in this way facilitates better integration of edge features into the image generation process, thereby improving processing speed. In other words, after extracting the edge features of the first image using the initial feature extraction network, the edge features of the first image are transformed to obtain a first feature vector and a second feature vector, both of which can represent the edge features of the first image.

[0081] S400, the first feature vector is fused with random noise to obtain the third feature vector.

[0082] By incorporating random noise into the first feature vector, the resulting output becomes more diverse, avoiding overfitting of the U-Net network in the consistency or diffusion models, which would otherwise produce a single output. This approach also allows for a wider variety of output results.

[0083] Specifically, obtaining the third feature vector includes steps S401 and S402.

[0084] S401, obtain random noise.

[0085] S402, the first feature vector is fused with random noise to obtain the third feature vector.

[0086] The third feature vector represents a feature vector with the same size as the first text feature vector. Random noise can be implemented using a Gaussian distribution. This application does not limit how random noise is generated.

[0087] Specifically, the method used to fuse the first feature vector and the random noise can be set as needed, and this application does not limit the fusion method. For example, the first feature vector and the random noise can be fused using a weighted method, that is, the first feature vector and the random noise are linearly combined according to certain weights to obtain a fused vector, namely the third feature vector. The weights can be determined according to requirements.

[0088] S500 inputs the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image.

[0089] In this process, after obtaining the first text feature vector, the second feature vector, and the third feature vector, the first text feature vector, the second feature vector, and the third feature vector are respectively input into the U-Net network of the initial consistency model or the initial diffusion model, and processed using the initial consistency model or the initial diffusion model U-Net network to obtain the target image.

[0090] Specifically, obtaining the target image includes steps S501 to S503.

[0091] S501, input the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model, and assign weight values ​​to the first text feature vector, the second feature vector, and the third feature vector.

[0092] S502, the target feature vector is obtained by processing the first text feature vector, the second feature vector, and the third feature vector according to the weight values ​​assigned to them.

[0093] S503 uses the target feature vector to obtain the target image.

[0094] The three feature vectors—the first text feature vector, the second feature vector, and the third feature vector—can be assigned weight values ​​either after being input into the U-Net network of the initial consistency model or the initial diffusion model, or before being input into the U-Net network of the initial consistency model or the initial diffusion model. The weight values ​​for the first text feature vector, the second feature vector, and the third feature vector can be set as needed; this application does not impose any limitations on this.

[0095] After assigning weights to the first, second, and third text feature vectors, the feature vectors are fused using a vector fusion method. This application does not limit the fusion method; for example, weighted fusion can be used. Multiple vectors are linearly combined according to certain weights to obtain a new fused vector. This new vector is then processed in a U-Net network with an initial consistency model or an initial diffusion model to obtain the target image.

[0096] S600 compares the target image with the first image, and updates the initial feature extraction network and the initial consistency model or the initial diffusion model based on the comparison results, to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0097] The process of obtaining the target feature extraction network and the target consistency model or target diffusion model includes steps S601 and S602.

[0098] S601, using the target image and the first image, calculate the loss function to obtain the difference value.

[0099] S602, according to the preset gradient descent optimization algorithm, update the parameter weights of the initial feature extraction network and the initial consistency model or the initial diffusion model, and repeat steps S100 to S500 until the preset maximum number of iterations is reached or the loss function converges, to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0100] The target image obtained in the aforementioned steps is an image generated by the U-Net network of the diffusion model based on the feature vectors of the first image and the first text. Next, the target image and the first image need to be compared for difference. For example, the difference comparison can use the squared loss error method to measure the difference between the two. The squared loss error method maps all difference values ​​to a positive range, so that larger difference values ​​are amplified in the calculation, so as to more sensitively reflect the degree of difference between the target image and the first image.

[0101] Next, based on the degree of difference, the parameter weights of the initial feature extraction network and the initial consistency model or initial diffusion model U-Net network are adjusted using a preset gradient descent optimization algorithm. That is, after obtaining the degree of difference, the parameter weights of the initial feature extraction network and the initial consistency model or initial diffusion model U-Net network are adjusted according to the preset gradient descent step size. Once the adjustment is complete, one step in the training process is finished. Next, steps S100 to S500 are repeated. Then, based on the next degree of difference obtained in the initial feature extraction network and the initial consistency model or initial diffusion model U-Net network after the adjustment step size, the initial feature extraction network and the initial consistency model or initial diffusion model U-Net network are adjusted again with a preset step size. The adjustment can end based on the preset maximum number of iterations or the convergence of the loss function value. Repeating the above process several times completes one training cycle. Finally, the target feature extraction network and the target consistency model or target diffusion model are obtained.

[0102] like Figure 3 As shown, for example, after completing the above training process, the user can perform image synthesis. Specifically:

[0103] Step 1: The user inputs the second text and a product image, where the product image is a product image without a background, also known as an object image.

[0104] Step 2: Extract the second text in the CLIP model, generate the second text feature vector, and use the image encoder to extract image features from the product image and map the image features into the latent space; the edge detection algorithm processes the edges of the product image to obtain the edge image of the second image, and uses the image encoder to extract features from the edge image of the second image to obtain the edge features of the second image, where the image encoder represents the target feature extraction network.

[0105] Step 3: Obtain the first feature vector and the second feature vector of the edge features of the second image.

[0106] Step 4: Fuse the first feature vector and the image features mapped to the latent space to obtain the feature fusion.

[0107] Step 5: Based on feature fusion, add random noise to the non-object parts. This random noise is generated under the control of the second text feature vector.

[0108] Step 6: Input the second text feature vector, random noise added to the non-object part, and the second feature vector into the target consistency model or target diffusion model.

[0109] Step 7: Obtain the generated image.

[0110] It should be noted that the random noise controlled by the second text is used to generate the background of the second image. Combining this background with the second image yields a product image with a background. Furthermore, the first and second feature vectors here are generated based on the image edges of the product image, not the image edges of the first image. For ease of understanding, the same names are used here. After this optimization, the target consistency model or target diffusion model can perceive the content of the object and ensure that the object edges are correctly preserved.

[0111] In summary, the object background synthesis method based on latent space feature fusion provided in this application is applicable to scenarios such as e-commerce product image generation and photo background replacement. By training an additional feature extraction network, the consistency model or diffusion model can perceive the object content and edges, ensuring the quality and rationality of the generated results, while also avoiding the repeated training of large-scale pre-trained models.

[0112] This application provides a method for object background synthesis based on latent space feature fusion. Through the design and fusion of edge detection algorithms, it enhances the ability to process local details while ensuring the consistency of the model's perception of global content. It yields better results in applications such as product image generation and image editing, and has the following innovative features:

[0113] 1. A method is proposed to enhance the consistency model or diffusion model framework by jointly using image features and edge features. This method leverages image features, edge features, and feature fusion to improve the ability of the consistency model or diffusion model to process image details.

[0114] 2. Design an edge feature network (edge ​​detection and feature extraction network) to make edge features effective features that can be used for image reconstruction, providing support for subsequent feature fusion.

[0115] 3. Multi-source fusion of product image content features and edge features. Aligning and fusing content features and edge features in the latent space to complement each other and enhance the expressive power of consistency or diffusion models is key to achieving detail generation.

[0116] 4. Based on edge features, the consistency model or diffusion model can ensure global content consistency while reproducing local details more accurately and precisely, significantly improving the quality of detail generation and the effect of generated images.

[0117] A second aspect of this application provides an object background synthesis system based on latent space feature fusion, applied to the aforementioned object background synthesis method based on latent space feature fusion, the system comprising:

[0118] The acquisition module is used to acquire a first image and a first text feature vector, wherein the first image represents an image randomly acquired from the database, and the first text feature vector represents a vector converted from the first text input by the user;

[0119] The feature extraction module is used to process the first image using the initial feature extraction network to obtain the edge features of the first image;

[0120] The feature extraction module is also used to process the edge features of the first image to obtain a first feature vector and a second feature vector, wherein the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image;

[0121] The fusion module is used to fuse the first feature vector with random noise to obtain the third feature vector;

[0122] The input module is used to input the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image;

[0123] The results module is used to compare the target image with the first image, and update the initial feature extraction network and the initial consistency model or the initial diffusion model based on the comparison results, so as to obtain the target feature extraction network and the target consistency model or the target diffusion model.

[0124] In a third aspect, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the aforementioned object background synthesis method based on latent space feature fusion.

[0125] In a fourth aspect, this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the aforementioned object background synthesis method based on latent space feature fusion.

[0126] In the description of the embodiments of the present invention, those skilled in the art should understand that the embodiments of the present invention can be implemented as methods, apparatuses, electronic devices, and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. Furthermore, in some embodiments, the embodiments of the present invention can also be implemented as a computer program product in one or more computer-readable storage media, the computer-readable storage media containing computer program code.

[0127] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable storage media. Computer-readable storage media include: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination thereof. In embodiments of the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The computer program code contained in the aforementioned computer-readable storage medium may be transmitted using any suitable medium, including wireless, wire, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0129] Computer program code for performing the operations of the embodiments of the present invention can be written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer or an external computer via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0130] The embodiments of the present invention describe the provided methods, apparatus, and electronic devices through flowcharts and / or block diagrams.

[0131] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by a computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0132] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to function in a particular manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction apparatus product that includes the functions / operations specified in the blocks of a flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable data processing apparatus provide a process for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0134] In the specification and claims of this invention, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0135] In embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0136] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for object background synthesis based on latent space feature fusion, characterized in that, The methods include: S100, obtain a first image and a first text feature vector, wherein the first image represents an image randomly obtained from the database, and the first text feature vector represents a vector converted from the first text input by the user; S200, the first image is processed using an initial feature extraction network to obtain the edge features of the first image; S300, the edge features of the first image are processed to obtain a first feature vector and a second feature vector, wherein the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image; S400, the first feature vector is fused with random noise to obtain the third feature vector; S500, input the first text feature vector, the second feature vector and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image; S600, the target image and the first image are compared, and the initial feature extraction network and the initial consistency model or the initial diffusion model are updated according to the comparison result to obtain the target feature extraction network and the target consistency model or the target diffusion model; wherein, The step of processing the edge features of the first image to obtain the first feature vector and the second feature vector includes: Obtain the latent space, map the edge features of the first image to the latent space, and obtain the first feature vector with the same size as the latent space; Based on the first text feature vector, the edge features of the first image are processed to obtain a second feature vector with the same size as the first text feature vector.

2. The object background synthesis method based on latent space feature fusion according to claim 1, characterized in that, The steps for obtaining the feature vectors of the first image and the first text include: S101, Randomly retrieve an image from the database to obtain the first image; S102, Obtain the first text input by the user; S103, using the CLIP model, process the first text to obtain the feature vector of the first text.

3. The object background synthesis method based on latent space feature fusion according to claim 1, characterized in that, The step of extracting edge features from the first image to obtain the edge features of the first image includes: S201, use an edge detection algorithm to perform edge processing on the first image to obtain the edge image of the first image; S202, the edge image is extracted using the initial feature extraction network to obtain the edge features of the first image.

4. The object background synthesis method based on latent space feature fusion according to claim 1, characterized in that, The step of fusing the first feature vector with random noise to obtain the third feature vector includes: S401, Obtain the random noise; S402, the first feature vector is fused with the random noise to obtain the third feature vector, wherein the third feature vector represents a feature vector with the same size as the first text feature vector.

5. The object background synthesis method based on latent space feature fusion according to claim 4, characterized in that, The step of inputting the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image includes: S501, input the first text feature vector, the second feature vector and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model, and assign weight values ​​to the first text feature vector, the second feature vector and the third feature vector; S502, the target feature vector is obtained by processing the first text feature vector, the second feature vector and the third feature vector according to the weight values ​​assigned to them; S503, the target image is obtained using the target feature vector.

6. The object background synthesis method based on latent space feature fusion according to claim 1, characterized in that, The step of comparing the target image and the first image, and updating the initial feature extraction network and the initial consistency model or the initial diffusion model based on the comparison result to obtain the target feature extraction network and the target consistency model or the target diffusion model includes: S601, using the target image and the first image, calculate the loss function to obtain the difference value; S602, according to the preset gradient descent optimization algorithm, update the parameter weights of the initial feature extraction network and the U-Net network of the initial consistency model or the initial diffusion model, and repeat steps S100 to S500 until the preset maximum number of iterations is reached or the loss function converges, to obtain the target feature extraction network and the target consistency model or the target diffusion model.

7. An object background synthesis system based on latent space feature fusion, characterized in that, The system, which is applied to the object background synthesis method based on latent space feature fusion according to any one of claims 1-6, comprises: The acquisition module is used to acquire a first image and a first text feature vector, wherein the first image represents an image randomly acquired from a database, and the first text feature vector represents a vector converted from the first text input by the user; The feature extraction module is used to process the first image using an initial feature extraction network to obtain the edge features of the first image; The feature extraction module is also used to process the edge features of the first image to obtain a first feature vector and a second feature vector, wherein the first feature vector and the second feature vector both represent feature vectors of the edge features of the first image; The fusion module is used to fuse the first feature vector with random noise to obtain a third feature vector; The input module is used to input the first text feature vector, the second feature vector, and the third feature vector into the U-Net network of the initial consistency model or the initial diffusion model to obtain the target image; The result module is used to compare the target image and the first image, and update the initial feature extraction network and the initial consistency model or the initial diffusion model according to the comparison result, so as to obtain the target feature extraction network and the target consistency model or the target diffusion model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the object background synthesis method based on latent space feature fusion as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: At least one processor; The at least one processor is also connected in communication with a memory, wherein the memory stores a computer program that can be executed by the at least one processor to cause the at least one processor to perform the object background synthesis method based on latent space feature fusion as described in any one of claims 1-6.

Images