Image generation method and device, and electronic device

Abstract

The embodiment of the present application provides an image generation method and its device, and electronic device, the method comprising: masking the background area of ​​the original image to obtain a first image, the first image comprising the foreground area of ​​the original image and the masked background area; obtaining a first latent vector of the first image; obtaining multiple control information of the foreground area according to the first latent vector; obtaining a second latent vector according to the first latent vector, the multiple control information and the text vector corresponding to the preset background description; decoding the second latent vector to obtain a second image. The present application can automatically replace the background and keep the foreground consistent according to the original image and background description input by the user, thereby achieving natural, rich and efficient background generation.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, relates to image processing technology, and in particular to an image generation method and its device and electronic equipment. Background Art

[0002] The related technologies that use image processing technology to add backgrounds to product images still have significant limitations and shortcomings. For example, the related technologies usually require users to use image editing tools to manually synthesize pre-stored background images and product images. The synthesis effect is unnatural and the degree of integration is low, which affects consumers' willingness to buy. In addition, there are significant limitations in the way background images are obtained in the related technologies. For example, the related technologies usually rely on existing image resources or manually designed background images. The diversity and personalized expression of the image resources obtained in this way are poor, and the production cost and time cost are also high. Summary of the invention

[0003] The embodiments of the present application provide an image generation method and device, and an electronic device, which can solve the existing problem of difficulty in achieving natural, rich and efficient processing of the background of product images.

[0004] A first aspect of an embodiment of the present application provides an image generation method, including: masking a background area of ​​an original image to obtain a first image, the first image including a foreground area of ​​the original image and a masked background area; obtaining a first latent vector of the first image; obtaining multiple control information of the foreground area based on the first latent vector; obtaining a second latent vector based on the first latent vector, the multiple control information and a text vector corresponding to a preset background description; and decoding the second latent vector to obtain a second image.

[0005] According to an embodiment of the present application, obtaining multiple control information of the foreground area based on the first latent vector includes: downsampling the first latent vector through multiple first downsampling modules of the control model to obtain multiple first downsampling results of the first latent vector; and obtaining the multiple control information based on the control information corresponding to each first downsampling result.

[0006] According to an embodiment of the present application, for the j-th first downsampling module among the multiple first downsampling modules, the downsampling processing of the first latent vector includes: if j is equal to 1, the downsampling processing is performed on the first latent vector through the first first downsampling module of the control model to obtain the first first downsampling result; if j is an integer greater than 1, the j-1-th first downsampling result is downsampled through the j-th first downsampling module of the control model to obtain the j-th first downsampling result.

[0007] According to an embodiment of the present application, the second latent vector is obtained according to the first latent vector, the multiple control information and the text vector corresponding to the preset background description, including: downsampling the first latent vector through a second downsampling module of a diffusion model to obtain a second downsampling result of the first latent vector; fusing the second downsampling result with the text vector to obtain a fused vector; extracting features of the fused vector through an intermediate layer of the diffusion model to obtain intermediate features; and obtaining the second latent vector through an upsampling module of the diffusion model based on the intermediate features and the multiple control information.

[0008] According to an embodiment of the present application, the diffusion model includes N second downsampling modules, where N represents an integer greater than or equal to 2, and the downsampling processing of the first latent vector by the second downsampling module of the diffusion model to obtain a second downsampling result of the first latent vector includes: performing N downsampling processing on the first latent vector through the N second downsampling modules; and obtaining the second downsampling result according to the result of the Nth downsampling processing.

[0009] According to an embodiment of the present application, the control information includes N control information, where N represents an integer greater than or equal to 2; the obtaining the second latent vector according to the intermediate feature and the multiple control information through the upsampling module of the diffusion model includes: determining the weighted sum f(0) of the intermediate feature and the Nth control information in the control information, and upsampling the weighted sum f(0) through the first upsampling module of the diffusion model to obtain an upsampling result F(1); determining the upsampling result F(k-1) and the weighted sum f(k-1) of the N-(k-1)th control information in the control information, and upsampling the weighted sum f(k-1) through the kth upsampling module of the diffusion model to obtain an upsampling result F(k); wherein k represents an integer greater than or equal to 2 and less than or equal to N; determining the second latent vector according to the upsampling result F(N).

[0010] According to an embodiment of the present application, the fusing of the second downsampling result and the text vector to obtain a fused vector includes: adjusting the dimension of the text vector according to the spatial dimension of the second downsampling result; and fusing the dimensionally adjusted text vector with the second downsampling result to obtain the fused vector.

[0011] According to an embodiment of the present application, the training method of the diffusion model includes: obtaining training data, the training data including a sample image, a third image obtained by masking the background area of ​​the sample image, and background description text of the sample image; generating a fourth image through the diffusion model based on the third image and the background description text; determining a loss value based on the fourth image and the sample image, and training the diffusion model based on the loss value.

[0012] The second aspect of the embodiment of the present application provides an image generation device, the device comprising: a mask module, used to mask the background area of ​​the original image to obtain a first image, the first image comprising the foreground area of ​​the original image and the masked background area; an encoding module, used to obtain a first latent vector of the first image. An acquisition module, used to obtain multiple control information of the foreground area according to the first latent vector; a fusion module, used to obtain a second latent vector according to the first latent vector, multiple control information and a text vector corresponding to a preset background description; a decoding module, used to decode the second latent vector to obtain a second image.

[0013] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, and a processor, wherein the processor executes computer-readable instructions stored in the memory to implement the image generation method.

[0014] The image generation method provided in the embodiment of the present application can update the background image in the original image according to the original image input by the user and the preset background description to obtain a second image. In addition, the foreground image in the second image is consistent with the foreground image in the original image, the background image in the second image corresponds to the preset background description input by the user, and the style features of the background area and the foreground area of ​​the second image are consistent. In this way, the background can be automatically replaced and the foreground can be kept consistent according to the original image and background description input by the user, which significantly improves the fusion degree of the background and foreground and the overall effect of the picture, and realizes natural, rich and efficient background processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A schematic diagram of an application environment of an image generation method provided in an embodiment of the present application.

[0017] Figure 2 A flowchart of an image generation method provided in an embodiment of the present application.

[0018] Figure 3 An example diagram of the mask process provided in an embodiment of the present application.

[0019] Figure 4 An example diagram of the model structure provided in the embodiments of the present application.

[0020] Figure 5 A schematic diagram of a flow chart of a method for obtaining a second latent vector provided in an embodiment of the present application.

[0021] Figure 6 A specific flow chart of step S504 provided for an embodiment of the present application.

[0022] Figure 7 A flowchart of a method for training a diffusion model provided in an embodiment of the present application.

[0023] Figure 8 This is an example diagram of the image generation process provided in an embodiment of the present application.

[0024] Fig. 9 A functional block diagram of an image generating device provided in an embodiment of the present application.

[0025] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0029] See also Figure 1 , is a schematic diagram of an application environment of an image generation method provided in an embodiment of the present application. Figure 1 As shown, the user terminal 10 communicates with the server 20 through a network. The network may be a wired network communication or a wireless network communication. The wired network may be any one of a local area network, a metropolitan area network and a wide area network, a wireless fidelity (Wireless Fidelity, Wi-Fi), a self-organizing network wireless communication (ZigBee Wireless Networks, ZigBee) technology, an ultra-wideband (Ultra Wideband, UWB) technology, a wireless universal serial bus (Universal Serial Bus, USB) and the like.

[0030] The user terminal 10 may be an electronic device such as a mobile phone, a tablet computer, a multimedia player, a personal computer (PC), a wearable device, etc. The user terminal 10 may be a client with various applications installed, such as education applications, consulting applications, information broadcast applications, live broadcast applications, e-commerce applications, etc.

[0031] The server 20 is used to provide background services for the application in the user terminal 10. For example, the server 20 can be the background server of the above-mentioned e-commerce application. The server 20 can be a computer device. In one embodiment of the present application, it can be a server, a server cluster composed of multiple servers, or a cloud computing service center.

[0032] Exemplarily, an e-commerce application is used as an example for explanation. The user performs the operation of inputting the original image and the preset background description in the client (for example, the user terminal 10) on which the e-commerce application is installed, and the user terminal 10 sends the original image and the preset background description to the server 20. The server 20 masks the background area of ​​the original image to obtain a first image, and obtains a first latent vector of the first image. The server 20 obtains multiple control information of the foreground area based on the first latent vector. The server 20 fuses the first latent vector, multiple control information and the text vector corresponding to the preset background description to obtain a second latent vector. The server 20 decodes the second latent vector to obtain a second image. The server 20 sends the second image to the user terminal 10, and the user terminal 10 presents the second image to the user using a graphical user interface.

[0033] In this way, the server 20 can update the background image in the original image according to the original image and the preset background description input by the user to obtain a second image. Moreover, the foreground image in the second image is consistent with the foreground image in the original image, and the background image in the second image corresponds to the preset background description input by the user. In this way, the background can be automatically replaced and the foreground can be kept consistent according to the original image and the background description input by the user, which significantly improves the fusion degree of the background and the foreground and the overall effect of the picture, and realizes natural, rich and efficient background processing.

[0034] The following will be a computer program product running on a server (such as Figure 1 The server 20 shown in FIG. Figure 2 FIG. 1 is a flow chart of an image generation method provided in an embodiment of the present application. In an embodiment of the present application, the method includes the following steps: Step S201, masking the background area of ​​the original image to obtain a first image.

[0035] In some embodiments of the present application, the original image includes any image determined by the user that the user wants to update the background of. For example, the original image may be a product image. If the product is clothing or apparel, the original product may also be an image of a model wearing the product. In one example, the original image includes a foreground area and a background area. For example, if the original image is an image of a model wearing the product, the foreground area in the original image represents the area where the model is located, and the background area represents the area outside the area where the model is located. For example Figure 3 As shown, the foreground area in the original image is the area where the little boy is located, and the background area is the area outside the little boy.

[0036] In some embodiments of the present application, the background area of ​​the original image may be masked to obtain a first image, wherein the first image includes the foreground area of ​​the original image and the masked background area.

[0037] In one example, masking the background area of ​​the original image to obtain the first image may include: performing foreground area detection on the original image, for example, using a semantic segmentation algorithm to detect the foreground area of ​​the original image. Constructing a mask image based on the detected foreground area, for example, a mask image having the same size as the original image, a transparent foreground area, and a background area outside the foreground area having a preset color may be constructed. Figure 3 As shown, the mask image has the same size as the original image. In the mask image, according to the range corresponding to the foreground area of ​​the original image, such as the range where the little boy is located, the range is set as a transparent area; the rest of the background area is set to black. The original image is masked based on the mask image to obtain the first image. For example Figure 3As shown, the original image is masked by the mask image to obtain a first image, the first image has the same size as the original image, the first image includes a foreground area consistent with the original image, and the background area in the first image is black.

[0038] Based on the above embodiment, the foreground area of ​​the original image can be extracted so that the first image contains the foreground area consistent with the original image. By generating the background of the first image, the user's desired image with the same foreground as the original image and a different background can be obtained.

[0039] Step S202, obtaining a first latent vector of the first image.

[0040] In some embodiments of the present application, a first latent vector of the first image may be obtained, and the first latent vector includes a potential feature representation of the foreground area. For example, the first latent vector may include style features of the foreground area, including but not limited to shape, color, texture, brightness, etc. The first latent vector may also be referred to as a first feature map, a first feature vector, etc. In this way, the first latent vector may be used to guide the subsequent background generation of the first image, ensuring that the newly generated background is consistent with the style features of the foreground area in the first image.

[0041] In some embodiments of the present application, an encoder may be used to obtain a first latent vector of the first image. In one example, an encoder (Encoder) based on a pre-trained variational autoencoder (VAE) may be used to map the first image to a low-dimensional latent space to obtain the first latent vector.

[0042] In one example, VAE combines autoencoders and probabilistic coding techniques to achieve a more flexible and controllable sample generation process by modeling the latent representation. Unlike traditional autoencoders, VAE assumes that the latent representation obeys a prior distribution (such as a Gaussian distribution) and maps the input data to the distribution parameters of the latent space through the encoder. The decoder maps the sampled results in the latent space back to the original input space to generate new samples.

[0043] The training process of VAE can be achieved by minimizing the reconstruction error and KL divergence (Kullback Leibler Divergence). The reconstruction error is used to measure the difference between the generated sample and the original input sample, and the KL divergence is used to measure the difference between the potential distribution and the prior distribution (such as the standard normal distribution). By minimizing the reconstruction error and KL divergence, VAE can learn the main features of the input data and generate new samples similar to the input data.

[0044] In some embodiments of the present application, an encoder of a VAE may be used to map input data to a function of a latent space, and a first latent vector of a first image may be obtained through the encoder of the VAE.

[0045] In one example, obtaining a first latent vector of a first image through an encoder of a VAE may include: normalizing the first image, for example, scaling the pixel values ​​of the first image to the interval [0, 1] or [-1, 1] to ensure that the pixel values ​​of the input data of the VAE are within a suitable numerical range. Resizing the first image after the normalization process, for example, cropping or scaling the first image after the normalization process according to the size requirement of the input data of the VAE. Inputting the resized first image into the VAE, and obtaining the first latent vector of the first image using the encoder of the VAE. For example, the encoder may include a multi-layer convolutional neural network, which is used to extract a feature map of the input image, and the size of the feature map gradually decreases as the network layer goes deeper. In this way, the encoder maps the first image to the distribution parameters of the latent space, and outputs a mean vector (μ) and a standard deviation vector (σ). The mean vector and the standard deviation vector together define a Gaussian distribution in the latent space, and the first latent vector can be obtained by sampling from the Gaussian distribution. For example, based on the reparameterization technique, a random vector ε is sampled from a standard normal distribution, and the first latent vector z is obtained by the formula z=μ+ε×σ.

[0046] Based on the above embodiment, the first latent vector contains the potential feature representation of the foreground area of ​​the original image. These latent feature representations can be used to guide the subsequent background generation process, thereby ensuring that the newly generated background is consistent with the foreground features in the first latent vector, thereby achieving a harmonious unity of the foreground and background of the generated image in terms of style features.

[0047] Step S203: Acquire multiple control information of the foreground area according to the first latent vector.

[0048] In some embodiments of the present application, the first latent vector may be downsampled (subsampled) by multiple first downsampling modules of the control model (Control Net) to obtain multiple first downsampling results of the first latent vector. Multiple control information is obtained according to the control information corresponding to each first downsampling result.

[0049] In one example, for the jth first downsampling module among multiple first downsampling modules, downsampling processing of the first latent vector includes: if j is equal to 1, downsampling processing is performed on the first latent vector by controlling the first first downsampling module of the model to obtain the first first downsampling result. If j is an integer greater than 1, downsampling processing is performed on the j-1th first downsampling result by controlling the jth first downsampling module of the model to obtain the jth first downsampling result. Wherein, j represents an integer greater than or equal to 1 and less than or equal to N, and N represents the total number of first downsampling modules of the control model.

[0050] In some embodiments of the present application, the first downsampling module may include but is not limited to a convolution layer, and the downsampling process may include but is not limited to a convolution operation performed using a convolution layer. The size, number, step size, padding, and activation function of the convolution kernel in the convolution layer are adjusted according to the task requirements to keep the output resolution lower than the input, and the number of input and output channels unchanged or appropriately increased. In another example, the first downsampling module may also include a pooling layer, and the downsampling process may also include a pooling operation performed using a pooling layer, which is not specifically limited in the present application.

[0051] For example Figure 4 As shown, the control model includes N=4 first downsampling modules. The spatial dimension of the first latent vector latent_i can be 64×64×320, where the resolution (or size) is 64×64 and the number of channels is 320.

[0052] The first latent vector latent_i passes through the first first downsampling module CNS1, and the spatial dimension of the first downsampling result latent_i2x can be 32×32×512. Compared with latent_i, the resolution of latent_i2x is reduced by half, and the number of channels is increased.

[0053] The first down-sampling result latent_i2x passes through the second first down-sampling module CNS2, and the spatial dimension of the obtained first down-sampling result latent_i4x may be 16×16×512. Compared with latent_i2x, the resolution of latent_i4x is reduced by half, and the number of channels remains unchanged.

[0054] The spatial dimension of the first down-sampling result latent_i4x obtained by the third first down-sampling module CNS3 may be 8×8×1024. Compared with latent_i4x, the resolution of latent_i8x is reduced by half, and the number of channels is increased.

[0055] The first down-sampling result latent_i8x passes through the fourth first down-sampling module CNS4, and the spatial dimension of the obtained first down-sampling result latent_i16x may be 4×4×1024. Compared with latent_i8x, the resolution of latent_i16x is reduced by half, and the number of channels remains unchanged.

[0056] Through the above method, multiple first downsampling results can be obtained, the resolution of each first downsampling result is smaller than the resolution of the previous first downsampling result, and the number of channels of each first downsampling result is greater than or equal to the number of channels of the previous first downsampling result. Taking each first downsampling result as a control information, multiple control information of different spatial dimensions can be obtained. Each control information represents a feature representation of a corresponding spatial dimension of the foreground area, so that the details and hierarchical structure of the foreground area can be captured based on the control information of multiple different spatial dimensions.

[0057] Based on the above embodiment, multiple control information of the foreground area can be obtained according to the control information corresponding to each first downsampling result. These control information can be used to guide the subsequent image generation process to ensure that the foreground area of ​​the generated image has the same style features and detail levels as the original image.

[0058] Step S204: obtaining a second latent vector according to the first latent vector, the plurality of control information and the text vector corresponding to the preset background description.

[0059] In some embodiments of the present application, a preset background description input by a user may be received. For example, the preset background description may be a description text constructed by the user based on the foreground area of ​​the original image, including a description text of the background required by the user. Figure 3 As shown in FIG. 1 , the foreground area is a smiling boy, and the corresponding preset background description may be “A boy on a cobblestone street, smiling under the bright sunlight”. Among them, texts such as “on a cobblestone street” and “under the bright sunlight” represent description texts of the background required by the user.

[0060] In some embodiments of the present application, a text encoder may be used to encode the preset background description, thereby converting the user's description of the background into a machine-understandable format to obtain a text vector. The text vector contains key information of the user's background, such as scene, environment, atmosphere, etc.

[0061] In one example, the text vector may include one or more text vectors. If the text vector includes multiple text vectors, the dimensions of each text vector may be different. For example, the text vector may include a first text vector with a dimension of 32×512, a second text vector with a dimension of 16×512, a third text vector with a dimension of 8×1024, and a fourth text vector with a dimension of 4×1024.

[0062] In some embodiments of the present application, the first latent vector, the text vectors corresponding to the plurality of control information and the preset background description may be fused to obtain the second latent vector. In one example, the fusion of the first latent vector, the text vectors corresponding to the plurality of control information and the preset background description may be achieved by channel fusion, and the obtained second latent vector is a representation vector of a complete image that includes both foreground features and meets the user's background requirements. In another example, the method for obtaining the second latent vector may refer to Figure 5 Flowchart shown.

[0063] Based on the above embodiment, the second latent vector obtained by fusion is a comprehensive representation, which not only includes the characteristics of the image foreground, but also fully meets the user's description and overall requirements for the background, thereby generating a complete image representation vector that has rich foreground details and meets the user's personalized background requirements. In this way, it is possible to flexibly generate expected image content according to the user's instructions and descriptions.

[0064] Step S205: decode the second latent vector to obtain a second image.

[0065] In some embodiments of the present application, a preset decoder may be used to decode the second latent vector. In one example, referring to the description of VAE in step S202, the decoder of VAE may be used to map the second latent vector back to the original input space to obtain a second image.

[0066] Based on the above embodiment, the second image obtained by decoding the second latent vector includes a foreground area consistent with the original image, and the background area of ​​the second image is consistent with the style features of the foreground area, which conforms to the background description input by the user.

[0067] The image generation method provided in the embodiment of the present application can update the background image in the original image according to the original image input by the user and the preset background description to obtain a second image. In addition, the foreground image in the second image is consistent with the foreground image in the original image, the background image in the second image corresponds to the preset background description input by the user, and the style features of the background area and the foreground area of ​​the second image are consistent. In this way, the background can be automatically replaced and the foreground can be kept consistent according to the original image and background description input by the user, which significantly improves the fusion degree of the background and foreground and the overall effect of the picture, and realizes natural, rich and efficient background processing.

[0068] In some embodiments of the present application, reference Figure 5 As shown, the method for obtaining the second latent vector includes the following process.

[0069] Step S501, downsampling the first latent vector through the second downsampling module of the diffusion model to obtain a second downsampling result of the first latent vector.

[0070] In some embodiments of the present application, the diffusion model includes a U-net network structure, and the U-net network structure includes a symmetrical encoder-decoder structure, wherein the encoder extracts image features by stepwise downsampling.

[0071] In some embodiments of the present application, in the U-net network structure of the diffusion model, the encoder includes multiple second downsampling modules. For example, the encoder of the diffusion model includes N second downsampling modules, where N represents an integer greater than or equal to 2.

[0072] In some embodiments of the present application, downsampling the first latent vector by the second downsampling module of the diffusion model to obtain a second downsampling result of the first latent vector includes: performing N downsampling processes on the first latent vector N times by N second downsampling modules. According to the result of the Nth downsampling process, the second downsampling result is obtained.

[0073] For example Figure 4 As shown, the encoder in the U-net network structure of the diffusion model includes N=4 second downsampling modules. The spatial dimension of the first latent vector latent_i can be 64×64×320, where the resolution (or size) is 64×64 and the number of channels is 320.

[0074] The first latent vector latent_i passes through the first second downsampling module UNS1, and the spatial dimension of the downsampling result latent_iu2x can be 32 × 32 × 512. Compared with latent_i, the resolution of latent_iu2x is reduced by half, and the number of channels is increased.

[0075] The down-sampling result atent_iu2x passes through the second down-sampling module UNS2, and the spatial dimension of the down-sampling result latent_iu4x can be 16 × 16 × 512. Compared with latent_iu2x, the resolution of latent_iu4x is reduced by half, and the number of channels remains unchanged.

[0076] The down-sampling result latent_iu4x passes through the third second down-sampling module UNS3, and the spatial dimension of the down-sampling result latent_iu8x obtained can be 8×8×1024. Compared with latent_iu4x, the resolution of latent_iu8x is reduced by half, and the number of channels is increased.

[0077] The down-sampling result latent_iu8x passes through the fourth second down-sampling module UNS4, and the spatial dimension of the down-sampling result latent_iu16x can be 4×4×1024. Compared with latent_iu8x, the resolution of latent_iu16x is reduced by half, and the number of channels remains unchanged. Latent_iu16x can be used as the second down-sampling result.

[0078] In this way, the encoder of the U-net network structure can extract low-level to high-level features of the first latent vector through step-by-step downsampling. As the resolution decreases and the number of channels increases, the model can capture more abstract and global features, which can provide a basis for the subsequent fusion process.

[0079] In one example, the number of the second downsampling modules is the same as the number of the first downsampling modules of the control model, and the introduction of the second downsampling modules can refer to the description of the first downsampling modules. The second downsampling modules and the first downsampling modules may have similar structures (for example, both contain convolutional layers, activation functions, etc.), and the input data and the spatial dimensions of the output downsampling results of the second downsampling modules and the first downsampling modules are the same. However, the network parameters (such as weights and biases) of the second downsampling modules and the first downsampling modules are different, so even if the input is the same or the spatial dimensions are similar, the output downsampling results will be different due to different parameters.

[0080] For example, the first downsampling module CNS1 and the second downsampling module UNS1 both receive the first latent vector latent_i, the first downsampling module CNS1 outputs the first downsampling result latent_i2x with a spatial dimension of 32×32×512, and the second downsampling module UNS1 outputs the downsampling result latent_iu2x with a spatial dimension of 32×32×512. However, the first downsampling result latent_i2x and the downsampling result latent_iu2x are different in terms of feature representation, detail retention, noise processing, etc.

[0081] For another example, the first downsampling module CNS4 receives the first downsampling result latent_i8x, and the second downsampling module UNS4 receives the downsampling result latent_iu8x, wherein the spatial dimensions of the first downsampling result latent_i8x and the downsampling result latent_iu8x are both 8×8×1024. The first downsampling module CNS4 outputs the first downsampling result latent_i16x with a spatial dimension of 4×4×1024, and the second downsampling module UNS4 outputs the downsampling result latent_iu16x with a spatial dimension of 4×4×1024. However, the first downsampling result latent_i16x and the second downsampling result latent_iu16x are different in terms of feature representation, detail retention, noise processing, etc.

[0082] Based on the above embodiment, the control model and the diffusion model can use different model parameters to downsample the first latent vector, and the obtained control information and the second downsampling result contain different feature representations, detail retention, etc. In this way, comprehensive feature extraction of the foreground area can be achieved, so that the third latent vector obtained by subsequent fusion can contain a comprehensive representation of the foreground area, so that the foreground in the generated image is consistent with the original image, and the fusion degree of the background and foreground is improved.

[0083] Step S502, fusing the second downsampling result with the text vector to obtain a fused vector.

[0084] In some embodiments of the present application, the spatial dimension of the text vector may be different from the spatial dimension of the second downsampling result, and the dimension of the text vector may be adjusted according to the spatial dimension of the second downsampling result. The dimension-adjusted text vector is fused with the second downsampling result to obtain a fused vector.

[0085] In one example, the spatial dimension of the second downsampling result is 4×4×1024, and the spatial dimension of the text vector is 4×1024, and the spatial dimensions of the two are different. Before fusion, the dimension of the text vector can be adjusted using methods such as linear transformation or interpolation. For example, the spatial dimension of the text vector is adjusted to 4×4×1024, so that the spatial dimension of the text vector after dimension adjustment is consistent with the spatial dimension of the second downsampling result, and fusion can be achieved. For example, a fusion vector can be obtained based on the weighted sum of the second downsampling result and the text vector after dimension adjustment, and the spatial dimension of the fusion vector can be 4×4×1024. The present application does not make any specific restrictions on the fusion algorithm used in the above embodiments, nor does it make any specific restrictions on the spatial dimension of the fusion vector, which can be selected or adjusted according to actual needs.

[0086] Based on the above embodiment, a fused vector can be obtained by fusing the second downsampling result with the dimensionally adjusted text vector. The fused vector can be used in the upsampling process of the U-net decoder, and can effectively integrate the background prompt information input by the user into the decoding stage of the diffusion model, ensuring that the second latent vector in the image generation process can reflect the background features required by the user, so that the generated image background can accurately meet the user's expectations.

[0087] Step S503, extracting features from the fused vector through the intermediate layer of the diffusion model to obtain intermediate features.

[0088] In some embodiments of the present application, the U-net of the diffusion model directly connects the corresponding layers of the encoder and the decoder by introducing a skip connection. Figure 4 As shown in FIG. 1 , the encoder and the decoder are connected through an intermediate layer (Middle conv). The intermediate layer can extract features from the fusion vector, and the obtained intermediate features can be consistent with the spatial dimension of the fusion vector. The intermediate features can be further refined features corresponding to the fusion vector.

[0089] Based on the above example, the intermediate layer can connect the encoder and decoder of U-net through skip connections, thereby achieving effective fusion of low-level feature information and high-level feature information. It is also possible to further refine and extract features to obtain intermediate features while maintaining the spatial dimension of the fusion vector, which helps to obtain more complex feature representations.

[0090] Step S504: Obtain a second latent vector through an upsampling module of a diffusion model according to the intermediate features and the multiple control information.

[0091] In some embodiments of the present application, the decoder of the U-net structure of the diffusion model includes multiple upsampling modules, and multiple upsampling modules can be used to hierarchically and progressively fuse the intermediate features with multiple control information to obtain the second latent vector. In one example, the specific process of step S504 can be referred to Figure 6 Flowchart shown.

[0092] Based on the above embodiment, through multiple upsampling modules in the U-net structure decoder of the diffusion model, the intermediate features and the control information can be combined hierarchically and progressively. In each upsampling process, the background prompt information in the intermediate features interacts with the control information to guide the second latent vector to be generated according to the background direction specified by the user. As the upsampling level goes deeper, the image features gradually transition from low resolution to high resolution, and the details and features of the image foreground can be gradually refined and enhanced based on the control information, and the background information can also be based on the intermediate features. Ensure a high degree of consistency with user expectations.

[0093] In some embodiments of the present application, reference Figure 6 As shown, the specific process of step S504 includes the following process.

[0094] Step S601, determine the weighted sum f(0) of the intermediate feature and the Nth control information in the control information, and upsample the weighted sum f(0) through the first upsampling module of the diffusion model to obtain the upsampling result F(1).

[0095] In some embodiments of the present application, for example Figure 4 As shown, N = 4. The weighted sum f(0) of the intermediate features and the fourth control information output by the first down-sampling module CNS4 of the control model is up-sampled by the first up-sampling module UNU1 of the diffusion model to obtain the up-sampling result F(1).

[0096] Step S602, determine the weighted sum f(k-1) of the upsampling result F(k-1) and the N-(k-1)th control information in the control information, and perform upsampling processing on the weighted sum f(k-1) through the kth upsampling module of the diffusion model to obtain the upsampling result F(k).

[0097] In some embodiments of the present application, k represents an integer greater than or equal to 2 and less than or equal to N.

[0098] For example Figure 4 As shown, k = 2. The weighted sum f(1) of the up-sampling result F(1) and the third control information output by the first down-sampling module CNS3 of the control model is up-sampled by the second up-sampling module UNU2 of the diffusion model to obtain the up-sampling result F(2).

[0099] k=3. The upsampling result F(2) and the weighted sum f(2) of the second control information output by the first downsampling module CNS2 of the control model are upsampled by the third upsampling module UNU3 of the diffusion model to obtain the upsampling result F(3).

[0100] k=4=N. The weighted sum f(3) of the up-sampling result F(3) and the first control information output by the first down-sampling module CNS1 of the control model is up-sampled by the fourth up-sampling module UNU4 of the diffusion model to obtain the up-sampling result F(4).

[0101] Step S603: determine a second latent vector according to the upsampling result F(N).

[0102] For example Figure 4As shown, N=4, the upsampling result F(4) output by the fourth upsampling module UNU4 of the diffusion model is used as the second latent vector, such as latent_o, and the spatial dimension is 64×64×320.

[0103] Based on the above embodiment, the intermediate features and control information can be combined step by step and hierarchically through multiple upsampling modules of the diffusion model. This fusion method not only ensures the effective guidance of the control information in the generation process, but also enables the background prompt information in the intermediate features to interact with the control information, thereby guiding the second latent vector to be generated according to the background direction specified by the user.

[0104] Specifically, as the upsampling level goes deeper, the image features gradually transition from low resolution to high resolution. In this process, the control information gradually refines and enhances the details and features of the image foreground. Since the intermediate features contain the background information expected by the user, the hierarchical progressive fusion can ensure that the generated image background is highly consistent with the user's expectations. This consistency is not only reflected in the overall style of the background, but also in the detailed features in the background. Through the hierarchical progressive fusion of the U-net structure decoder of the diffusion model, the effective information in the intermediate features and control information can be fully utilized to improve the efficiency and quality of the generated image. In this way, not only the efficiency and quality of the generated image are improved, but also the generation process is made more flexible and scalable.

[0105] In some other embodiments of the present application, if the text vector includes multiple text vectors of different spatial dimensions, the text vectors of the corresponding spatial dimensions can be fused layer by layer in each downsampling module and each upsampling module of the control model based on the layer-by-layer fusion of the above-mentioned intermediate features and control features. In this way, the user-desired background features contained in the text vector can not only indicate the generation process of the second latent vector of the decoder of the diffusion model, but also be used to indicate the acquisition process of the second downsampling result of the encoder of the diffusion model.

[0106] Based on the above embodiment, by fusing the text vectors of the corresponding spatial dimensions layer by layer to the output results of each downsampling module and each upsampling module of the control model, the diffusion model can more accurately understand and generate the background features expected by the user. This method enables the model to capture information consistent with the text description in the encoding stage, and further refine this information in the decoding stage to generate high-quality images. In addition, since the text vector contains information of different spatial dimensions, the model can generate a background with richer details and layering.

[0107] In some embodiments of the present application, for example Figure 7 As shown, the training method of the diffusion model includes the following process.

[0108] Step S701, obtaining training data.

[0109] In some embodiments of the present application, the training data includes a sample image, a third image obtained by masking the background area of ​​the sample image, and a background description text of the sample image. The third image can be obtained according to the acquisition method of the first image. The background description text is a text description of the background of the sample image, which is used to guide the model to generate a background that matches the description, and can be obtained using an open source multimodal image description model.

[0110] Step S702: Generate a fourth image through a diffusion model according to the third image and the background description text.

[0111] In some embodiments of the present application, the third image and the background description text are used as input to generate the fourth image through the diffusion model. In this process, the model needs to learn to generate a background that matches the description based on the background description text, and at the same time process the mask area, possibly filling these areas by generating new foreground content or retaining / reconstructing part of the original foreground content.

[0112] Step S703: determine a loss value based on the fourth image and the sample image, and train a diffusion model according to the loss value.

[0113] In some embodiments of the present application, a preset loss function may be used to calculate the loss value between the fourth image and the sample image. For example, the loss function includes but is not limited to L1 loss, L2 loss, perceptual loss, etc. The loss value reflects the similarity or difference between the image generated by the model and the real sample image. The larger the loss value, the worse the performance of the model.

[0114] In some embodiments of the present application, if the loss value is greater than a preset loss threshold, a back propagation algorithm and an optimizer (such as Adam, etc.) can be used to update the weight parameters of the model to minimize the loss value. Through multiple iterations of training, the model gradually learns how to generate an image consistent with the corresponding sample image based on the given background description text and the masked image.

[0115] Based on the above embodiment, the diffusion model can be trained by combining the background description text and the masked image, so that the model can generate an image that is consistent with the input background description text and has a similar background.

[0116] refer to Figure 8As shown, it is an example diagram of the image generation process provided by the embodiment of the present application. After the original image is masked with the mask image, a first image is obtained. The background description text is text-encoded to obtain a text vector. The first image obtains a first latent vector via a decoder, and multiple control information is obtained according to the first latent vector based on the control model. Based on the diffusion model, a second latent vector is obtained according to the first latent vector, multiple control information and the text vector corresponding to the preset background description. The second latent vector is decoded to obtain a second image.

[0117] Based on the above embodiment, according to the diffusion model, a natural and beautiful picture with a background can be generated by only needing the user to input the desired background. The control model can be used to control the foreground to remain unchanged and randomly generate the background so that the generated background and foreground are well integrated.

[0118] See also Fig. 9 , is a principle block diagram of an image generating device provided in an embodiment of the present application, and provides an image generating device adapted to one of the purposes of the present application, which is a functional embodiment of the image generating method of the present application. The image generating device includes: a mask module 91, which is used to mask the background area of ​​the original image to obtain a first image, and the first image includes the foreground area of ​​the original image and the masked background area. An encoding module 92 is used to obtain a first latent vector of the first image. An acquisition module 93 is used to obtain multiple control information of the foreground area according to the first latent vector. A fusion module 94 is used to obtain a second latent vector according to the first latent vector, multiple control information and a text vector corresponding to a preset background description. A decoding module 95 is used to decode the second latent vector to obtain a second image.

[0119] Another embodiment of the present application also provides an electronic device. Figure 1 The application environment is only an example. In other exemplary embodiments, the computer program product implementing the image generation method of the embodiment of the present application can also be run on any electronic device with sufficient computing power (such as Fig.10 In the electronic device shown in the figure), the various steps of the image generating method are executed to provide the image generating function.

[0120] See also Fig.10 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, in one embodiment of the present application, the electronic device 800 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 800.

[0121] like Fig.10 As shown, the electronic device 800 may include, but is not limited to, a communication module 81, a memory 82, a processor 83, an input / output (I / O) interface 84, and a bus 85. The processor 83 is coupled to the communication module 81, the memory 82, and the I / O interface 84 through the bus 85.

[0122] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 800 and does not constitute a limitation of the electronic device 800 , and may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the electronic device 800 may also include a network access device, etc.

[0123] The communication module 81 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0124] The memory 82 can be used to store computer-readable instructions and / or modules. The processor 83 implements various functions of the electronic device 800 by running or executing the computer-readable instructions and / or modules stored in the memory 82 and calling the data stored in the memory 82. The memory 82 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device 800, etc. The memory 82 can include non-volatile and volatile memories, such as: a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0125] The memory 82 may be an external memory and / or an internal memory of the electronic device 800. Furthermore, the memory 82 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0126] The processor 83 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 83 is the computing core and control center of the electronic device 800, and uses various interfaces and lines to connect various parts of the entire electronic device 800, and execute the operating system of the electronic device 800 and various installed applications, program codes, etc.

[0127] Exemplarily, the computer-readable instructions may be divided into one or more modules / sub-modules / units, one or more modules / sub-modules / units are stored in the memory 82 and executed by the processor 83 to complete the present application. One or more modules / sub-modules / units may be a series of computer-readable instruction segments capable of completing a specific function, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the electronic device 800. For example, the computer-readable instructions may be divided into a mask module 91, an encoding module 92, an acquisition module 93, a fusion module 94, and a decoding module 95.

[0128] If the module / unit integrated in the electronic device 800 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete it through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0129] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable files or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer-readable instruction codes, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM).

[0130] Combination Figures 2 to 8 The memory 82 in the electronic device 800 stores computer-readable instructions, and the processor 83 can execute the computer-readable instructions stored in the memory 82 to implement the following Figures 2 to 8 The image generation method shown. Specifically, the specific implementation method of the processor 83 for the above-mentioned computer readable instructions can refer to Figures 2 to 8 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0131] The I / O interface 84 is used to provide a channel for user input or output. For example, the I / O interface 84 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0132] The bus 85 is at least used to provide a channel for mutual communication among the communication module 81 , the memory 82 , the processor 83 , and the I / O interface 84 in the electronic device 800 .

[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0134] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0136] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0137] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Masking the background area of ​​the original image to obtain a first image, wherein the first image includes the foreground area of ​​the original image and the masked background area; Obtaining a first latent vector of the first image; Acquire a plurality of control information of the foreground area according to the first latent vector; Obtaining a second latent vector according to the first latent vector, the plurality of control information and text vectors corresponding to the preset background description; The second latent vector is decoded to obtain a second image.

2. The image generation method according to claim 1, characterized in that: The acquiring, according to the first latent vector, a plurality of control information of the foreground area comprises: Downsampling the first latent vector by controlling multiple first downsampling modules of the model to obtain multiple first downsampling results of the first latent vector; The multiple control information are obtained according to the control information corresponding to each first down-sampling result.

3. The image generation method according to claim 2, characterized in that: For the jth first down-sampling module among the multiple first down-sampling modules, the down-sampling processing on the first latent vector includes: If j is equal to 1, the first latent vector is downsampled by a first downsampling module of the control model to obtain a first downsampling result; If j is an integer greater than 1, the j-1th first downsampling result is downsampled by the jth first downsampling module of the control model to obtain the jth first downsampling result.

4. The image generation method according to claim 1, characterized in that: The obtaining a second latent vector according to the first latent vector, the plurality of control information and the text vector corresponding to the preset background description includes: Downsampling the first latent vector by a second downsampling module of a diffusion model to obtain a second downsampling result of the first latent vector; Fusing the second downsampling result with the text vector to obtain a fused vector; Extracting features from the fused vector through an intermediate layer of the diffusion model to obtain intermediate features; The second latent vector is obtained through an upsampling module of the diffusion model according to the intermediate features and the multiple control information.

5. The image generation method according to claim 4, characterized in that: The diffusion model includes N second downsampling modules, where N represents an integer greater than or equal to 2, and the second downsampling module of the diffusion model performs downsampling processing on the first latent vector to obtain a second downsampling result of the first latent vector, including: Performing N downsampling processing on the first latent vector through the N second downsampling modules; The second down-sampling result is obtained according to the result of the Nth down-sampling process.

6. The image generation method according to claim 4, characterized in that: The control information includes N control information, where N represents an integer greater than or equal to 2; The obtaining the second latent vector by using an upsampling module of the diffusion model according to the intermediate feature and the plurality of control information includes: Determine a weighted sum f(0) of the intermediate feature and the Nth control information in the control information, and perform upsampling processing on the weighted sum f(0) through a first upsampling module of the diffusion model to obtain an upsampling result F(1); Determine the weighted sum f(k-1) of the upsampling result F(k-1) and the N-(k-1)th control information in the control information, and perform upsampling processing on the weighted sum f(k-1) through the kth upsampling module of the diffusion model to obtain the upsampling result F(k); wherein k represents an integer greater than or equal to 2 and less than or equal to N; According to the upsampling result F(N), the second latent vector is determined.

7. The image generation method according to claim 4, characterized in that: The fusing the second down-sampling result with the text vector to obtain a fused vector includes: According to the spatial dimension of the second downsampling result, adjusting the dimension of the text vector; The dimension-adjusted text vector and the second downsampling result are fused to obtain the fused vector.

8. The image generation method according to claim 4, characterized in that: The training method of the diffusion model includes: Acquire training data, the training data comprising a sample image, a third image obtained by masking a background area of ​​the sample image, and background description text of the sample image; generating a fourth image by using the diffusion model according to the third image and the background description text; A loss value is determined based on the fourth image and the sample image, and the diffusion model is trained according to the loss value.

9. An image generating device, characterized in that: The image generating device is used to implement the image generating method according to any one of claims 1 to 8, and the image generating device comprises: A mask module, configured to mask the background area of ​​the original image to obtain a first image, wherein the first image includes the foreground area of ​​the original image and the masked background area; An encoding module, used to obtain a first latent vector of the first image; An acquisition module, configured to acquire a plurality of control information of the foreground area according to the first latent vector; A fusion module, configured to fuse the first latent vector, the plurality of control information and the text vector corresponding to the preset background description to obtain a second latent vector; A decoding module is used to decode the second latent vector to obtain a second image.

10. An electronic device, characterized in that: include: Memory, and A processor, wherein the processor executes computer-readable instructions stored in the memory to implement the image generation method according to any one of claims 1 to 8.

Images