Image generation method and device for realizing element replacement and electronic equipment

Abstract

The embodiment of the invention provides an image generation method and device for realizing element replacement and electronic equipment, and the method comprises the steps: inputting a main body image and a background image into a trained image generation model according to the given main body image and the background image, and enabling the main body image to have pre-selected main body elements which need to be replaced; an image coding module in the image generation model obtains a first feature vector of a main body image, a second feature vector of a background image and a mask vector of a main body element; the first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, an image generation module in the image generation model performs iterative processing according to the vector and a random Gaussian noise image and outputs a target image, and the target image is an image with a main body element located in the background image; the method solves the problems that in the prior art, the processing speed is low, the efficiency is low, and complex post-processing is needed for editing quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image generation method for implementing element replacement, an image generation device for implementing element replacement, an electronic device and a computer-readable medium. Background Art

[0002] In the field of digital image processing, the replacement of specific elements has always been one of the cutting-edge issues in image generation tasks. Existing technologies, including various deep learning algorithms, image editing software, and automated tools, provide users with the ability to identify, edit, and replace specific elements in images. These technologies mainly rely on advanced object detection algorithms, content-aware fill technology, and user interaction interfaces. Among them, object detection algorithms such as YOLO have been widely used in the market to identify specific objects in images and determine their boundaries. Content-aware technologies, such as the content-aware fill function of Adobe Photoshop, intelligently replace or delete objects in images by analyzing surrounding pixels.

[0003] Although existing technologies provide basic element replacement capabilities, they still have limitations in handling complex scenes and details. For example, when the image background is complex or the lighting conditions change, existing algorithms often fail to accurately segment and replace objects, causing the replaced images to look unnatural or have obvious traces of editing. In addition, most existing tools still require a considerable degree of manual intervention in operation, which not only increases the complexity of use, but also limits processing speed and efficiency.

[0004] With the rapid development of artificial intelligence and machine learning technologies, there is an increasing demand for smarter and more automatic image editing methods in the market, with the expectation of being able to edit images faster and more accurately, especially in terms of replacing specific elements. This demand has spawned research on more advanced image recognition and processing algorithms, aiming to reduce the need for manual editing while improving the quality of editing.

[0005] Although current technologies have made significant progress in the field of image-based image processing tasks for specific element replacement, their limitations in automation and accuracy highlight new research and development opportunities. Existing solutions often rely on a certain degree of user input and complex post-processing, which becomes a significant shortcoming in application scenarios that process large numbers of images or require fast response. Summary of the invention

[0006] The embodiment of the present invention provides an image generation method, device and electronic device for realizing element replacement, so as to solve the problems of slow processing speed, low efficiency and complicated post-processing required for editing quality in the prior art.

[0007] An embodiment of the present invention discloses an image generation method for implementing element replacement, the method comprising:

[0008] According to a given subject image and background image, the subject image and the background image are input into a trained image generation model, wherein the subject image contains pre-selected subject elements that need to be replaced;

[0009] The image encoding module in the image generation model obtains a first feature vector of the subject image, a second feature vector of the background image, and a mask vector of the subject element;

[0010] The first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image to output a target image, wherein the target image is an image in which the main element is located in the background image;

[0011] The image generation model is a model for generating images from images that is trained using a pre-constructed training data set.

[0012] Optionally, according to a given subject image and background image, inputting the subject image and background image into a trained image generation model comprises:

[0013] If there is text description information corresponding to the target image input by the user, the text description information, the main image and the background image are input into the trained image generation model;

[0014] Accordingly, the image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image and the mask vector of the main element, including:

[0015] The image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image, and the mask vector of the main element, and the text encoding module in the image generation model obtains the text vector of the text description information;

[0016] The first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, including:

[0017] The first feature vector, the second feature vector, the mask vector and the text vector are fused to generate a fused vector.

[0018] Optionally, the method further comprises:

[0019] The target image is post-processed to obtain an expected target image.

[0020] Optionally, the image encoding module includes: a Mask R-CNN or ViT model;

[0021] The image generation module includes: stable diffusion model or UNet model.

[0022] Optionally, the method further comprises:

[0023] Based on different video frames of the video stream, a training data set is constructed; the training data set includes: a plurality of groups of images, each group of images includes: a first main image, a first background image, and a replacement image;

[0024] The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image without the subject element; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image;

[0025] According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model;

[0026] The result image is matched with the first main image of the group of images, and the parameters of the image generation model are adjusted according to the matching result. When the matching result reaches the optimal value, a trained image generation model is obtained.

[0027] The embodiment of the present invention also discloses a training method for an image generation model for implementing element replacement, comprising:

[0028] Based on different video frames of the video stream, a training data set is constructed; the training data set includes: a plurality of groups of images, each group of images includes: a first main image, a first background image, and a replacement image;

[0029] The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image with the subject element removed; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image; the replacement image has a pre-selected subject element to be replaced;

[0030] According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model;

[0031] The result image is matched with the first main image of the group of images, and the parameters of the image generation model are adjusted according to the matching result. When the matching result reaches the optimal value, a trained image generation model is obtained.

[0032] Optionally, the step of using the first background image and the replacement image in each group of images as inputs of an image generation model to obtain a result image output by the image generation model includes:

[0033] The image encoding module in the image generation model obtains a first feature vector of the replacement image, a second feature vector of the background image, and a mask vector of a main element in the replacement image;

[0034] The first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing based on the vector and the random Gaussian noise image to output a result image.

[0035] The embodiment of the present invention further discloses an image generation device for implementing element replacement, comprising:

[0036] An input module, used for inputting the subject image and the background image into the trained image generation model according to a given subject image and a background image, wherein the subject image contains pre-selected subject elements that need to be replaced;

[0037] An image processing module, used for the image encoding module in the image generation model to obtain the first eigenvector of the main image, the second eigenvector of the background image and the mask vector of the main element; the first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image;

[0038] An output module, used for outputting a target image after being processed by the image processing module, wherein the target image is an image in which a main element is located in the background image;

[0039] The image generation model is a model for generating images from images that is trained using a pre-constructed training data set.

[0040] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0041] The memory is used to store computer programs;

[0042] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0043] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0044] The embodiments of the present invention include the following advantages:

[0045] In an embodiment of the present invention, a pre-trained end-to-end model, i.e., an image generation model, is used to perform element replacement processing on a main image and a background image, and a triplet vector is extracted as guiding information to realize element replacement, thereby achieving accuracy in feature recognition and making element replacement more natural and fast in processing speed.

[0046] In order to obtain a more efficient target image in this embodiment, text description information can also be combined. At this time, a text encoding module for generating text vectors can be added to the end-to-end image generation model, and then the aforementioned generated triple vector can be transformed into a quadruple vector, which better guides the image generation module to generate the final target image, thereby achieving a more efficient, automatic and accurate target image that replaces the main elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of the steps of an image generation method for implementing element replacement provided in an embodiment of the present invention;

[0048] Figure 2 is a flowchart of the steps of a training method for an image generation model that implements element replacement provided in an embodiment of the present invention;

[0049] Figure 3 is a structural block diagram of an image generation device for implementing element replacement provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of an image generation method for implementing element replacement in an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of an image generation method having a text encoding module in an embodiment of the present invention;

[0052] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] YOLO (You Only Look Once) is a real-time object detection system. YOLO divides an image into multiple grids and predicts the objects, their bounding boxes, and class probabilities in each grid. Its advantages are high speed and efficiency, and it can predict the categories and locations of multiple objects at the same time.

[0056] Mask R-CNN is a deep learning model used in computer vision for object detection and instance segmentation. It is an extension of Faster R-CNN with an additional branch for generating object segmentation masks.

[0057] ViT (Vision Transformer) is a deep learning model that applies the Transformer model to visual classification tasks. The image is divided into non-overlapping blocks and input into the Transformer model, and the global feature representation of the image is achieved through the self-attention mechanism.

[0058] UNet is a neural network mainly used for image segmentation tasks. The structure of UNet is U-shaped, consisting of a contraction path (encoder) and a symmetrical expansion path (decoder), which enables the network to accurately locate the area in the image while maintaining contextual information.

[0059] Stable Diffusion is a deep learning model based on the diffusion model, which can generate realistic images based on the input description, or convert a given image into a different style. It can capture complex image distributions and maintain high stability and practicality when generating new images.

[0060] Each of the above algorithms or models can be applied to the image generation model to achieve element replacement of the image. The image generation method of element replacement is widely used in the fields of image processing and computer vision. It replaces or modifies specific elements in an image to generate a new image. For example, it can be used for advertising and marketing. In advertisements, products of different colors or styles are displayed. In this case, there is no need to take multiple photos. The method of element replacement is used to adapt to different advertising needs. It can also be used for virtual fitting. In online shopping, the image generation method of element replacement is used to enable users to try different clothes in a virtual fitting room. It can also be used for scene reconstruction or role replacement in film and television special effects, etc. This embodiment is not limited to it. This embodiment provides a more automatic and intelligent image generation method for achieving element replacement.

[0061] Reference Figure 1 , shows a flowchart of a method for generating an image to achieve element replacement provided in an embodiment of the present invention, which may specifically include the following steps:

[0062] 101. According to a given subject image and background image, the subject image and the background image are input into a trained image generation model, wherein the subject image contains pre-selected subject elements that need to be replaced;

[0063] 102. An image encoding module in the image generation model obtains a first feature vector of the subject image, a second feature vector of the background image, and a mask vector of the subject element;

[0064] 103. The first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector. The image generation module in the image generation model performs iterative processing based on the vector and the random Gaussian noise image to output a target image, wherein the target image is an image in which the main elements are located in the background image.

[0065] The image generation model in this embodiment may be a model for generating images from images that is trained using a pre-constructed training data set.

[0066] Above Figure 1 The method of using an image generation model is shown, that is, through a pre-trained end-to-end model, namely the image generation model, the element replacement processing of the main image and the background image is performed, and the triplet vector is extracted as the guiding information to realize the element replacement, thereby achieving the accuracy of feature recognition, and making the element replacement more natural and the processing speed fast.

[0067] See also Figure 4 In the specific implementation process, the subject image and the background image are input into the trained image generation model. The image generation model includes an image encoding module and an image generation module. The image encoding module is also connected to a target detection head and a mask terminal to convert the image information into a numerical representation (vector) that the model can understand. The target detection head outputs the feature vector of the subject image, which is the first feature vector. The mask terminal outputs the mask vector of the subject in the background image. The image encoding module directly outputs the feature vector of the background image, which is the second feature vector.

[0068] The first eigenvector, the second eigenvector and the mask vector are concatenated / fused together as a triplet vector, the triplet vector is input into an image generation module, and a target image is output.

[0069] Specifically, the triplet vector provides guidance information for image generation. Starting from an image containing random noise, the initial noisy image is iteratively processed using a deep learning network such as UNet. Each step reduces the proportion of noise while gradually adding image content and details based on the triplet vector. Throughout the iterative process, the triplet vector is used to guide the direction of image generation; the image generation module transforms the image from random noise to a specific target image through multiple iterations. Throughout the process, the image generation module will make adjustments based on the difference between the current image state and the triplet vector, gradually increasing details and clarity.

[0070] After multiple iterations, the image generation module generates a clear, detailed image that meets the requirements.

[0071] In this embodiment, an end-to-end deep learning model, namely an image generation model, is adopted. In image processing, an image encoding module is used in advance to extract triplet vectors as guiding information, which is then input into the image generation module again to guide the generation of the target image, thereby realizing the task of replacing the image with a specific element. It has the advantages of being more automatic and intelligent, and can realize more accurate and natural replacement of specific elements.

[0072] In addition, the step 103 of fusing the first feature vector, the second feature vector and the mask vector to generate a fused vector specifically includes:

[0073] Make sure the first feature vector, the second feature vector, and the mask vector have the same size. If the feature vectors have different sizes, use upsampling or downsampling techniques to adjust the size of each vector;

[0074] Concatenate the resized first eigenvector, the second eigenvector, and the mask vector along the channel dimension;

[0075] The concatenated feature vectors are fused through two convolutional layers. The first convolutional layer converts the concatenated feature vectors into intermediate feature representations, and the second convolutional layer further processes these features to generate the final fused feature vector.

[0076] Among them, upsampling is usually used to enlarge the low-resolution feature map to high resolution, and downsampling is usually used to reduce the high-resolution feature map to low resolution.

[0077] In practical applications, in order to better meet the needs of users or improve the user experience, the above Figure 1 The method shown may further include step 104 described below which is not shown in the figure.

[0078] 104. Post-process the target image to obtain an expected target image.

[0079] For example, post-processing the target image may specifically include:

[0080] Perform color correction on the generated target image to ensure that it is consistent with the color and brightness of the background image; or, enhance the contrast of the target image to make it look more natural and clear; or, use image processing techniques (such as bilateral filtering, Gaussian blur, etc.) to smooth the edges of the target image to eliminate possible jagged and unnatural boundaries.

[0081] In practical applications, Gaussian blur is used to smooth the edge of the target image, which may specifically include the following steps not shown in the figure:

[0082] S1, load the target image that needs post-processing;

[0083] S2, determine the parameters of Gaussian blur, mainly including the size of Gaussian kernel and standard deviation σ;

[0084] S3, the size of the Gaussian kernel is usually an odd number, such as 3×3, 5×5, 7×7, etc. A larger kernel will bring a stronger smoothing effect, but it will also increase the computational complexity.

[0085] S4, calculating the weight of the Gaussian kernel according to the selected standard deviation σ, while ensuring that the sum of the weights of the Gaussian kernel is 1 to keep the brightness of the image unchanged;

[0086] S5. For each pixel in the image, a convolution operation is performed using a Gaussian kernel to calculate a new pixel value; the above convolution operation is performed on each pixel in the image to obtain a filtered image.

[0087] The above post-processing is mainly to better meet the needs of users and ensure the applicability of images in fields such as monitoring or artistic creation.

[0088] In addition, if Figure 5 As shown, when there is text description information corresponding to the target image input by the user, the main image and the background image are input into the trained image generation model, specifically including:

[0089] If there is text description information corresponding to the target image input by the user, the text description information, the main image and the background image are input into the trained image generation model;

[0090] Accordingly, the image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image and the mask vector of the main element, including:

[0091] The image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image, and the mask vector of the main element, and the text encoding module in the image generation model obtains the text vector of the text description information.

[0092] In order to obtain a more efficient target image in this embodiment, text description information can also be combined. At this time, a text encoding module for generating text vectors can be added to the end-to-end image generation model, and then the aforementioned generated triple vector can be transformed into a quadruple vector, which better guides the image generation module to generate the final target image, thereby achieving a more efficient, automatic and accurate target image that replaces the main elements.

[0093] For example, the text description is "a cat running on the lawn". The text description is encoded by the text encoding module in the image generation model, which converts the natural language "a cat running on the lawn" into a numerical representation that the image generation module can understand, and provides it as known information to the image generation module;

[0094] The first feature vector, the second feature vector, the mask vector and the text vector are fused to generate a fused vector.

[0095] That is to say, when there is text description information corresponding to the target image input by the user, the first feature vector, the second feature vector, the mask vector and the text vector are concatenated / fused together as a four-tuple vector, which is input into the image generation module to output the target image.

[0096] The quadruple vector provides guidance information for image generation. Starting from an image containing random noise, the initial noisy image is iteratively processed using a deep learning network such as UNet. Each step reduces the proportion of noise while gradually adding content and details of the image based on the quadruple vector. Throughout the iterative process, the quadruple vector is used to guide the direction of image generation; the image generation module transforms the image from random noise to specific content through multiple iterations. Throughout the process, the image generation module will make adjustments based on the difference between the current image state and the triplet vector, gradually increasing details and clarity.

[0097] This embodiment uses an end-to-end deep learning model to first extract the quadruple vector, input it into the image generation module, guide the generation of the target image, and realize the task of replacing the image with a specific element, which has the advantages of being more automatic and intelligent.

[0098] The image encoding module of the image generation model of this embodiment may include: Mask R-CNN or ViT model; the image generation module may include: stable diffusion model or UNet model.

[0099] Among them, the Mask R-CNN model is a deep learning model for instance segmentation, which can not only detect objects in images, but also generate accurate segmentation masks for each detected object;

[0100] The core components of the Mask R-CNN model include:

[0101] Backbone network: usually a pre-trained convolutional neural network, such as ResNet, used to extract feature maps from the input image. These feature maps contain high-level semantic information in the image and provide a basis for subsequent detection and segmentation tasks;

[0102] RPN: used to generate candidate regions. RPN predicts the position and size of the rectangular box that may contain the object based on the feature map extracted by the backbone network.

[0103] RoL Align layer: After RPN generates candidate regions, the RoL Align layer is responsible for extracting the feature vector corresponding to each candidate region from the feature map. It uses bilinear interpolation for sampling to avoid quantization errors, thereby retaining more spatial information, which is crucial for generating accurate segmentation masks.

[0104] Classification and regression branch: used to predict the category and fine-tune the bounding box of each candidate region, that is, to determine which category of object the region belongs to, fine-tune the position and size of the candidate region, and improve positioning accuracy;

[0105] Mask generation branch: used to generate the segmentation mask of each detected object. It receives the feature vector output by the RoIlAlign layer as input, generates a binary mask with the same resolution as the original image through a fully convolutional network (FCN), and identifies the specific location of the object.

[0106] Similarly, the ViT model is a computer vision model based on the Transformer architecture, which includes:

[0107] Image slicing layer: used to split the input image into non-overlapping blocks of fixed size and flatten each block into a one-dimensional vector;

[0108] Linear projection layer: used to convert a one-dimensional vector into a vector of fixed dimension;

[0109] Position embedding layer: used to add the vector of each block to its corresponding position embedding to obtain the final input sequence;

[0110] Transformer encoder layer: Each encoder layer consists of two sub-layers: multi-head self-attention and feed-forward neural network, which are used to obtain the features of each block and perform linear transformation on it;

[0111] Fully connected layer: used for classification tasks and output results.

[0112] The image generation module includes: stable diffusion model or UNet model.

[0113] The stablediffusion model is a generative artificial intelligence model based on the diffusion model, which is used to generate high-quality images; the diffusion model generates images through a series of step-by-step denoising processes.

[0114] The working principle of the stable diffusion model is: gradually add Gaussian noise to the image until the image becomes completely random and eventually becomes a pure noise image; gradually remove the noise from the pure noise image to restore the clarity of the image.

[0115] The stable diffusion model includes: a UNet encoder-decoder structure, in which the encoder is used to gradually downsample the input image to extract high-level features; the decoder gradually upsamples the high-level features to restore the clarity of the image. At the same time, a jump connection is added between the encoder and the decoder to transfer low-level feature information, which helps to generate finer image details;

[0116] Time step embedding component,In each denoising process, time step embedding incorporates the time step information into the input of the model through a position encoding mechanism.

[0117] In this embodiment, the stable diffusion model can generate a variety of high-quality, high-resolution graphics, and it also supports multiple conditional inputs, such as text, reference images, etc., and is suitable for a variety of application scenarios.

[0118] The framework of the above-mentioned image generation model can be used as an example for illustration. This embodiment is not limited to it and is selected and configured according to actual needs. In this embodiment, a triple or quadruple vector is generated as guiding information so that the target image output by the image generation model is more in line with the user's expectations.

[0119] On the other hand, in this embodiment, the image generation model is trained using a training data set constructed by the user himself, rather than a data set commonly used on the current network, thereby ensuring the training effect of the image generation model.

[0120] For example, before using / predicting the image generation model, a training data set may be constructed based on different video frames of the video stream; the training data set includes: multiple groups of images, each group of images includes: a first main image, a first background image, and a replacement image;

[0121] In this embodiment, a subject segmentation algorithm such as Mask R-CNN is used to segment the subject and background of the original photo to obtain a first background image;

[0122] The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image without the subject element; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image;

[0123] According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model;

[0124] The result image is matched with the first main image of the group of images, and the parameters of the image generation model are adjusted according to the matching result. When the matching result reaches the optimal value, a trained image generation model is obtained.

[0125] In the training data construction, this embodiment uses a subject segmentation algorithm to segment the subject and background of the original photo to obtain a first subject image and a first background image, and the first subject image is pasted on a pre-prepared random background image to obtain a replacement image. The first subject image, the first background image and the replacement image constitute a training data set, thereby realizing large-scale training data construction, and having the characteristics of convenient and fast large-scale construction of training data sets. In addition, different frames can be extracted from the video as the original images of the training set to realize dynamic and natural replacement of specific elements.

[0126] In the specific implementation process, different frames can be extracted from the same video as original photos of the training set.

[0127] In the above-mentioned image generation method for realizing element replacement, an end-to-end deep learning model is adopted, that is, an image encoding module is used to extract the first feature vector, the second feature vector, and the mask vector to form a triplet vector (the feature vector of the text representation extracted by the text encoding module can also be used to form a quadruple vector), which is input into the image generation module to guide the generation of the target image, thereby realizing the task of generating an image of a specific element replacement image, which has the advantages of being more automatic and intelligent.

[0128] Reference Figure 2 This embodiment also discloses a training method for an image generation model for implementing element replacement, comprising the following steps:

[0129] 201. Construct a training data set based on different video frames of a video stream; the training data set includes: a plurality of groups of images, each group of images includes: a first main image, a first background image, and a replacement image;

[0130] The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image with the subject element removed; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image; the replacement image has a pre-selected subject element to be replaced;

[0131] 202. According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model;

[0132] 203. Match the result image with the first main image of the group of images, adjust the parameters of the image generation model according to the matching result, and obtain the trained image generation model when the matching result reaches the optimal one.

[0133] In the specific implementation process, when capturing video frames from a video stream, it is necessary to ensure that the video stream contains a rich variety of scenes and subject elements. The video source can be a surveillance camera, a movie clip, a video uploaded by a user, etc.

[0134] When capturing video frames from a video stream, use a video processing tool such as OpenCV to extract video frames. You may choose to extract every few frames to reduce redundancy while ensuring that the extracted video frames have sufficient resolution and clarity.

[0135] In this embodiment, the image generation model includes an image encoding module and an image generation module. The image encoding module is also connected to a target detection head and a mask terminal to convert image information into a numerical representation that the model can understand. The target detection algorithm is used to detect the subject elements in the video frame. The subject elements can be people, animals, vehicles, etc. According to the detected subject elements, the background image is separated from the video frame, that is, the background image after removing the subject elements;

[0136] Select a random background image, place the detected main elements into the random background image, and generate a replacement image.

[0137] Using the first background image and the replacement image in each group of images as inputs of the image generation model, and obtaining a result image output by the image generation model;

[0138] Match the result image with the first main image of the group of images, and calculate the loss between the generated result image and the first main image. Commonly used loss functions include mean square error, structural similarity index and perceptual loss. Assuming that the mean square error loss function is used, the parameters of the model are updated through the back propagation algorithm until the mean square error loss function is minimized.

[0139] Furthermore, an independent validation set is used to evaluate the performance of the trained image generation model. The validation set contains the first subject image, the first background image, and the replacement image of another video stream. The loss and indicators on the validation set are calculated to evaluate the generalization ability of the model.

[0140] In addition, you can also increase the diversity of the training data set and improve the robustness of the model through data augmentation (such as rotation, scaling, flipping, etc.); use early stopping to prevent the model from overfitting and stop training when performance is no longer improving.

[0141] The step of using the first background image and the replacement image in each group of images as inputs of the image generation model and obtaining a result image output by the image generation model comprises:

[0142] The image encoding module in the image generation model obtains a first feature vector of the replacement image, a second feature vector of the background image, and a mask vector of a main element in the replacement image;

[0143] The first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing based on the vector and the random Gaussian noise image to output a result image.

[0144] The image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image, and the output result graph includes:

[0145] Assume that the first eigenvector, the second eigenvector and the mask vector are fused, and the generated fused vector is a triplet vector. Starting from an image containing random noise, a deep learning network such as UNet is used to iteratively process the initial noisy image. Each step will reduce the proportion of noise, while gradually adding the content and details of the image according to the triplet vector. Throughout the iterative process, the triplet vector is used to guide the direction of image generation; the image generation module transforms the image from random noise to specific content through multiple iterations. Throughout the process, the image generation module will adjust according to the difference between the current image state and the triplet vector, gradually increasing the details and clarity.

[0146] In this embodiment, fusing the first feature vector, the second feature vector and the mask vector to generate a fused vector specifically includes:

[0147] Make sure the first feature vector, the second feature vector, and the mask vector have the same size. If the feature vectors have different sizes, use upsampling or downsampling techniques to adjust their sizes;

[0148] Concatenate the resized first eigenvector, the second eigenvector, and the mask vector along the channel dimension;

[0149] The concatenated feature vectors are fused through two convolutional layers. The first convolutional layer converts the concatenated feature vectors into intermediate feature representations, and the second convolutional layer further processes these features to generate the final fused feature vector.

[0150] In addition, when the video frames in the video stream contain text description information, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain the result image output by the image generation model, specifically including:

[0151] Input text description information, main image and background image into the image generation model;

[0152] Accordingly, the image encoding module in the image generation model obtains the first feature vector of the replacement image, the second feature vector of the background image and the mask vector of the main element in the replacement image; the text encoding module in the image generation model obtains the text vector of the text description information;

[0153] The first feature vector, the second feature vector, the mask vector and the text vector are fused to generate a fused vector. The image generation module in the image generation model performs iterative processing based on the vector and the random Gaussian noise image to output a result image.

[0154] The image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image, and the output result graph includes:

[0155] When there is text description information corresponding to the target image input by the user, the first feature vector, the second feature vector, the mask vector and the text vector are concatenated / fused together as a four-tuple vector, which is input into the image generation module to output the result image.

[0156] The quadruple vector provides guidance information for image generation. Starting from an image containing random noise, the initial noisy image is iteratively processed using a deep learning network such as UNet. Each step reduces the proportion of noise while gradually adding content and details of the image based on the quadruple vector. Throughout the iterative process, the quadruple vector is used to guide the direction of image generation; the image generation module transforms the image from random noise to specific content through multiple iterations. Throughout the process, the image generation module will make adjustments based on the difference between the current image state and the triplet vector, gradually increasing details and clarity.

[0157] A training method for an image generation model that implements element replacement in an embodiment of the present invention has the characteristics of convenient and fast large-scale construction of training data sets. In addition, different frames can be extracted from the video as original images of the training set to achieve dynamic and natural replacement of specific elements.

[0158] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0159] The embodiment of the present invention also discloses an image generation device for realizing element replacement, such as Figure 3 As shown, including:

[0160] An input module, used for inputting the subject image and the background image into the trained image generation model according to a given subject image and a background image, wherein the subject image contains pre-selected subject elements that need to be replaced;

[0161] An image processing module, used for the image encoding module in the image generation model to obtain the first eigenvector of the main image, the second eigenvector of the background image and the mask vector of the main element; the first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image;

[0162] An output module, used for outputting a target image after being processed by the image processing module, wherein the target image is an image in which a main element is located in the background image;

[0163] The image generation model is a model for generating images from images that is trained using a pre-constructed training data set.

[0164] The device of this embodiment uses a pre-trained end-to-end model, namely an image generation model, to perform element replacement processing on the main image and the background image, extracts triplet vectors as guiding information, and then realizes element replacement, thereby achieving accuracy in feature recognition, while making element replacement more natural and fast in processing speed.

[0165] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0167] Memory 1303, used for storing computer programs;

[0168] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0169] According to a given subject image and background image, the subject image and the background image are input into a trained image generation model, wherein the subject image contains pre-selected subject elements that need to be replaced;

[0170] The image encoding module in the image generation model obtains a first feature vector of the subject image, a second feature vector of the background image, and a mask vector of the subject element;

[0171] The first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image to output a target image, wherein the target image is an image in which the main element is located in the background image;

[0172] The image generation module is a model for image generation that is trained using a pre-constructed training data set.

[0173] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0174] The communication interface is used for communication between the above terminal and other devices.

[0175] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0176] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0177] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium 1401 is run on a computer, the computer executes an image generation method for implementing element replacement described in the above embodiment.

[0178] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes an image generation method for implementing element replacement described in the above embodiment.

[0179] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0180] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0181] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An image generation method for realizing element replacement, characterized in that: include: According to a given subject image and background image, the subject image and the background image are input into a trained image generation model, wherein the subject image contains pre-selected subject elements that need to be replaced; The image encoding module in the image generation model obtains a first feature vector of the subject image, a second feature vector of the background image, and a mask vector of the subject element; The first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image to output a target image, wherein the target image is an image in which the main element is located in the background image; The image generation module is a model for image generation that is trained using a pre-constructed training data set.

2. The method according to claim 1, characterized in that According to a given subject image and background image, the subject image and background image are input into a trained image generation model, including: If there is text description information corresponding to the target image input by the user, the text description information, the main image and the background image are input into the trained image generation model; Accordingly, the image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image and the mask vector of the main element, including: The image encoding module in the image generation model obtains the first feature vector of the main image, the second feature vector of the background image, and the mask vector of the main element, and the text encoding module in the image generation model obtains the text vector of the text description information; The first feature vector, the second feature vector and the mask vector are fused to generate a fused vector, including: The first feature vector, the second feature vector, the mask vector and the text vector are fused to generate a fused vector.

3. The method according to claim 2, characterized in that The method further comprises: The target image is post-processed to obtain an expected target image.

4. The method according to claim 2, characterized in that: The image encoding module includes: Mask R-CNN or ViT model; The image generation module includes: stable diffusion model or UNet model.

5. The method according to claim 1, characterized in that The method further comprises: Based on different video frames of the video stream, a training data set is constructed; the training data set includes: a plurality of groups of images, each group of images includes: a first main image, a first background image, and a replacement image; The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image without the subject element; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image; According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model; The result image is matched with the first main image of the group of images, and the parameters of the image generation model are adjusted according to the matching result. When the matching result reaches the optimal value, a trained image generation model is obtained.

6. A training method for an image generation model for implementing element replacement, characterized in that: include: Construct a training data set based on different video frames of the video stream; The training data set includes: a plurality of groups of images, each group of images includes: a first main image, a first background image, and a replacement image; The first subject image is an original image corresponding to the video frame and having a pre-selected subject element to be replaced; the first background image is the background of the first subject image with the subject element removed; the replacement image is a target image obtained by replacing the subject element in the first subject image with a random background image; the replacement image has a pre-selected subject element to be replaced; According to each group of images in the training data set, the first background image and the replacement image in each group of images are used as inputs of the image generation model to obtain a result image output by the image generation model; The result image is matched with the first main image of the group of images, and the parameters of the image generation model are adjusted according to the matching result. When the matching result reaches the optimal value, a trained image generation model is obtained.

7. The method according to claim 6, characterized in that The step of using the first background image and the replacement image in each group of images as inputs of the image generation model and obtaining a result image output by the image generation model comprises: The image encoding module in the image generation model obtains a first feature vector of the replacement image, a second feature vector of the background image, and a mask vector of a main element in the replacement image; The first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing based on the vector and the random Gaussian noise image to output a result image.

8. An image generation device for realizing element replacement, characterized in that: include: An input module, used for inputting the subject image and the background image into the trained image generation model according to a given subject image and a background image, wherein the subject image contains pre-selected subject elements that need to be replaced; An image processing module, used for the image encoding module in the image generation model to obtain the first eigenvector of the main image, the second eigenvector of the background image and the mask vector of the main element; the first eigenvector, the second eigenvector and the mask vector are fused to generate a fused vector, and the image generation module in the image generation model performs iterative processing according to the vector and the random Gaussian noise image; An output module, used for outputting a target image after being processed by the image processing module, wherein the target image is an image in which a main element is located in the background image; The image generation model is a model for generating images from images that is trained using a pre-constructed training data set.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images