Image synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining first image data containing a target object, and segmenting the mask data of the target object from the first image data; in response to an image tag input operation during image synthesis of the target object, determining image tag information corresponding to the image tag input operation; and inputting the image label information and the mask data into a target model for synthesizing an image, and outputting a synthesized image taking target image data matched with the image label information as a background and the mask data as a main body. According to the embodiment of the invention, the method achieves the quick synthesis of an image with a brand new background based on the input image tag, shortens the image processing time, and improves the image processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image data processing, and in particular to an image synthesis method and device, electronic equipment, and storage medium. Background Art

[0002] Currently, users can freely create the pictures they need by choosing from a variety of different picture materials, such as combining the user's own image with a background image in a TV series, or combining the user's image in a scene in the TV series.

[0003] However, when synthesizing images, users usually cut out a certain picture manually, and then manually use another picture to stitch the cutout picture. This picture processing process requires users to manually cut out and stitch, and usually requires constant debugging in this process to obtain a satisfactory picture effect. If the user is not satisfied with the current picture after selecting the picture, he needs to reselect the picture and start the cutout and stitching again. This image processing process consumes a lot of time and has low processing efficiency. Summary of the invention

[0004] In view of the above problems, an image synthesis method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of the present invention, a method for image synthesis is proposed, the method comprising:

[0006] Acquire first image data containing a target object, and segment mask data of the target object from the first image data;

[0007] In response to an image tag input operation when performing image synthesis on the target object, determining image tag information corresponding to the image tag input operation;

[0008] The image label information and the mask data are input into a target model for synthesizing an image, and a synthesized image is outputted with the target image data matched with the image label information as a background and the mask data as a main body.

[0009] In a second aspect of the present invention, an image synthesis device is provided, the device comprising:

[0010] A mask data determination module, used for acquiring first image data containing a target object, and segmenting mask data of the target object from the first image data;

[0011] An image tag information determination module, configured to determine, in response to an image tag input operation when performing image synthesis on the target object, image tag information corresponding to the image tag input operation;

[0012] The image synthesis module is used to input the image label information and the mask data into a target model for synthesizing an image, and output a synthesized image with the target image data matched with the image label information as the background and the mask data as the main body.

[0013] In a third aspect of the present invention, an electronic device is proposed, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above image synthesis method is implemented when the computer program is executed by the processor.

[0014] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image synthesis method described above is implemented.

[0015] The embodiments of the present invention have the following advantages:

[0016] In the embodiment of the present invention, by inputting the segmented mask data and the set image label information into the trained target model, a synthetic image with the target image data matching the image label information as the background and the mask data as the main body can be quickly synthesized in the target model, thereby shortening the image processing time and improving the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1a is a flowchart of a method for synthesizing an image provided by an embodiment of the present invention;

[0019] Figure 1b is a schematic diagram of a process for determining sample image data provided by an embodiment of the present invention;

[0020] Figure 1c is a schematic diagram of a training process of a first model provided by an embodiment of the present invention;

[0021] Figure 1d is a schematic diagram of a process for determining sample image data provided by an embodiment of the present invention;

[0022] Figure 1e is a schematic diagram of another process for determining sample image data provided by an embodiment of the present invention;

[0023] Figure 2ais a schematic diagram of a model training process provided by an embodiment of the present invention;

[0024] Figure 2b is a third structural schematic diagram provided by an embodiment of the present invention;

[0025] Figure 3a is a structural schematic diagram of an image synthesis device provided by an embodiment of the present invention;

[0026] Figure 3b is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention; Figure 3c is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention;

[0027] Figure 3d is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention;

[0028] Figure 3e is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention;

[0029] Figure 3f is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention;

[0030] Figure 3g is a schematic structural diagram of another image synthesis device provided by an embodiment of the present invention;

[0031] Figure 3h It is a schematic diagram of the structure of another image synthesis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Reference Figure 1a , shows a flowchart of a method for image synthesis provided by an embodiment of the present invention, which may specifically include the following steps:

[0034] Step 101, acquiring first image data containing a target object, and segmenting mask data of the target object from the first image data;

[0035] In actual applications, when a user needs to synthesize a new image, he or she may select a first image data containing a target object, wherein the target object may be an object that the user plans to use as a subject of the synthesized image, such as a person object.

[0036] After the first image data is determined, the mask data of the target object may be segmented from the first image data using image processing technology.

[0037] In one example, an image segmentation method can be used to segment the image and remove the background; in another example, a deep learning model can be combined to segment the image and remove the background to obtain mask data of the target object, such as a convolutional neural network (CNN), a fully convolutional neural network (FCN), etc. In some embodiments, an online image processing tool can also be called to quickly implement intelligent image processing.

[0038] Step 102, in response to an image tag input operation when performing image synthesis on a target object, determining image tag information corresponding to the image tag input operation;

[0039] On the image synthesis interface, an input box can be provided for users to input image synthesis requirements. Users can enter image tags in the input box and then enter background requirements for the synthesized image. After the user enters the image tag, the image tag information entered by the user is determined.

[0040] The image tag information is used to represent the characteristics of the synthetic image background. The image tag information may be the name of a TV series or a region name in the TV series image data. The image tag information may also be scene elements contained in the TV series image data, such as flowers, grass, trees, buildings, landforms, etc. For example, the image tag information may be a region name or scene element corresponding to a famous scene image in a TV series.

[0041] Step 103: input the image label information and the mask data into a target model for synthesizing an image, and output a synthesized image with the target image data matched with the image label information as the background and the mask data as the main body.

[0042] After determining the image label information and mask data, the image label information and mask data can be input into the trained target model for image synthesis. The target model is an image synthesis model that can quickly synthesize a synthetic image with the target image data matching the image label information as the background and the mask data as the main body.

[0043] In one embodiment of the present invention, Figure 1bAs shown, step 103 may include the following sub-steps:

[0044] Sub-step S11, inputting image label information and mask data into a target model for synthesizing an image;

[0045] In practical applications, after acquiring the image label data and the mask data, the image label data and the mask data may be input into a target model for synthesizing an image.

[0046] Sub-step S12, generating target image data corresponding to the image tag information;

[0047] After the image label information and mask data are input into the target model, the matching target image data can be determined according to the image label information.

[0048] Sub-step S13, determining the target area corresponding to the mask data in the target image data, and replacing the target area of ​​the target image data with the mask data, so as to generate a composite image with the target image data as the background and the mask data as the main body.

[0049] The target image data output by the first model can be synthesized with the mask data, and the synthesis process is: determine the target area corresponding to the mask data from the target image data, and the target area is an area suitable for placing the mask data in the background of the target image data. Specifically, the target area can be the area where the main body in the target image data is located, or it can be a blank area in the target image data.

[0050] In one embodiment of the present invention, the target model includes an image processing layer, such as Outpainting. Outpainting is an application of the Stable Diffusion model. Outpainting allows the user to extend the boundaries of the image, thereby generating the surrounding content of the image. In the image processing layer, the mask redrawing principle can be applied to perform local image drawing, and a partial area can be drawn to synthesize an image. Specifically, the mask redrawing principle can be used to redraw the mask data in the target area of ​​the target image data, thereby generating a synthetic image with the target image data as the background and the mask data as the main body.

[0051] In one embodiment of the present invention, the target model also includes a first model, which can be used to generate target image data corresponding to the image label information. The first model can also include a first module for forward expansion, a second module for generating conditional feature vectors, and a third module for reverse denoising. The third module is composed of a linear layer and a low-rank adaptive structure in parallel, such as Figure 1c As shown, the training step of the first model may include the following steps:

[0052] Step S21, obtaining sample image data for first model training, and determining a sample label corresponding to each sample image data;

[0053] In practical applications, image data of different scenes may be collected in advance as sample image data for training the first model, wherein the image data of different scenes may be screened from various image databases or screened by capturing image frames from various video data.

[0054] After the sample image data is determined, the background of each sample image data can be annotated by analyzing the elements contained in each sample image data to determine the sample label corresponding to each sample image data, wherein the sample label is used to describe the background characteristics of the sample image data, for example, the sample label can be the name of a film or TV series, the name of a region in the image data of a film or TV series, or the scene element contained in the image data of a film or TV series, such as flowers, grass, trees, buildings, landforms, etc. Specifically, the sample label can be the region name or scene element corresponding to the famous scene image in the film or TV series.

[0055] Step S22, in the first module, encoding the sample image data into second image data of a preset dimension, and adding noise to the second image data to generate third image data;

[0056] In the first module, the sample image data can be encoded, and specifically, the sample image data can be encoded according to a preset dimension. It should be noted that the preset dimension can be set according to actual needs, and there is no excessive restriction on this in the embodiment of the present invention. Among them, the dimension of the generated second image data is smaller than the dimension of the sample image data. The encoding process realizes image dimensionality reduction processing and reduces data load, so as to facilitate subsequent processing of image data.

[0057] Among them, a dimensionality reduction algorithm can be set in the encoder used for dimensionality reduction processing, for example, a dimensionality reduction algorithm of principal component analysis (PCA). The image dimensionality reduction processing process in the encoder can include: extracting image features (such as color, texture, shape and other image features), applying the dimensionality reduction algorithm in the encoder to perform dimensionality reduction processing on the extracted features, thereby converting high-dimensional data into low-dimensional data while retaining the main features in the data.

[0058] After the encoding is completed, noise may be added to the second image data generated by the encoding, thereby generating the third image data.

[0059] In one example, a suitable noise type and parameter may be selected as needed, wherein the noise type may include but is not limited to one or more of Gaussian noise and salt and pepper noise. Salt and pepper noise is noise with randomly distributed black and white spots in the image like peppers and salt grains.

[0060] The noise parameters may include any one or more of the mean and variance of Gaussian noise, the density of salt and pepper noise, the intensity and distribution of noise. By setting the noise parameters, it is possible to avoid excessive or insufficient noise affecting the image quality and recognition effect.

[0061] In one example, adding noise to an image can be achieved by calling a function for adding noise in a code library, for example, using a NumPy library to generate noise, and then using an image processing function in an OpenCV library to add noise to the image.

[0062] For example: Gaussian noise: Use NumPy's np.random.normal function to generate Gaussian distributed random numbers, and then add the noise to the image through OpenCV's cv2.add function.

[0063] Salt and pepper noise: Use NumPy’s np.random.rand function to generate random numbers, and then set the pixels to 0 or 255 based on the values ​​of the random numbers.

[0064] Poisson noise: Use NumPy’s np.random.poisson function to generate Poisson-distributed random numbers, and then add the noise to the image via OpenCV’s cv2.add function.

[0065] Uniform noise: Use NumPy’s np.random.uniform function to generate uniformly distributed random numbers, and then add the noise to the image via OpenCV’s cv2.add function.

[0066] Step S23, in the second module, encoding the sample label into a conditional feature vector;

[0067] In the second module, the sample labels of the text type can be encoded into conditional feature vectors that can be recognized and processed by computers. By encoding the sample labels into conditional feature vectors, data processing based on the conditional feature vectors encoded by the sample labels is facilitated in the first model.

[0068] Step S24, inputting the third image data and the conditional feature vector into the third module, and performing denoising processing on the third image data using a linear layer and a low-rank adaptive structure according to the conditional feature vector to generate fourth image data corresponding to the sample label;

[0069] The third image data with noise and the conditional feature vector are input into the third module for denoising, and the fourth image data predicted based on the sample label is obtained to learn the sample label in the first model. Specifically, the denoising of the third image data with noise is to restore the image data, and for the association between the sample image and the sample image data, the conditional feature vector can be used as a condition for denoising optimization.

[0070] Step S25, adjusting target model parameters in the low-rank adaptive structure based on the sample image data and the fourth image data;

[0071] After the sample image data and the fourth image data are obtained, the sample image data is the expected image after the sample label is processed based on the first model, and the fourth image data is the predicted image after the sample label is processed based on the first model.

[0072] By comparing the sample image data and the fourth image data, the target model parameters of the first model can be adjusted in reverse order to make the fourth image data infinitely close to the sample image data.

[0073] Specifically, during the training of the first model, the model parameters of the linear layers in the first module, the second module and the third module can be kept unchanged, and only the target model parameters in the low-rank adaptive structure are trained, so as to obtain the expected model by training a small number of model parameters.

[0074] In practical applications, a preset loss function can be used to calculate a loss value based on the sample image data and the fourth image data, and then the target model parameters in the low-rank adaptive structure are adjusted according to the loss value. The loss function can be set according to the actual scene requirements, and there is no excessive restriction on this in the embodiments of the present invention.

[0075] Step S26, when the loss value corresponding to the target model parameter converges, stop model training and output the first model corresponding to the target model parameter.

[0076] In practical applications, the loss value corresponding to the training result can be used as a condition to determine whether the model needs to stop training. After each training of the first model, the loss value corresponding to the target model parameters obtained by training is calculated. And it is determined whether the loss value converges. If it is determined that the loss value converges, the model training can be stopped. The first model corresponding to the current target model parameters is output. If the loss value does not converge, iterative training is performed based on the target model obtained by the current training until the loss value of the target model parameters obtained by training converges.

[0077] In one example, the target model may be an Outpainting model (image extension model) of a Stablediffusion model, that is, the target model includes a Stablediffusion model (i.e., the first model in the embodiment of the present invention) and an Outpainting layer (i.e., the image processing layer in the embodiment of the present invention). In the embodiment of the present invention, a Lora (Low-Rank Adaptation) structure may be connected in parallel to the linear layer in the CLIP (Contrastive Language-Image Pre-training) model in the Stablediffusion model and the cross-attention layer of the U-net (i.e., the QKV part of the U-Net noise predictor), and the Lora structure may include a parallel A matrix and a B matrix. When the entire Stablediffusion model is trained, the Stablediffusion model freezes the model parameters except the Lora structure, that is, keeps some model parameters unchanged, and only adjusts the target model parameters of the Lora structure to perform the Stablediffusion model training. This training method requires less training data, reduces the complexity of data calculation, and quickly achieves optimization.

[0078] Specifically, during the model training process, the matrix elements of the A matrix and the B matrix are updated by updating the model parameters, thereby updating the Stablediffusion model.

[0079] In one example, the first module for forward expansion in the Stablediffusion model includes an encoder and a forward expansion submodule, and the second module includes a Conditioning module. After the sample label is input into the Stablediffusion model, the encoder can be used to encode it into the second image data of the preset dimension. After the second image data is randomly added with noise by the forward expansion submodule, the third image data is generated. After the sample label is input into the Stablediffusion model, the Conditioning module can be used to encode the text information that the model cannot recognize and calculate into a conditional feature vector, and then the third image data and the conditional feature vector can be input into the third module for denoising. In the third module, the QKV linear layer is connected in parallel with the Lora structure to denoise the image data according to the optimization condition to restore the image, wherein the optimization condition is the conditional feature vector, so that the association relationship between the sample label and the target image data can be established, that is, the Stablediffusion model learns the association relationship between the sample label and the target image data, and the trained Stablediffusion model can generate matching target image data based on the image label input by the user.

[0080] In one embodiment of the present invention, Figure 1d As shown, step S21 may include the following sub-steps:

[0081] Sub-step S31, obtaining initial image data for synthesizing the image background;

[0082] In practical applications, the initial image data may be images of famous scenes in movies or TV series, and the initial image data may also be captured from a web page according to user needs.

[0083] Sub-step S32, obtaining a target video including a synthetic image background, and extracting frames from the target video to generate a plurality of video image data;

[0084] The target video is a video containing famous scene images, and the target video can be captured from a film and television resource library for storing film and television resources or a social networking site.

[0085] A preset period may be set for extracting frames for the target video, for example, extracting one frame every five frames, thereby obtaining an image set consisting of a plurality of video image data.

[0086] Sub-step S33, determining candidate image data matching the initial image data from the plurality of video image data;

[0087] By matching the initial image data with the video image data, matching candidate image data can be quickly found from the target video, and the amount of sample data that meets the conditions can be rapidly expanded.

[0088] Sub-step S34, determining sample image data for performing first model training based on the candidate image data.

[0089] In one embodiment of the present invention, sub-step S33 may specifically include the following process: calculating the similarity between each video image data and the initial image data; when the similarity between the target video image data among multiple video image data and the initial image data is greater than a preset similarity threshold, determining that the target video image data matches the initial image data.

[0090] In practical applications, a preset similarity algorithm may be applied to calculate the similarity between images, wherein the similarity algorithm may be a proxyAnchor algorithm (proxy anchor algorithm).

[0091] In one embodiment of the present invention, Figure 1e As shown, sub-step S34 may include the following sub-steps:

[0092] Sub-step S41, determining fifth image data from the candidate image data according to a preset image screening rule;

[0093] The image screening rules are composed of any one or more of image quality, target type of objects, and positional relationships between different types of objects.

[0094] In actual applications, after obtaining multiple candidate image data, the candidate image data can be screened according to the screening rules to determine the fifth image data. During the screening process, a quality score can be performed for each candidate image data based on the image screening rules to screen out high-quality fifth image data.

[0095] Multiple image evaluation indicators can be set in the image screening rules, and then a conversion relationship between each image evaluation indicator and the quality score can be set. Based on the conversion relationship, the quality score of each candidate image data can be determined, and then the fifth image data can be screened from high to low according to the score.

[0096] For example, image evaluation indicators can be image quality, target type objects, and positional relationships between different types of objects, where image quality involves image resolution. The higher the resolution, the higher the image quality and the higher the corresponding score. The higher the number of target type objects (objects related to the subject), the more complex the image processing process, and the greater the number of target type objects, the lower the corresponding score. The positional relationship between different types of objects: the closer the positional relationship between different types of objects, the more difficult it is to segment each type of object, and the lower the corresponding score.

[0097] Sub-step S42, pre-processing the fifth image data to generate sample image data for first model training.

[0098] After obtaining the fifth image data, the fifth image data can be preprocessed to make the fifth image data more suitable as the background of the synthesized image. For example, when the subject of the image to be synthesized is a person and the background is a famous scene, for the screened famous scene image, icons and texts in the famous scene image can be removed, and preprocessing operations such as removing the person can be performed to save the complete famous scene background.

[0099] In one embodiment of the present invention, sub-step S42 may specifically include: obtaining a preset image restoration algorithm; using the preset image restoration algorithm to remove image data in the fifth image data that is not compatible with the target object, and generating sample image data for first model training.

[0100] In practical applications, the preprocessing process of the target image data is image restoration processing, and a preset image restoration algorithm can be applied to remove image data incompatible with the target object in the fifth image data to achieve image restoration processing. Among them, the image restoration processing algorithm in the embodiment of the present invention can be the inpaint algorithm (image restoration algorithm) of Stablediffusion, and the incompatible image data can include logos, texts and objects of the target object type in the image.

[0101] In one example, the image restoration algorithm can perform image restoration according to preset restoration rules. The preset restoration rules can be determined based on preset areas, preset shapes, and preset types of objects in the image. The preset areas, preset shapes, and preset types can be set according to actual scenes, and no excessive restrictions are imposed on this in the embodiments of the present invention.

[0102] Specifically, when the preset repair rule is to clear any one or more of the preset area, preset shape, and preset type of objects in the fifth image data, the preset area, preset shape, and preset type of objects can be determined according to the image repair algorithm, and the part of the image data can be cleared to obtain sample image data.

[0103] In the image restoration algorithm, the user can input a prompt or select a provided prompt tone, and the image element corresponding to the prompt is generated in the image element clearing area.

[0104] In an embodiment of the present invention, first image data containing a target object can be obtained, and mask data of the target object can be segmented from the first image data; in response to an image label input operation when performing image synthesis on the target object, image label information corresponding to the image label input operation is determined; the image label information and the mask data are input into a target model for synthesizing the image, and a synthesized image with the image data matching the image label information as the background and the mask data as the main body is output, thereby realizing rapid synthesis of an image with a completely new background based on the input image label, shortening image processing time, and improving image processing efficiency.

[0105] In an embodiment of the present invention, a Stablediffusion model with a linear layer parallel LoRA structure can be applied to realize the generation of synthetic images of famous scenes from film and television dramas. The Stablediffusion model can be used to realize the synthesis of synthetic images with famous scenes from film and television dramas corresponding to the image labels as the background and the target objects in the original image as the main body based on the image labels input by the user and the mask data of the original image, thereby realizing the rapid generation of synthetic images with completely new backgrounds and improving image processing efficiency.

[0106] The Stable diffusion model in the embodiment of the present invention may be a model obtained by training and fine-tuning the LoRA structure based on a pre-trained Stable diffusion model.

[0107] Reference Figure 2a , shows a schematic diagram of a Stablediffusion model training process in an embodiment of the present invention, the Stablediffusion model includes a first module consisting of an encoder 1 and a forward diffusion process, a second module and a third module, wherein the second module is a conditioning module, including an encoder 2, the encoder is a directional encoder, and can realize encoding of input text and / or semantic maps. The third module is used to realize reverse denoising, and the third module may include a QKV linear layer and a LoRA structure connected in parallel with the linear layer, and the LoRA structure may be composed of an A matrix and a B matrix, such as Figure 2b FIG. 1 is a schematic diagram of the structure of a third module in an embodiment of the present invention.

[0108] The Stablediffusion model training process includes the following steps:

[0109] (I) Pre-processing of training data:

[0110] Select the image 1 of a specific famous scene in the film and television drama A, and obtain the video clip of the film and television drama A on the film and television drama database or social network; use the proxyAnchor algorithm to match the image 1 with the video frame image 2 extracted from the video clip, and calculate the similarity s (where 0 < s < 1). When s is greater than the set threshold (such as 0.8), it is determined that the video frame image 2 is the image of the specific famous scene in the film and television drama A.

[0111] According to the above matching method, multiple famous scene images are obtained, and all the famous scene images are scored and screened based on the preset image screening rules. For the selected famous scene images, use the image restoration algorithm to remove the icons, text, and interfering human objects in the images, and then add sample labels to the restored images.

[0112] (2) Stablediffusion training process:

[0113] During the training process of the Stablediffusion model, the model parameters of the first module and the second module, as well as the model parameters of the QKV linear layer in the reverse denoising process of the third module, are fixed, and only the LoRA structure in the third model is trained.

[0114] Take the image obtained in the process (1) as the original image and input it into the encoder 1 for dimensionality reduction encoding to obtain the image 3. Add random noise to the image 3 to perform the forward diffusion process to obtain the image 4.

[0115] Input the sample label in the process (1) into the conditional module, and the encoder 2 can be used for encoding in the conditional module to obtain the conditional feature vector.

[0116] Input the image 4 and the conditional feature vector into the U-Net (U-shaped network) to perform the reverse denoising process. Control the denoising process of the U-net for the image 4 through the conditional feature vector to generate an image matching the text, and obtain the predicted image 5.

[0117] Use the preset loss function to calculate the loss function loss of the predicted image 5 and the image 4, and then backpropagate the loss function loss into the U-Net to fine-tune the target model parameters in the LoRA structure of the U-Net. If the loss function loss (i.e., the loss value) of this training converges, the model training ends. If the loss function loss of this training does not converge, iterative training is performed based on the target model parameters until the loss function loss converges, and then the model training ends, and the Stablediffusion model corresponding to the target model parameters at the end of the model training is output.

[0118] (3) Stablediffusion model application process:

[0119] Get the person photo uploaded by the user, and segment the person's head through the automatic matting method (matting) to obtain the mask data of the person's head.

[0120] In response to the label text of the movie or TV drama name scene image input by the user, the mask and the label text are input into the Outpainting model of Stablediffusion;

[0121] After the label text is converted into a conditional feature vector using a conditionalization module, the TV drama name scene images matching the conditional feature vector can be queried in the TV drama database.

[0122] In the Outpainting layer, the famous scene images of film and television dramas are combined with the mask data using the mask repainting principle to obtain a composite image with the mask data as the main body and the famous scene images as the background.

[0123] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0124] Reference Figure 3a , shows a schematic diagram of the structure of an image synthesis device provided by an embodiment of the present invention, which may specifically include the following modules:

[0125] The mask data determination module 301 is used to obtain first image data containing a target object and segment mask data of the target object from the first image data;

[0126] An image tag information determination module 302 is used to respond to an image tag input operation when performing image synthesis on a target object and determine image tag information corresponding to the image tag input operation;

[0127] The image synthesis module 303 is used to input the image label information and the mask data into the target model for synthesizing the image, and output a synthesized image with the image data matched with the image label information as the background and the mask data as the main body.

[0128] In one embodiment of the present invention, Figure 3b As shown, the image synthesis module 303 may include:

[0129] A data input submodule 11, used for inputting the image label information and the mask data into a target model for synthesizing an image;

[0130] A target image data generating submodule 12 is used to generate target image data corresponding to the image tag information;

[0131] The synthetic image submodule 13 is used to determine the target area corresponding to the mask data in the target image data, and replace the target area of ​​the target image data with the mask data to generate a synthetic image with the target image data as the background and the mask data as the main body.

[0132] In one embodiment of the present invention, the target model includes a first model, the first model includes a first module for forward expansion, a second module for generating a conditional feature vector, and a third module for reverse denoising, the third module is composed of a linear layer and a low-rank adaptive structure in parallel, such as Figure 3c As shown, the device also includes:

[0133] A sample label determination module 304 is used to obtain sample image data for first model training and determine a sample label corresponding to each sample image data;

[0134] The encoding and noise module 305 is used to encode the sample image data into second image data of a preset dimension in the first module, and add noise to the second image data to generate third image data;

[0135] A conditional feature vector generating module 306, configured to encode the sample label into a conditional feature vector in the second module;

[0136] A denoising module 307, configured to input the third image data and the conditional feature vector into the third module, and perform denoising processing on the third image data using the linear layer and the low-rank adaptive structure according to the conditional feature vector to generate fourth image data corresponding to the sample label;

[0137] A model parameter adjustment module 308, configured to adjust target model parameters in the low-rank adaptive structure based on the sample image data and the fourth image data;

[0138] The model training termination module 309 is used to stop the model training when the loss value corresponding to the target model parameter converges, and output the first model corresponding to the target model parameter.

[0139] In one embodiment of the present invention, Figure 3d As shown, the model parameter adjustment module 308 may include:

[0140] A loss value determination submodule 21, configured to determine a loss value based on the sample image data and the fourth image data;

[0141] The parameter adjustment submodule 22 is used to adjust the target model parameters in the low-rank adaptive structure according to the loss value.

[0142] In one embodiment of the present invention, Figure 3e As shown, the sample label determination module 304 includes:

[0143] The initial image data acquisition submodule 31 is used to acquire initial image data for synthesizing the image background;

[0144] The video image data generating submodule 32 is used to obtain a target video including a synthetic image background, and extract frames from the target video to generate a plurality of video image data;

[0145] An image matching submodule 33, used to determine candidate image data matching the initial image data from a plurality of video image data;

[0146] The sample image matching submodule 34 is used to determine sample image data for first model training based on the candidate image data.

[0147] In one embodiment of the present invention, Figure 3f As shown, the image matching submodule 33 includes:

[0148] A similarity calculation unit 41, used to calculate the similarity between each video image data and the initial image data;

[0149] The image matching unit 42 is configured to determine that the target video image data matches the initial image data when the similarity between the target video image data among the multiple video image data and the initial image data is greater than a preset similarity threshold.

[0150] In one embodiment of the present invention, Figure 3g As shown, the sample image matching submodule 34 includes:

[0151] a target image data determining unit 43, configured to determine fifth image data from the candidate image data according to a preset image screening rule, wherein the image screening rule is composed of any one or more of image quality, an object of a target type, and a positional relationship between objects of different types;

[0152] The sample image data determining unit 44 pre-processes the fifth image data to generate sample image data for first model training.

[0153] In one embodiment of the present invention, Figure 3h As shown, the sample image data determination unit 44 includes:

[0154] An image restoration algorithm acquisition subunit 51 is used to acquire a preset image restoration algorithm;

[0155] The preprocessing subunit 52 is used to use the preset image restoration algorithm to remove the image data in the fifth image data that is not compatible with the target object, and generate sample image data for first model training.

[0156] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above image synthesis method when executed by the processor.

[0157] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above image synthesis method is implemented.

[0158] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0159] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0160] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0161] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0162] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0164] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0166] The above provides a detailed introduction to an image synthesis method and device, electronic device, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image synthesis method, It is characterized in that The method comprises: Acquire first image data containing a target object, and segment mask data of the target object from the first image data; In response to an image tag input operation when performing image synthesis on the target object, determining image tag information corresponding to the image tag input operation; The image label information and the mask data are input into a target model for synthesizing an image, and a synthesized image is outputted with the target image data matched with the image label information as a background and the mask data as a main body.

2. The method according to claim 1, It is characterized in that The step of inputting the image label information and the mask data into a target model for synthesizing an image, and outputting a synthesized image with the target image data matched with the image label information as a background and the mask data as a main body, comprises: Inputting the image label information and the mask data into a target model for synthesizing an image; Generate target image data corresponding to the image tag information; A target area corresponding to the mask data is determined in the target image data, and the mask data replaces the target area of ​​the target image data to generate a synthetic image with the target image data as a background and the mask data as a main body.

3. The method according to claim 2, It is characterized in that The target model includes a first model, the first model is used to generate target image data corresponding to the image label information, the first module includes a first module for forward expansion, a second module for generating conditional feature vectors, and a third module for reverse denoising, the third module is composed of a linear layer and a low-rank adaptive structure in parallel, and the training steps of the first model include: Obtain sample image data for first model training, and determine a sample label corresponding to each sample image data; In the first module, the sample image data is encoded into second image data of a preset dimension, and noise is added to the second image data to generate third image data; In the second module, the sample label is encoded into a conditional feature vector; Inputting the third image data and the conditional feature vector into the third module, and performing denoising processing on the third image data using the linear layer and the low-rank adaptive structure according to the conditional feature vector to generate fourth image data corresponding to the sample label; Adjusting target model parameters in the low-rank adaptive structure based on the sample image data and the fourth image data; When the loss value corresponding to the target model parameter converges, the model training is stopped and the first model corresponding to the target model parameter is output.

4. The method according to claim 3, It is characterized in that The adjusting the target model parameters in the low-rank adaptive structure based on the sample image data and the fourth image data includes: determining a loss value based on the sample image data and the fourth image data; The target model parameters in the low-rank adaptive structure are adjusted according to the loss value.

5. The method according to claim 3, It is characterized in that The obtaining of sample image data for performing first model training includes: Acquire initial image data for synthesizing an image background; Acquire a target video containing the synthetic image background, and extract frames from the target video to generate a plurality of video image data; Determining candidate image data matching the initial image data from the plurality of video image data; Sample image data for performing first model training is determined based on the candidate image data.

6. The method according to claim 3, It is characterized in that The step of determining candidate image data matching the initial image data from the video image data comprises: Calculating the similarity between each video image data and the initial image data; When the similarity between the target video image data among the multiple video image data and the initial image data is greater than a preset similarity threshold, it is determined that the target video image data matches the initial image data.

7. The method according to claim 5, It is characterized in that The determining, based on the candidate image data, sample image data for performing first model training comprises: Determine fifth image data from the candidate image data according to a preset image screening rule, wherein the image screening rule is composed of any one or more of image quality, target type of object, and positional relationship between different types of objects; The fifth image data is preprocessed to generate sample image data for first model training.

8. The method according to claim 7, It is characterized in that The preprocessing of the fifth image data to generate sample image data for first model training includes: Get the preset image restoration algorithm; The preset image restoration algorithm is used to remove image data incompatible with the target object in the fifth image data, so as to generate sample image data for first model training.

9. An image synthesis device, It is characterized in that The device comprises: A mask data determination module, used for acquiring first image data containing a target object, and segmenting mask data of the target object from the first image data; An image tag information determination module, configured to determine, in response to an image tag input operation when performing image synthesis on the target object, image tag information corresponding to the image tag input operation; The image synthesis module is used to input the image label information and the mask data into a target model for synthesizing an image, and output a synthesized image with the target image data matched with the image label information as the background and the mask data as the main body.

10. An electronic device, It is characterized in that The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the image synthesis method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image synthesis method according to any one of claims 1 to 8 is implemented.

Images