Virtual group photo generation method and device
By using portrait segmentation algorithm and diffusion model in virtual photo generation, the characters and background are naturally integrated, and the problems of unnatural characters and backgrounds in the existing technology and the need to manually select positions are solved, achieving high naturalness and convenient virtual photo collection generate.
Patent Information
- Application Number
- CN202311481486.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-31
- Filing Date
- 2023-11-08
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, when generating virtual photos, the characters and background are not natural enough, and users need to manually select the positions of the characters, which limits the limitations of the application.
By receiving images and background images containing characters, extracting portraits using portrait segmentation algorithm, and inputting portraits and background images into pre-trained diffusion model, obtaining virtual photos of portraits and background images fused by the diffusion model output.
The natural fusion of characters and background is realized, making the characters and background in the generated virtual photos more natural, and there is no need for the user to manually select the position of the characters, which improves the convenience and naturalness of the application.
Smart Images
Figure CN119444553A_ABST
Abstract
Description
[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on July 31, 2023, with application number 202310948144.9 and application name “A method and device for generating a virtual group photo”, the entire contents of which are incorporated by reference in this application. Technical Field
[0002] The present application relates to the technical field of image generation or image synthesis, and in particular to a method and device for generating a virtual group photo. Background Art
[0003] Virtual group photo is a technology that virtually combines the characters in one image with the background in another image. Virtual group photo can be used in entertainment works such as movies, TV series, games, etc. to make virtual characters more realistic and enhance the audience experience. At the same time, it also provides creators with more freedom and creative possibilities to create unprecedented character images. In addition, virtual group photo can also be applied to other scenarios.
[0004] The synthesis process commonly used in virtual group photos in related technologies can be expressed by the formula: I = α*F + (1-α) * B, where F is the pixel value of the pixel point of the portrait, α is the corresponding weight, and B is the pixel value of the pixel point of the background image. The current virtual group photo technology has some defects. First, when the color of the portrait and the background image are inconsistent, there will be an obvious sense of abruptness and floating of the portrait after synthesis, that is, the person and the background in the virtual group photo are abrupt and unnatural; second, it can be seen from the above formula that the size of the portrait and the background image must be consistent. Although the image size can be ignored by using layer overlay, this also requires the user to manually select the position of the person, which also increases the limitation of the application. Summary of the invention
[0005] The embodiments of the present application provide a method and device for generating a virtual group photo, which are used to solve the problem in the prior art that when generating a virtual group photo, the characters and the background are not natural enough and the user needs to manually select the position of the characters.
[0006] In a first aspect, an embodiment of the present application provides a method for generating a virtual group photo, the method comprising:
[0007] Receiving an image including a person and a background image;
[0008] Extracting the portrait in the image by using a portrait segmentation algorithm;
[0009] The portrait and the background image are input into a pre-trained diffusion model, and a virtual photo output by the diffusion model is obtained by merging the portrait and the background image.
[0010] In a second aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of the virtual group photo generation method as described in any one of the above items when executing a computer program stored in the memory.
[0011] In the embodiment of the present application, the electronic device receives an image containing a person and a background image; extracts the person in the image through a portrait segmentation algorithm; inputs the person and the background image into a pre-trained diffusion model, and obtains a virtual photo after the person and the background image are fused. The diffusion model can well achieve the fusion of the person and the background, so that the person and the background in the generated virtual photo are more natural, and there is no need for the user to manually select the position of the person. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0013] Figure 1 A schematic diagram of a process of generating a virtual group photo provided in an embodiment of the present application;
[0014] Figure 2 A schematic diagram of a portrait provided in an embodiment of the present application;
[0015] Figure 3 A schematic diagram of a detailed process of generating a virtual group photo provided in an embodiment of the present application;
[0016] Figure 4 A schematic diagram of a detailed process of generating a virtual group photo provided in an embodiment of the present application;
[0017] Figure 5 A schematic diagram of a noise adding process provided in an embodiment of the present application;
[0018] Figure 6 A schematic diagram of a noise reduction process provided by an embodiment of the present application;
[0019] Figure 7 A schematic diagram of generating a sample background image provided by an embodiment of the present application;
[0020] Figure 8 A schematic diagram of a training process provided in an embodiment of the present application;
[0021] Fig. 9 A schematic diagram of the structure of a virtual group photo generating device provided in an embodiment of the present application;
[0022] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] The present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0024] In order to generate a virtual photo with a natural background and a person, the embodiment of the present application provides a method and device for generating a virtual photo.
[0025] The virtual group photo generation method includes: receiving an image containing a person and a background image; extracting the person in the image through a portrait segmentation algorithm; inputting the person and the background image into a pre-trained diffusion model to obtain a virtual group photo output by the diffusion model after the person and the background image are fused. Thus, a virtual group photo is generated accurately and effectively.
[0026] Figure 1 A schematic diagram of a process of generating a virtual group photo provided in an embodiment of the present application, the process comprising the following steps:
[0027] S101: receiving an image containing a person and a background image.
[0028] The virtual group photo generation method provided in the embodiment of the present application is applied to an electronic device, which may be an intelligent device such as a terminal, a PC or a server.
[0029] In order to generate a virtual group photo, the electronic device may first receive an image containing a person and a background image. Specifically, when a user needs to generate a virtual group photo, he or she may select an image containing a person and a background image on a preset page of the device used by the user, and click a preset button, such as a "generate" button. The device used by the user may send the image containing a person and the background image to the electronic device, and the electronic device may receive the image containing a person and the background image.
[0030] S102: extracting the portrait in the image by using a portrait segmentation algorithm.
[0031] After receiving an image containing a person, the electronic device may use a portrait segmentation algorithm to extract the portrait from the image containing the person. The portrait may be a portrait mask image, which may be used to mark a foreground area and a background area in the image containing the person, wherein the foreground area is the area where the person in the image is located. Specifically, how the electronic device extracts a portrait from an image using a portrait segmentation algorithm is a prior art and will not be described in detail herein.
[0032] Figure 2 A schematic diagram of a portrait provided in an embodiment of the present application.
[0033] Depend on Figure 2 It can be seen that the person and the background in the portrait can be distinguished.
[0034] S103: Inputting the portrait and the background image into a pre-trained diffusion model, and obtaining a virtual photo output by the diffusion model in which the portrait and the background image are fused.
[0035] In order to accurately generate a virtual group photo, the electronic device locally stores a pre-trained diffusion model. After obtaining the portrait, the electronic device can input the portrait and the background image into the pre-trained diffusion model, and obtain the output of the diffusion model. The output of the diffusion model is the virtual group photo after the portrait and the background image are fused. The embodiment of the present application is equivalent to regenerating the background area of the image based on the diffusion model to achieve the synthesis of the portrait and the new background image. Among them, the diffusion model is a generative algorithm. Among them, the electronic device can also input the image containing the person into the diffusion model together.
[0036] It should be noted that the electronic device is equivalent to using the foreground portrait after portrait segmentation as the base image and the new background image as a condition, and generates an image of the person in the image in the new background through a diffusion model, that is, a virtual photo.
[0037] In an embodiment of the present application, a method for realizing a virtual group photo using image generation based on portrait segmentation and diffusion model is used. Compared with the existing virtual group photo method, this method no longer uses image synthesis to synthesize the background, but uses the characteristic of the diffusion model that can generate a new image based on the base image and the reference image to realize the generative virtual group photo technology based on the diffusion model. In this method, the portrait segmentation algorithm is used to distinguish the portrait area and the background area in the original image. First, based on the result of the segmentation, the background area of the original image is removed to obtain the portrait. Then the portrait and the new background image are used as the input of the diffusion model, where the portrait is used as the base image and the background image is used as the reference image (the conditions of the diffusion model, such as certain styles and certain color biases), and the "missing" background in the portrait is regenerated according to the style of the background image, and finally a virtual group photo of the portrait in the style of the background image is obtained.
[0038] Figure 3 A schematic diagram of a detailed process of generating a virtual group photo provided in an embodiment of the present application.
[0039] Depend on Figure 3It can be seen that the electronic device performs portrait segmentation on the image containing the person to obtain the portrait, inputs the portrait and the background image into the diffusion model, and obtains a virtual photo of the person in the new background output by the diffusion model.
[0040] It should be noted that the background of the final generated virtual group photo is not completely consistent with the background in the background image, they are only similar in style or certain elements. This solution uses the portrait after portrait segmentation as the base image, and generates the background of the portrait according to the style of the background image. Therefore, even when the portrait segmentation algorithm has the phenomenon of mis-segmentation and wrong segmentation, in the stage of virtual group photo generation, the diffusion model will still be regenerated based on the base image content and the style of the new background image. Compared with the portrait segmentation and image synthesis method, this solution can effectively reduce the impact of the segmentation algorithm accuracy. And the foreground portrait and the new background image do not need to be the same size. The solution involved in this application is real-time and reliable, and meets the trustworthy characteristics.
[0041] In the embodiment of the present application, the electronic device receives an image containing a person and a background image; extracts the person in the image through a portrait segmentation algorithm; inputs the person and the background image into a pre-trained diffusion model, and obtains a virtual photo after the person and the background image are fused. The diffusion model can well achieve the fusion of the person and the background, so that the person and the background in the generated virtual photo are more natural, and there is no need for the user to manually select the position of the person.
[0042] In order to accurately generate a virtual group photo, based on the above embodiment, in the embodiment of the present application, the step of obtaining the virtual group photo after the image output by the diffusion model is fused with the background image includes:
[0043] The network layer of the diffusion model generates a random noise image; obtains the remaining number of repetitions saved in advance, and generates an embedding vector corresponding to the background image; and loops through the following steps until there are no remaining repetitions, and outputs the noise image as a virtual photo:
[0044] Input the noise image, the embedding vector, the portrait and the vector corresponding to the remaining number of repetitions into a diffusion network Unet pre-trained in the diffusion model, and the Unet outputs a predicted noise image;
[0045] The predicted noise image is used to replace the noise image, and the remaining number of repetitions is updated.
[0046] In an embodiment of the present application, the network layer of the diffusion model generates a random noise image, wherein, specifically, the network layer of the diffusion model can randomly generate a Gaussian noise image of the same size as the image. And obtain the remaining number of repetitions saved in advance, wherein the larger the remaining number of repetitions, the more iterations are required. Generally speaking, the quality of the generated virtual group photo is also higher, but it also makes the virtual group photo more time-consuming to generate. Under the premise of comprehensively considering the effect and time, the value range of the remaining number of repetitions is set to 50-100. The electronic device also generates an embedding vector corresponding to the background image. Specifically, the electronic device uses the Contrastive Language-Image Pretraining (CLIP) algorithm to process the background image and generate an embedding vector.
[0047] The network layer of the diffusion model loops through the following steps until there are no more repetitions left:
[0048] The network layer of the diffusion model inputs the noise image, the embedding vector, the portrait, and the vector corresponding to the remaining number of repetitions into the Unet pre-trained in the diffusion model to obtain the predicted noise image output by the Unet; the network layer of the diffusion model replaces the noise image with the predicted noise image and updates the remaining number of repetitions. The remaining number of repetitions can also be called a moment, and the vector corresponding to the remaining number of repetitions is an embedding vector. For example, when the remaining number of repetitions is 14, the corresponding vector is [0.2, 0.4, ..., 0.02], and the length of the vector is generally 1280.
[0049] When there are no remaining repetitions, the diffusion model determines the acquired predicted noise image as the target image and outputs the target image as a virtual group photo. The final result will generate a synthesized virtual group photo of the same size as the original image containing the person after image decoding.
[0050] Figure 4 A schematic diagram of a detailed process of generating a virtual group photo provided in an embodiment of the present application.
[0051] Figure 4 The image containing the person is I d , the background image is I b The electronic device performs portrait segmentation on the image containing the person and obtains the portrait I s The electronic device inputs the background image and the portrait into the diffusion model, and the diffusion model generates a background image I b Corresponding embedding vector, the diffusion model generates a random noise image X t, the embedding vector and the vector corresponding to the remaining number of repetitions t are input into Unet to obtain the predicted noise E output by Unet. The diffusion model replaces the noise image X with the noise E t , and update the remaining number of repetitions. Figure 4 The area selected in the middle is iterated t times. Finally, the target image is output as a virtual photo.
[0052] In order to accurately obtain the virtual group photo, based on the above embodiments, in the embodiment of the present application, the Unet is trained in the following manner:
[0053] Obtain a pre-saved image of any person, and add any preset noise to the image; input the image after adding noise, the vector corresponding to any number of times, the person corresponding to the image, and the embedding vector corresponding to a background image into Unet, and obtain the predicted noise output by the Unet;
[0054] The Unet is trained according to the predicted noise and the preset noise.
[0055] In order to train Unet, the electronic device can first obtain a pre-saved image of any person, add any preset noise to the image, and input the image after adding noise, the vector corresponding to any number of times, the image obtained after portrait segmentation, and the embedding vector corresponding to a background image into Unet, obtain the predicted noise output by Unet, and train Unet according to the predicted noise and the preset noise. Specifically, the electronic device can calculate the loss based on the cross entropy of the predicted noise distribution and the actual added noise distribution, and train Unet according to the loss. Specifically, the electronic device can continuously change the image of the person, iterate the training, and minimize the loss.
[0056] The bottom layer of the diffusion model is a Unet, whose main task is to predict the noise in the image. In the actual scenario, during the training phase, the diffusion model adds noise to the image x0 at time t0 to obtain x1, and lets Unet predict the noise added in the process of obtaining x1 from x0, and then continuously transforms the sample x0-x t , until the loss between the predicted output noise and the actual added noise is minimized, and the network converges. In the inference phase, given the noise sample x at time t, t , predict the Gaussian noise added at time t through the Unet network, and then let x t Subtract the predicted noise to get x t-1 Repeat the above process to continue x t-1 Get x t-2, and so on, we finally get image x0. Among them, the extended model also supports conditions and base maps as input. For base maps, such as hand-drawn and draft maps, it means generating based on these base maps; for conditions, such as: cartoon style, realistic style, or red flowers in the picture, these conditions are usually generated by CLIP algorithm Embedding vectors are then combined with noise samples x t The image is input into Unet together with the base map, and the final image generated will be the base map with conditional elements added.
[0057] Figure 5 A schematic diagram of a noise adding process provided in an embodiment of the present application.
[0058] Depend on Figure 5 It can be seen that when adding noise, noise is added to the image in turn to obtain the image x t-1 , followed by image x t-1 Add noise q(x t |x t-1 ) to get the image x t , then add noise to the image in turn to get the image x T .
[0059] Figure 6 A schematic diagram of a noise reduction process provided in an embodiment of the present application.
[0060] Depend on Figure 6 It can be seen that when reducing noise, the image x T Subtract the noise and get the image x t , followed by image x t Subtract the noise q(x t-1 |x t ) to get the image x t-1 , and then subtract the noise from the image to get the image x0. Figure 6 It can be seen that this process can obtain a clear image.
[0061] In order to obtain a trained diffusion model, based on the above embodiments, in the embodiment of the present application, the diffusion model is trained in the following manner:
[0062] Obtain any sample group photo in the sample set, and a sample portrait of a person and a sample background image of the background saved for the sample group photo;
[0063] Inputting the sample portrait and the sample background image into the original diffusion model to obtain a predicted group photo output by the original diffusion model;
[0064] The original diffusion model is trained according to the predicted group photo and the sample group photo.
[0065] In order to train the diffusion model, a sample set for training is stored in the embodiment of the present application, and the sample set includes a sample group photo, and a sample portrait of a person and a sample background image of the background stored for the sample group photo. In order to train the diffusion model, the electronic device can obtain any sample group photo in the sample set, and obtain a sample portrait of a person and a sample background image of the background stored for the sample group photo.
[0066] In an embodiment of the present application, after obtaining any sample group photo in the sample set and the sample portrait and sample background image corresponding to the sample group photo, the electronic device can input the sample portrait set and the sample background image into the original diffusion model, and the original diffusion model outputs the predicted group photo.
[0067] After the original diffusion model outputs the predicted group photo, the original diffusion model is trained according to the sample group photo and the predicted group photo to obtain a trained diffusion model.
[0068] In order to accurately generate a virtual group photo, based on the above embodiments, in the embodiment of the present application, a sample portrait and a sample background image corresponding to a sample group photo are obtained by the following method:
[0069] Extracting a sample portrait from the sample group photo by using a portrait segmentation algorithm, and confirming the sample portrait as the sample portrait;
[0070] The sample portrait and the sample group photo are processed by using an image repair algorithm to generate a sample background image.
[0071] The electronic device can extract the portrait in the sample group photo through a portrait segmentation algorithm, and the portrait is the sample portrait. After obtaining the sample portrait, the electronic device can use an image repair algorithm to process the sample portrait and the sample group photo to generate a sample background image. Among them, the image repair algorithm can repair the missing content (or holes) in the image. The image repair algorithm in the embodiment of the present application can be an open source image repair (Lama) algorithm, the input of which is a sample group photo and a sample portrait obtained by portrait segmentation, and the output is an image in which the portrait in the sample group photo is removed and the removed area is filled, and the image is a pure background image without a portrait.
[0072] Figure 7 A schematic diagram of generating a sample background image provided in an embodiment of the present application.
[0073] Depend on Figure 7 It can be seen that after obtaining the sample image and the portrait, a pure background image can be obtained through the Lama algorithm, that is, the sample background image described in the embodiment of the present application.
[0074] In order to accurately generate a virtual group photo, based on the above embodiments, in the embodiment of the present application, the method further includes:
[0075] The sample portrait is generalized, and the sample portrait is replaced with the generalized image.
[0076] Since in actual scenes, the portraits and background images input into the diffusion model are usually images collected in different scenes, different lighting, etc., and the sample images are processed by the method provided in the embodiment of the present application to obtain sample portraits and sample background images that are images collected in the same scene, etc. If the diffusion model is trained only by such sample images, it is impossible to effectively obtain a diffusion model that can generate a more natural virtual photo. Therefore, in the embodiment of the present application, the electronic device can generalize the sample portrait after obtaining the sample portrait, and replace the sample portrait with the generalized image. Thereby, the sample portrait and the sample background image are not images collected in the same scene, and the training of the diffusion model can be further realized.
[0077] In order to accurately generate a virtual group photo, based on the above embodiments, in the embodiment of the present application, the method further includes:
[0078] The sample background image is generalized and the processed image is used to replace the sample background image.
[0079] Since in actual scenes, the portrait and background images input to the diffusion model are usually images collected in different scenes, different lighting, etc., and the sample images are processed by the method provided in the embodiment of the present application to obtain sample portraits and sample background images that are images collected in the same scene. If the diffusion model is trained only by such sample images, it is impossible to effectively obtain a diffusion model that can generate a more natural virtual photo. Therefore, in an embodiment of the present application, the electronic device can generalize the sample background image after obtaining the sample background image, and replace the sample background image with the generalized image. Thereby, the sample portrait and sample background image are not images collected in the same scene, and the training of the diffusion model is further realized.
[0080] In order to accurately implement the training of the diffusion model, based on the above embodiments, in the embodiment of the present application, the generalization process includes at least one of the following processes:
[0081] Color conversion processing, gamma conversion processing, image enhancement processing, blurring processing, image flipping processing and image cropping processing.
[0082] The generalized processing described in the embodiments of the present application includes at least one of the following processing: color transformation processing, gamma transformation processing, image enhancement processing, blurring processing, image flipping processing and image capture processing.
[0083] The goal of a virtual group photo is to generate a photo of a person that is realistic and harmonious in a new background. To this end, this application uses sample group photos of different tasks in different backgrounds taken in real life as labels for algorithm training, and then uses methods such as portrait segmentation and image inpainting to create sample people and sample background images required for diffusion model training. Then, methods such as image capture and data enhancement are used to simulate a variety of real-life scenarios in virtual group photos, thereby enhancing the generalization ability of the model.
[0084] By enhancing the data of the foreground portrait, the diffusion model has the ability to harmonize the image. In the generated image, the foreground portrait and the background will be color adjusted, making the new image more natural and realistic.
[0085] Usually, in the newly generated virtual group photo, the background and the background image have the same style, but the content is not completely the same, and the size of the image containing the person and the background image uploaded by the user are also different. Therefore, during the training process, the electronic device obtains multiple background images by randomly cutting out areas of different sizes on the sample background image, and also collects some background images with similar styles. Further data augmentation is performed by color transformation, blurring, and left-right flipping, so that the new background image is consistent with the background in the image containing the person in style and inconsistent with the background in the image containing the person in content, so that enough samples for training can be constructed. For the foreground portrait, considering that in actual applications, the image containing the person uploaded by the user and the background image have inconsistent colors and lighting, the foreground portrait is subjected to color transformation and gamma transformation to simulate the situation of the diversity of user portrait lighting. By performing data augmentation on the foreground portrait and background, the diffusion model can adapt to the scene with inconsistent colors between the foreground portrait and the background and diverse background images, so that the generated group photo is real and natural.
[0086] As mentioned above, for the diffusion model, the new sample background images obtained through data augmentation are similar in style to the original sample background images, but these newly generated sample background images and the original sample background images are used as labels in the training process. After the sample portraits are processed by color adjustment and gamma transformation, their training label data does not change, and the labels of the original sample portraits can also be used as labels in the training process. Therefore, the labels of the multiple input data obtained through data augmentation have not changed, and they are still the original real-life images, and do not need to be generated separately, which also simplifies the complexity of training data annotation during network training.
[0087] Figure 8 A schematic diagram of a training process provided in an embodiment of the present application.
[0088] Depend on Figure 8 It can be seen that the electronic device can first obtain a sample group photo, extract a sample portrait from the sample group photo through a portrait segmentation algorithm, and perform generalization processing on the sample portrait to obtain a sample portrait trained by the user; use an image repair algorithm to process the sample portrait and the sample group photo to generate a sample background image, and perform generalization processing on the sample background image to obtain a sample background image trained by the user;
[0089] The electronic device inputs the sample background image and the sample portrait into the original diffusion model to obtain the predicted group photo output by the diffusion model. The original diffusion model is trained based on the predicted group photo and the sample group photo.
[0090] In order to accurately generate a virtual group photo, based on the above embodiments, in the embodiment of the present application, after extracting the portrait in the image and before inputting the portrait and the background image into the pre-trained diffusion model, the method further includes:
[0091] For each pixel in the portrait, if the pixel value of the pixel is less than a preset pixel threshold, the pixel value of the pixel is adjusted to a first preset pixel value; if the pixel value is not less than the preset pixel threshold, the pixel value of the pixel is adjusted to a second preset pixel value; for the adjusted portrait, a subsequent step of inputting the portrait and the background image into a pre-trained diffusion model is performed.
[0092] After the image containing the person passes through the portrait segmentation algorithm, the foreground and background will be distinguished. The foreground is the person, and the portrait can be obtained. However, after the portrait segmentation, the pixel points on the edge adjacent to the foreground and the background may not be clear. Therefore, after obtaining the portrait, the electronic device can binarize the portrait to clarify the foreground and background areas. The pixel threshold of the binarization can be 128. The electronic device can determine that each pixel point in the portrait is a pixel point in the background if the pixel value of the pixel point is less than the preset pixel threshold, and adjust the pixel value of the pixel point to a first preset pixel value, which can be a pixel value corresponding to black. If the pixel point is not less than the preset pixel threshold, it is determined that the pixel point is a pixel point in the person, that is, the foreground, and the pixel value of the pixel point can be adjusted to a second preset pixel value, which can be a pixel value corresponding to white; and the subsequent step of inputting the portrait and background image into the pre-trained diffusion model is performed for the adjusted portrait.
[0093] Fig. 9A schematic diagram of a virtual group photo generation device provided in an embodiment of the present application, the device comprising:
[0094] A receiving module 901 is used to receive an image containing a person and a background image;
[0095] An extraction module 902 is used to extract the human portrait in the image by using a human portrait segmentation algorithm;
[0096] The processing module 903 is used to input the portrait and the background image into a pre-trained diffusion model to obtain a virtual photo output by the diffusion model in which the portrait and the background image are fused.
[0097] Furthermore, the processing module 903 is specifically used for the network layer of the diffusion model to generate a random noise image; obtain the remaining number of repetitions saved in advance, and generate an embedding vector corresponding to the background image; loop the following steps until there are no remaining repetitions, and output the noise image as a virtual photo: input the noise image, the embedding vector, the portrait and the vector corresponding to the remaining number of repetitions into the diffusion network Unet pre-trained in the diffusion model, and the Unet outputs a predicted noise image; use the predicted noise image to replace the noise image, and update the remaining number of repetitions.
[0098] Furthermore, the processing module 903 is also used to obtain a pre-saved image of any person, add any preset noise to the image; input the image after adding noise, the vector corresponding to any number of times, the portrait corresponding to the image, and the embedding vector corresponding to a background image into Unet to obtain the predicted noise output by the Unet; and train the Unet according to the predicted noise and the preset noise.
[0099] Furthermore, the processing module 903 is also used to obtain any sample group photo in the sample set, and a sample portrait and a sample background image saved for the sample group photo; input the sample portrait and the sample background image into the original diffusion model to obtain a predicted group photo output by the original diffusion model; and train the original diffusion model based on the predicted group photo and the sample group photo.
[0100] Furthermore, the processing module 903 is further configured to extract a sample portrait from the sample group photo by using a portrait segmentation algorithm; and process the sample portrait and the sample group photo by using an image repair algorithm to generate a sample background image.
[0101] Furthermore, the processing module 903 is also used to perform generalization processing on the sample portrait, and replace the sample portrait with the generalized image.
[0102] Furthermore, the processing module 903 is also used to perform generalization processing on the sample background image and replace the sample background image with the processed image.
[0103] Furthermore, the processing module 903 is also used to, for each pixel point in the portrait, if the pixel value of the pixel point is less than a preset pixel threshold, adjust the pixel value of the pixel point to a first preset pixel value; if the pixel point is not less than the preset pixel threshold, adjust the pixel value of the pixel point to a second preset pixel value; and execute the subsequent step of inputting the portrait and the background image into a pre-trained diffusion model for the adjusted portrait.
[0104] Fig.10 The present invention provides a schematic diagram of an electronic device structure according to an embodiment of the present invention. Based on the above embodiments, the present invention further provides an electronic device, such as Fig.10 As shown, it includes: a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004;
[0105] The memory 1003 stores a computer program. When the program is executed by the processor 1001, the processor 1001 performs the following steps:
[0106] Receiving an image including a person and a background image;
[0107] Extracting the portrait in the image by using a portrait segmentation algorithm;
[0108] The portrait and the background image are input into a pre-trained diffusion model, and a virtual photo output by the diffusion model is obtained by merging the portrait and the background image.
[0109] Further, the processor 1001 is specifically configured to generate a random noise image at the network layer of the diffusion model; obtain a pre-saved remaining number of repetitions, and generate an embedding vector corresponding to the background image; and loop through the following steps until there is no remaining number of repetitions, and output the noise image as a virtual photo:
[0110] Input the noise image, the embedding vector, the portrait and the vector corresponding to the remaining number of repetitions into a diffusion network Unet pre-trained in the diffusion model, and the Unet outputs a predicted noise image;
[0111] The predicted noise image is used to replace the noise image, and the remaining number of repetitions is updated.
[0112] Further, the processor 1001 is further configured to obtain a pre-saved image of any person, add any preset noise to the image; input the image after adding the noise, a vector corresponding to any number of times, a person corresponding to the image, and an embedding vector corresponding to a background image into Unet to obtain the predicted noise output by the Unet;
[0113] The Unet is trained according to the predicted noise and the preset noise.
[0114] Furthermore, the processor 1001 is further configured to obtain any sample group photo in the sample set, and a sample portrait and a sample background image saved for the sample group photo;
[0115] Inputting the sample portrait and the sample background image into the original diffusion model to obtain a predicted group photo output by the original diffusion model;
[0116] The original diffusion model is trained according to the predicted group photo and the sample group photo.
[0117] Furthermore, the processor 1001 is further configured to extract a sample portrait from the sample group photo by using a portrait segmentation algorithm;
[0118] The sample portrait and the sample group photo are processed by using an image repair algorithm to generate a sample background image.
[0119] Furthermore, the processor 1001 is further configured to perform generalization processing on the sample portrait, and replace the sample portrait with the generalized image.
[0120] Furthermore, the processor 1001 is further configured to perform generalization processing on the sample background image, and replace the sample background image with the processed image.
[0121] Furthermore, the processor 1001 is also used to, for each pixel point in the portrait, if the pixel value of the pixel point is less than a preset pixel threshold, adjust the pixel value of the pixel point to a first preset pixel value; if the pixel point is not less than the preset pixel threshold, adjust the pixel value of the pixel point to a second preset pixel value; and execute the subsequent step of inputting the portrait and the background image into a pre-trained diffusion model for the adjusted portrait.
[0122] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0123] The communication interface is used for communication between the above electronic device and other devices.
[0124] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0125] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
[0126] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device implements the following steps when executing:
[0127] The memory stores a computer program, and when the program is executed by the processor, the processor performs the following steps:
[0128] Receiving an image including a person and a background image;
[0129] Extracting the portrait in the image by using a portrait segmentation algorithm;
[0130] The portrait and the background image are input into a pre-trained diffusion model, and a virtual photo output by the diffusion model is obtained by merging the portrait and the background image.
[0131] In a possible implementation manner, obtaining a virtual photo obtained by merging the portrait output by the diffusion model with the background image includes:
[0132] The network layer of the diffusion model generates a random noise image; obtains the remaining number of repetitions saved in advance, generates an embedding vector corresponding to the background image; and loops through the following steps until there are no remaining repetitions, and outputs the noise image as a virtual photo:
[0133] Input the noise image, the embedding vector, the portrait and the vector corresponding to the remaining number of repetitions into a diffusion network Unet pre-trained in the diffusion model, and the Unet outputs a predicted noise image;
[0134] The predicted noise image is used to replace the noise image, and the remaining number of repetitions is updated.
[0135] In one possible implementation, the Unet is trained in the following manner:
[0136] Obtain a pre-saved image of any person, and add any preset noise to the image; input the image after adding noise, the vector corresponding to any number of times, the person corresponding to the image, and the embedding vector corresponding to a background image into Unet, and obtain the predicted noise output by the Unet;
[0137] The Unet is trained according to the predicted noise and the preset noise.
[0138] In a possible implementation, the diffusion model is trained in the following manner:
[0139] Obtain any sample group photo in the sample set, and the sample portrait and sample background image saved for the sample group photo;
[0140] Inputting the sample portrait and the sample background image into the original diffusion model to obtain a predicted group photo output by the original diffusion model;
[0141] The original diffusion model is trained according to the predicted group photo and the sample group photo.
[0142] In a possible implementation, a sample portrait and a sample background image corresponding to a sample group photo are obtained in the following manner:
[0143] Extracting a sample portrait from the sample group photo using a portrait segmentation algorithm;
[0144] The sample portrait and the sample group photo are processed by using an image repair algorithm to generate a sample background image.
[0145] In a possible implementation, the method further includes:
[0146] The sample portrait is generalized, and the sample portrait is replaced with the generalized image.
[0147] In a possible implementation, the method further includes:
[0148] The sample background image is generalized and the processed image is used to replace the sample background image.
[0149] In a possible implementation, the generalization process includes at least one of the following processes:
[0150] Color transformation processing, gamma transformation processing, image enhancement processing, blurring processing, image flipping processing and image capture processing.
[0151] In a possible implementation manner, after extracting the portrait from the image and before inputting the portrait and the background image into a pre-trained diffusion model, the method further includes:
[0152] For each pixel in the portrait, if the pixel value of the pixel is less than a preset pixel threshold, the pixel value of the pixel is adjusted to a first preset pixel value; if the pixel value is not less than the preset pixel threshold, the pixel value of the pixel is adjusted to a second preset pixel value; for the adjusted portrait, a subsequent step of inputting the portrait and the background image into a pre-trained diffusion model is performed.
[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0157] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for generating a virtual group photo, characterized in that: The method comprises: Receiving an image including a person and a background image; Extracting the portrait in the image by using a portrait segmentation algorithm; The portrait and the background image are input into a pre-trained diffusion model, and a virtual photo output by the diffusion model is obtained by merging the portrait and the background image.
2. The method according to claim 1, characterized in that: The step of obtaining a virtual photo of the person image output by the diffusion model and the background image fused together comprises: The network layer of the diffusion model generates a random noise image; obtains the remaining number of repetitions saved in advance, and generates an embedding vector corresponding to the background image; and loops through the following steps until there are no remaining repetitions, and outputs the noise image as a virtual photo: Input the noise image, the embedding vector, the portrait and the vector corresponding to the remaining number of repetitions into a diffusion network Unet pre-trained in the diffusion model, and the Unet outputs a predicted noise image; The predicted noise image is used to replace the noise image, and the remaining number of repetitions is updated.
3. The method according to claim 2, characterized in that The Unet is trained in the following way: Obtain a pre-saved image of any person, and add any preset noise to the image; input the image after adding noise, the vector corresponding to any number of times, the person corresponding to the image, and the embedding vector corresponding to a background image into Unet, and obtain the predicted noise output by the Unet; The Unet is trained according to the predicted noise and the preset noise.
4. The method according to claim 1, characterized in that The diffusion model is trained in the following way: Obtain any sample group photo in the sample set, and the sample portrait and sample background image saved for the sample group photo; Inputting the sample portrait and the sample background image into the original diffusion model to obtain a predicted group photo output by the original diffusion model; The original diffusion model is trained according to the predicted group photo and the sample group photo.
5. The method according to claim 4, characterized in that The sample portrait and sample background image corresponding to a sample group photo are obtained in the following way: Extracting a sample portrait from the sample group photo using a portrait segmentation algorithm; The sample portrait and the sample group photo are processed by using an image repair algorithm to generate a sample background image.
6. The method according to claim 5, characterized in that The method further comprises: The sample portrait is generalized, and the sample portrait is replaced with the generalized image.
7. The method according to claim 5, characterized in that The method further comprises: The sample background image is generalized and the processed image is used to replace the sample background image.
8. The method according to claim 6 or 7, characterized in that: The generalization process includes at least one of the following processes: Color transformation processing, gamma transformation processing, image enhancement processing, blurring processing, image flipping processing and image capture processing.
9. The method according to claim 1, characterized in that: After extracting the portrait from the image and before inputting the portrait and the background image into a pre-trained diffusion model, the method further includes: For each pixel in the portrait, if the pixel value of the pixel is less than a preset pixel threshold, the pixel value of the pixel is adjusted to a first preset pixel value; if the pixel value is not less than the preset pixel threshold, the pixel value of the pixel is adjusted to a second preset pixel value; for the adjusted portrait, a subsequent step of inputting the portrait and the background image into a pre-trained diffusion model is performed.
10. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the virtual group photo generation method as described in any one of claims 1 to 9 when executing the computer program stored in the memory.