A method and device for generating a cutout image, a computer device and a storage medium

Abstract

The application discloses a kind of image cutting data generation method, device, computer equipment and storage medium, the method includes: respectively obtaining initial foreground image, background image and initial foreground image corresponding alpha image;By deep learning model, initial foreground image and background image are carried out color distribution conversion, and first foreground image is output by deep learning model;Respectively calculate the RGB average value corresponding to initial foreground image and first foreground image respectively, and according to the RGB average value conversion corresponding second foreground image is obtained;Second foreground image, background image and alpha image are fused, and the image after fusion is used as final image cutting data.The application is converted to initial foreground image by color distribution conversion, and is fused with background image and alpha image again in combination with RGB average value, improve the generation quality of image cutting data.Meanwhile, only need to replace background image, new image cutting data can be generated, so as to improve the generation efficiency of image cutting data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method and device for generating image matting data, a computer device and a storage medium. BACKGROUND

[0002] Image matting algorithms are widely used in current video products, and have great commercial value in special effect functions using body contours, background replacement in meetings, and blur functions. In the training of image matting algorithm models, the lack of training data has always been a common problem. Too little training data makes it difficult to train a good model. The main reason is that a lot of human resources are needed for data annotation and collection. To solve this problem, the common method is to use data generation to increase the amount of training data. However, the past data generation is prone to generate unrealistic data that affects the training results. Therefore, how to generate data close to the real situation and improve the generation efficiency is a problem that needs to be solved by those skilled in the art. SUMMARY

[0003] Embodiments of the present application provide a method and device for generating image matting data, a computer device and a storage medium, aiming to improve the generation quality of image matting data and improve the generation efficiency of image matting data.

[0004] In a first aspect, embodiments of the present application provide a method for generating image matting data, comprising:

[0005] obtaining an initial foreground image, a background image and an alpha image corresponding to the initial foreground image, respectively;

[0006] performing color distribution conversion on the initial foreground image and the background image by a deep learning model, and outputting a first foreground image from the deep learning model;

[0007] calculating the respective RGB average values of the initial foreground image and the first foreground image, respectively, and converting the respective second foreground images according to the RGB average values;

[0008] fusing the second foreground image, the background image and the alpha image, and taking the fused image as the final image matting data.

[0009] In a second aspect, embodiments of the present application provide a device for generating image matting data, comprising:

[0010] an image acquisition unit configured to obtain an initial foreground image, a background image and an alpha image corresponding to the initial foreground image, respectively;

[0011] a first conversion unit configured to perform color distribution conversion on the initial foreground image and the background image by using a deep learning model, and output a first foreground image from the deep learning model;

[0012] a second conversion unit configured to respectively calculate RGB average values corresponding to the initial foreground image and the first foreground image, and convert a second foreground image corresponding to the initial foreground image and the first foreground image according to the RGB average values;

[0013] a first fusion unit configured to fuse the second foreground image, the background image, and the alpha image, and take the fused image as final cutout data.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the cutout data generation method in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the cutout data generation method in the first aspect.

[0016] The embodiment of the present application provides a cutout data generation method, device, computer device, and storage medium, and the method includes: acquiring an initial foreground image, a background image, and an alpha image corresponding to the initial foreground image; performing color distribution conversion on the initial foreground image and the background image by using a deep learning model, and outputting a first foreground image from the deep learning model; respectively calculating RGB average values corresponding to the initial foreground image and the first foreground image, and converting a second foreground image corresponding to the initial foreground image and the first foreground image according to the RGB average values; fusing the second foreground image, the background image, and the alpha image, and taking the fused image as final cutout data. The embodiment of the present application converts the initial foreground image and the background image in color distribution, and further converts the initial foreground image in combination with the RGB average value, so as to fuse the converted second foreground image with the background image and the alpha image, and improve the generation quality of the cutout data. Meanwhile, the embodiment of the present application only needs to replace the background image when generating the cutout data, and can generate new cutout data, so as to greatly improve the generation efficiency of the cutout data. BRIEF DESCRIPTION OF DRAWINGS

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for generating keying data according to an embodiment of the present invention;

[0019] Figure 2 Parts a, b, and c in the figure are schematic diagrams of the initial foreground image, alpha image, and background image in a keying data generation method provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic block diagram of a keying data generation device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] Please see below. Figure 1 , Figure 1 The flowchart of a keying data generation method provided in an embodiment of the present invention is shown, specifically including steps S101 to S104.

[0026] S101. Obtain the initial foreground image, the background image, and the alpha image corresponding to the initial foreground image, respectively;

[0027] S102. The initial foreground image and background image are subjected to color distribution transformation through a deep learning model, and the first foreground image is output by the deep learning model.

[0028] S103. Calculate the RGB average values ​​of the initial foreground image and the first foreground image respectively, and convert them according to the RGB average values ​​to obtain the corresponding second foreground image;

[0029] S104. The second foreground image, background image and alpha image are fused together, and the fused image is used as the final keying data.

[0030] In this embodiment, combined with Figure 2 First, an initial foreground image, a background image, and an alpha image are obtained separately. The initial foreground image can be a foreground image including a portrait, while the background image is an image different from the background in the initial foreground image. The alpha image, or alpha channel, refers to the transparency and semi-transparency of an image. Then, a deep learning model is used to transform and fuse the initial foreground image and the background image into a first foreground image. Further, based on the RGB average values ​​of the initial foreground image and the background image, a second foreground image corresponding to the initial foreground image is calculated. Subsequently, the second foreground image is fused with the background image and the alpha image to obtain the final matting data.

[0031] This embodiment improves the quality of keying data generation by performing color distribution transformation on the initial foreground and background images and further transforming the initial foreground image using RGB average values. The transformed second foreground image is then fused with the background and alpha images. Furthermore, this embodiment only requires replacing the background image to generate new keying data, significantly improving the generation efficiency and reducing the annotation cost.

[0032] In one embodiment, step S101 includes:

[0033] The initial foreground image is converted to a grayscale image, and the edge pixels of the grayscale image are isolated.

[0034] Invert the isolated grayscale image and create a mask;

[0035] The boundary information of the gray image is extracted by a bit operation, and the alpha image is extracted by superimposing the gray image based on the boundary information.

[0036] In this embodiment, the initial foreground image and the corresponding alpha image are separated by image mask operation to extract the alpha image. In digital image processing, image mask is mainly used for: ① extracting the region of interest, multiplying the region of interest mask prepared in advance with the image to be processed to obtain the region of interest image, and the image value in the region of interest remains unchanged, while the image value outside the region is 0. ② Shielding, using a mask to shield some areas of the image, so that they do not participate in processing or calculation of processing parameters, or only the shielded area is processed or counted. ③ Structure feature extraction, using similarity variables or image matching method to detect and extract the structure features similar to the mask in the image. ④ Production of special shape image.

[0037] In an embodiment, the deep learning model is an SSH self-supervised synthesis framework.

[0038] Further, the step S102 comprises:

[0039] The content features in the initial foreground image are extracted by a content network in the SSH self-supervised synthesis framework, and the appearance features in the background image are extracted by a reference network in the SSH self-supervised synthesis framework.

[0040] The content features and the appearance features are fused by a fusion network in the SSH self-supervised synthesis framework to obtain the first foreground image.

[0041] In this embodiment, the SSH self-supervised synthesis framework is an integrated framework based on struts+spring+hibernate, wherein Struts is an MVC framework based on Sun J2EE platform, which is mainly implemented by using Servlet and JSP technology; Spring is a powerful framework that solves many common problems in J2EE development; Hibernate is an open source object-relational mapping framework, which provides a very lightweight object encapsulation for JDBC, so that users can freely use object programming thinking to manipulate the database. When the SSH self-supervised synthesis framework is used for color distribution conversion of the initial foreground image and the background image to output the first foreground image, the SSH self-supervised synthesis framework extracts the appearance features of the background image and the content features of the initial foreground image by using the reference network and the content network respectively, and then reconstructs and coordinates the appearance features and the content features by using the fusion network to output the first foreground image.

[0042] In an embodiment, the step S103 comprises:

[0043] The second foreground image is calculated and converted according to the following formula:

[0044] F" = F - (mean(F) - mean(F'))

[0045] In the formula, F" is the second foreground image, F' is the first foreground image, F is the initial foreground image, and mean is the RGB average value.

[0046] In this embodiment, the initial foreground image is converted into the second foreground image by the above formula, and the second foreground image has reduced color distribution differences compared to the initial foreground image, so that the subsequent fusion of the cutout data has higher quality.

[0047] In a specific embodiment, the step S102 further comprises:

[0048] The RGB average values of the initial foreground image and the first foreground image are calculated based on the Python platform, respectively. Of course, in other embodiments, other ways can be used to calculate the RGB average value, such as calculating by Photoshop software, etc.

[0049] In an embodiment, the step S104 comprises:

[0050] The second foreground image, the background image, and the alpha image are fused according to the following formula:

[0051] I = aF" + (1-a)B

[0052] In the formula, I represents the fused cutout data, a represents the alpha image, and B represents the background image.

[0053] In this embodiment, the second foreground image, the background image, and the alpha image are fused into the final cutout data according to the above fusion formula, rather than the initial foreground image and the background image and the alpha image being fused into the final cutout data. This reduces the different color distributions caused by different light sources and venues. In addition, based on the above fusion formula, when more cutout data is needed, the number of background images B can be increased to achieve this, i.e. improving the generation efficiency of the cutout data.

[0054] Figure 3 A schematic block diagram of a cutout data generation device 300 provided by an embodiment of the present application is provided, and the device 300 comprises:

[0055] The image acquisition unit 301 is configured to acquire an initial foreground image, a background image and an alpha image corresponding to the initial foreground image respectively.

[0056] The first conversion unit 302 is configured to perform color distribution conversion on the initial foreground image and the background image by using a deep learning model, and output a first foreground image by using the deep learning model.

[0057] The second conversion unit 303 is configured to calculate respective RGB average values of the initial foreground image and the first foreground image respectively, and convert a corresponding second foreground image according to the RGB average values.

[0058] The first fusion unit 304 is configured to fuse the second foreground image, the background image and the alpha image, and take the fused image as final image matting data.

[0059] In an embodiment, the image acquisition unit 301 comprises:

[0060] The grayscale conversion unit is configured to convert the initial foreground image into a grayscale image, and isolate edge pixels of the grayscale image.

[0061] The image inversion unit is configured to invert the isolated grayscale image, and create a mask.

[0062] The information extraction unit is configured to extract boundary information of the grayscale image by using bit operation, and superimpose the grayscale image based on the boundary information, so as to extract the alpha image.

[0063] In an embodiment, the deep learning model is an SSH self-supervised synthesis framework.

[0064] In an embodiment, the first conversion unit 302 comprises:

[0065] The feature extraction unit is configured to extract content features in the initial foreground image by using a content network in the SSH self-supervised synthesis framework, and extract appearance features in the background image by using a reference network in the SSH self-supervised synthesis framework.

[0066] The feature fusion unit is configured to fuse the content features and the appearance features by using a fusion network in the SSH self-supervised synthesis framework, so as to obtain the first foreground image.

[0067] In an embodiment, the second conversion unit 303 comprises:

[0068] The image conversion unit is configured to calculate and convert the second foreground image according to the following formula:

[0069] F" = F - (mean(F) - mean(F'))

[0070] In the formula, F" is the second foreground image, F' is the first foreground image, F is the initial foreground image, and mean is the RGB average value.

[0071] In an embodiment, the first fusion unit 304 comprises:

[0072] A second fusion unit is configured to fuse the second foreground image, the background image and the alpha image according to the following formula:

[0073] I = aF" + (1-a)B

[0074] In the formula, I represents the fused cutout data, a represents the alpha image, B represents the background image, and a represents the alpha image.

[0075] In an embodiment, the second conversion unit 303 further comprises:

[0076] An average value calculation unit is configured to calculate the RGB average values of the initial foreground image and the first foreground image respectively based on a Python platform.

[0077] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and are not described here.

[0078] The embodiment of the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program can implement the steps provided by the above embodiment when executed. The storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

[0079] The embodiment of the present application further provides a computer device, which can include a memory and a processor, the memory has a computer program stored therein, and the processor can implement the steps provided by the above embodiment when calling the computer program in the memory. Of course, the computer device can also include various network interfaces, power supplies and other components.

[0080] The various embodiments described in the specification are presented for purposes of illustration and description. Each of the embodiments highlight a different aspect of the application. The embodiments are not mutually exclusive, and can be combined in various manners. The embodiments disclosed herein are not exhaustive of the ways in which the application can be practiced. Numerous modifications and adaptations will be apparent to those skilled in the art. The embodiments disclosed herein are merely exemplary in nature and are not intended to limit the scope of the application.

[0081] It should also be noted that the terms "first", "second", and the like, do not denote any order, quantity, combination, or importance, but rather are used to identify one element from another. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A method for generating keying data, characterized in that, include: The initial foreground image, the background image, and the alpha image corresponding to the initial foreground image are obtained respectively. The initial foreground and background images are subjected to color distribution transformation using a deep learning model, and the first foreground image is output by the deep learning model; the deep learning model is the SSH self-supervised synthesis framework. Calculate the RGB average values ​​of the initial foreground image and the first foreground image respectively, and convert them according to the RGB average values ​​to obtain the corresponding second foreground image; The second foreground image, background image, and alpha image are fused together, and the fused image is used as the final keying data; The step of performing color distribution transformation on the initial foreground and background images using a deep learning model, and outputting a first foreground image by the deep learning model, includes: Content features in the initial foreground image are extracted using the content network in the SSH self-supervised synthesis framework, and appearance features in the background image are extracted using the reference network in the SSH self-supervised synthesis framework. The content features and appearance features are fused using a fusion network in the SSH self-supervised synthesis framework to obtain the first foreground image; The step of calculating the RGB average values ​​of the initial foreground image and the first foreground image respectively, and converting the RGB average values ​​to obtain the corresponding second foreground image, includes: The second foreground image is obtained by calculating and converting according to the following formula: F''=F-(mean(F)-mean(F')) In the formula, F'' is the second foreground image, F' is the first foreground image, F is the initial foreground image, and mean is the RGB average value.

2. The keying data generation method according to claim 1, characterized in that, The step of obtaining the initial foreground image, the background image, and the alpha image corresponding to the initial foreground image includes: The initial foreground image is converted to a grayscale image, and the edge pixels of the grayscale image are isolated. Invert the isolated grayscale image and create a mask; The boundary information of the grayscale image is extracted by bitwise operations, and the grayscale image is superimposed based on the boundary information to extract the alpha image.

3. The keying data generation method according to claim 1, characterized in that, The step of fusing the second foreground image, background image, and alpha image, and using the fused image as the final matting data, includes: The second foreground image, background image, and alpha image are fused according to the following formula: I=αF''+(1-α)B In the formula, I represents the fused keying data, α represents the alpha image, and B represents the background image.

4. The keying data generation method according to claim 1, characterized in that, The step of calculating the RGB average values ​​corresponding to the initial foreground image and the first foreground image respectively, and converting the RGB average values ​​to obtain the corresponding second foreground image, further includes: The RGB average values ​​of the initial foreground image and the first foreground image were calculated using the Python platform.

5. A keying data generation device, characterized in that, include: An image acquisition unit is used to acquire an initial foreground image, a background image, and an alpha image corresponding to the initial foreground image, respectively. The first conversion unit is used to perform color distribution conversion on the initial foreground image and background image through a deep learning model, and outputs the first foreground image by the deep learning model; the deep learning model is the SSH self-supervised synthesis framework. The second conversion unit is used to calculate the RGB average values ​​corresponding to the initial foreground image and the first foreground image respectively, and convert them according to the RGB average values ​​to obtain the corresponding second foreground image; The first fusion unit is used to fuse the second foreground image, background image and alpha image, and use the fused image as the final keying data; The first conversion unit includes: The feature extraction unit is used to extract content features from the initial foreground image through the content network in the SSH self-supervised synthesis framework, and to extract appearance features from the background image through the reference network in the SSH self-supervised synthesis framework. The feature fusion unit is used to fuse the content features and appearance features through the fusion network in the SSH self-supervised synthesis framework to obtain the first foreground image; The second conversion unit includes: The image conversion unit is used to calculate and convert the second foreground image according to the following formula: F''=F-(mean(F)-mean(F')) In the formula, F'' is the second foreground image, F' is the first foreground image, F is the initial foreground image, and mean is the RGB average value.

6. A computer device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the keying data generation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the keying data generation method as described in any one of claims 1 to 4.

Images