Green screen keying method, device and electronic device

Abstract

The present disclosure relates to a green screen keying method, apparatus and electronic device, and particularly relates to the field of image processing technology. It includes: obtaining a first image; inputting the first image into a target parameter prediction model, and obtaining a target parameter map based on the target parameter prediction model, where the target parameter map includes transparency adjustment parameters of at least some pixel points in the first image; determining a target opacity map of the foreground image in the first image according to the transparency adjustment parameters of at least some pixel points and the central color distance of at least some pixel points; and calculating the foreground image by combining the target opacity map and the color values of the first image. The embodiments of the present disclosure are used to solve the problem that the scenario of using global parameters for adjusting parameters in the process of green screen keying has limitations, resulting in poor accuracy of green screen keying for some scenarios, and the resulting foreground image is inaccurate.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and particularly to a green screen keying method, apparatus, and electronic device. Background Art

[0002] Green screen keying Figure 1 Generally includes obtaining the foreground opacity (alpha) by keying, and removing green from the foreground. The purpose of obtaining the foreground opacity is to separate the foreground and the background. Removing green from the foreground mainly removes green from semi-transparent areas, segmentation edges, and green reflections of foreground objects, so as to superimpose other backgrounds for material synthesis. Currently, in the process of green screen keying, some adjustment parameters are used, and these adjustment parameters are often applied globally. However, due to reasons such as uneven illumination and uneven green screen, the requirements for these adjustment parameters in different regions of the image are often different. Therefore, the scenario of using the adjustment parameters in a global parameter manner in the process of green screen keying has limitations, resulting in poor accuracy of keying for some scenarios during green screen keying, and making the final obtained foreground image inaccurate. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a green screen keying method, apparatus, and electronic device, which can determine corresponding transparency adjustment parameters according to pixel points in an image, determine the target opacity map of the foreground image, and determine the foreground image based on the target opacity map. Therefore, this green screen keying method can calculate an accurate foreground image in any scenario.

[0004] To achieve the above object, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a first aspect, a green screen keying method is provided, including:

[0006] Obtain a first image;

[0007] Input the first image into a target parameter prediction model, and obtain a target parameter map based on the target parameter prediction model. The target parameter map includes transparency adjustment parameters of at least some pixel points in the first image;

[0008] Determine the target opacity map of the foreground image in the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points;

[0009] Calculate the foreground image by combining the target opacity map and the color values of the first image.

[0010] As an optional implementation manner of the embodiments of the present disclosure, the transparency adjustment parameters include: foreground adjustment parameters and / or background adjustment parameters.

[0011] As an optional implementation manner of an embodiment of the present disclosure, the determining the target opacity map of the foreground image in the first image according to the transparency adjustment parameter of at least some of the pixel points and the central color distance of at least some of the pixel points includes:

[0012] Determining an initial opacity map of the foreground image in the first image according to the transparency adjustment parameter of at least some of the pixel points and the central color distance of at least some of the pixel points;

[0013] Taking the grayscale image of the first image as a guidance image, and performing guided filtering on the initial opacity map to obtain the target opacity map.

[0014] As an optional implementation manner of an embodiment of the present disclosure, the calculating the foreground image by combining the target opacity map and the color value of the first image includes:

[0015] Obtaining a fusion opacity coefficient;

[0016] Calculating the color value of the foreground image according to the fusion opacity coefficient, the color value of the first image, and the color value of the background image.

[0017] As an optional implementation manner of an embodiment of the present disclosure, the obtaining the fusion opacity coefficient includes:

[0018] When the color value of the G channel in the first image is less than or equal to a target color value, determining the fusion opacity coefficient to be 1,

[0019] When the color value of the G channel in the first image is greater than the target color value, determining the fusion opacity coefficient according to a first color distance and a second color distance, where the first color distance is the distance from the color value in the first image to the green limit boundary plane, and the second color distance is the distance from the background color mean value to the green limit boundary plane;

[0020] Wherein, the target color value is the sum of the color values of the R channel and the B channel, and the green limit boundary plane is the plane determined when the color value of the G channel is equal to the target color value.

[0021] As an optional implementation manner of an embodiment of the present disclosure, before inputting the first image into a target parameter prediction model and obtaining a target parameter map based on the target parameter prediction model, the method further includes:

[0022] Training an initial parameter prediction model based on sample information to obtain the target parameter prediction model;

[0023] The sample information includes multiple sample images and a first parameter map corresponding to each sample image. The sample images include a foreground image, a background image, and a random green screen image. The first parameter map is a parameter map determined based on the UV coordinate vectors of the pixel points of the foreground image, the UV coordinate vectors of the pixel points of the random green screen image, and the central color distance of the pixel points, and / or a parameter map determined based on the UV coordinate vectors of the background image pixel points of the random green screen image, the UV coordinate vectors of the pixel points of the random green screen image, and the central color distance of the pixel points; the random green screen image is obtained by fusing the color channels of the foreground image and the background image according to the opacity of the foreground image; the background image is obtained by superimposing random green on a real picture.

[0024] As an optional implementation manner of an embodiment of the present disclosure, training the initial parameter prediction model based on the sample information to obtain the target parameter prediction model includes:

[0025] Performing the following steps at least once to obtain the target parameter prediction model:

[0026] Obtain a target sample image from the multiple sample images, input the target sample image into the initial parameter prediction model, and obtain the output parameter map of the target sample image output by the initial image processing model;

[0027] Determine a target loss function according to the output parameter map and the first parameter map corresponding to the target sample image;

[0028] Modify the initial parameter prediction model based on the target loss function;

[0029] Wherein, the target loss function includes:

[0030] The first loss function corresponding to the foreground adjustment parameter;

[0031] And / or

[0032] The second loss function corresponding to the background adjustment parameter.

[0033] As an optional implementation manner of an embodiment of the present disclosure, modifying the initial parameter prediction model based on the target loss function includes:

[0034] Based on the output parameter map, determine the first opacity map of the foreground image of the target sample image;

[0035] Based on the first parameter map corresponding to the target sample image, determine the second opacity map of the foreground image of the target sample image;

[0036] Determine a third loss function according to the first opacity map and the second opacity map;

[0037] Modify the initial parameter prediction model based on the target loss function and the third loss function.

[0038] In a second aspect, a green screen keying device is provided, including:

[0039] An acquisition module, configured to acquire a first image;

[0040] A prediction module, configured to input the first image into a target parameter prediction model, and acquire a target parameter map based on the target parameter prediction model, where the target parameter map includes transparency adjustment parameters of at least some pixel points in the first image;

[0041] A determination module, configured to determine an opacity map of the foreground image in the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points;

[0042] A keying module, configured to calculate the foreground image by combining the target opacity map and the color values of the first image.

[0043] In a third aspect, an electronic device is provided, including: a processor, a memory, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, the green screen keying method as described in the first aspect or any one of its optional implementation manners is implemented.

[0044] In a fourth aspect, a computer-readable storage medium is provided, including: a computer program stored on the computer-readable storage medium, where when the computer program is executed by a processor, the green screen keying method as described in the first aspect or any one of its optional implementation manners is implemented.

[0045] In a fifth aspect, a computer program product is provided, including: when the computer program product runs on a computer, the computer is enabled to implement the green screen keying method as described in the first aspect or any one of its optional implementation manners.

[0046] The green screen keying method provided by the embodiments of the present disclosure obtains a first image; inputs the first image into a target parameter prediction model, and obtains a target parameter map based on the target parameter prediction model. The target parameter map includes transparency adjustment parameters of at least some pixel points in the first image; determines an opacity map of the foreground image in the first image according to the transparency adjustment parameters of at least some pixel points and the central color distance of at least some pixel points; combines the target opacity map and the color values of the first image to calculate the foreground image. Through this solution, in the process of obtaining the opacity map of the foreground image, since the transparency adjustment parameters of at least some pixel points in the first image can be obtained first through the target parameter prediction model, even for images with uneven illumination and uneven green screens, because the local parameters (transparency adjustment parameters) are determined based on pixel points, and the opacity map of the foreground image is determined based on these local parameters, and then the opacity map of the foreground image and the color values of the first image are combined to calculate the foreground image of the first image. Therefore, this green screen keying method can calculate an accurate foreground image in any scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0048] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] Figure 1 It is a schematic flowchart of a green screen keying method provided by the embodiments of the present disclosure;

[0050] Figure 2 It is a schematic diagram of t1, t2, and d in the UV coordinates provided by the embodiments of the present disclosure;

[0051] Figure 3 It is a schematic diagram of the processing process for obtaining a target parameter map provided by the embodiments of the present disclosure;

[0052] Figure 4 It is a schematic diagram of calculating the target opacity map of the foreground image provided by the embodiments of the present disclosure;

[0053] Figure 5 It is a schematic diagram of a green limit boundary plane provided by the embodiments of the present disclosure;

[0054] Figure 6 It is a structural block diagram of a green screen keying device provided by the embodiments of the present disclosure;

[0055] Figure 7 The structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. Specific implementation manners

[0056] In order to more clearly understand the above objects, features, and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0057] Many specific details are set forth in the following description in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0058] Currently, during the green screen keying process, some adjustment parameters are used. These adjustment parameters are often applied globally. However, due to reasons such as uneven illumination and uneven green screen, the requirements for these adjustment parameters in different regions of the image are often different. Therefore, the scenario of using the adjustment parameters in a global parameter manner during the green screen keying process has limitations, resulting in poor accuracy of green screen keying for some scenarios, and making the final obtained foreground image inaccurate.

[0059] To solve the above problems, the embodiment of the present disclosure provides a green screen keying method. During the process of obtaining the opacity map of the foreground image, since the transparency adjustment parameters of at least some pixel points in the first image can be obtained through the target parameter prediction model first, even for an image with uneven illumination and uneven green screen, because the local parameters (transparency adjustment parameters) are determined based on the pixel points, and the opacity map of the foreground image is determined based on these local parameters, and then the foreground image of the first image is calculated by combining the opacity map of the foreground image and the color value of the first image. Therefore, this green screen keying method can calculate an accurate foreground image in any scenario.

[0060] The green screen keying method provided in the embodiment of the present disclosure can be implemented by an electronic device or by a green screen keying device. The green screen keying device can be a functional module or a functional entity in the electronic device for implementing this green screen keying method.

[0061] In the embodiments of the present disclosure, the above-mentioned electronic devices may include: mobile phones, tablet computers, laptop computers, handheld computers, vehicle-mounted terminals, wearable devices, ultra-mobile personal computers (UMPCs), personal digital assistants (PDAs), personal computers (PCs), etc. The embodiments of the present disclosure do not make specific limitations thereto.

[0062] As Figure 1 shown, it is a schematic flowchart of a green screen keying method provided by an embodiment of the present disclosure. The method includes the following steps 101 to 104:

[0063] 101. Obtain a first image.

[0064] Wherein, the first image may be any green screen image.

[0065] 102. Input the first image into a target parameter prediction model, and obtain a target parameter map based on the target parameter prediction model.

[0066] Wherein, the target parameter map includes transparency adjustment parameters of at least some pixel points in the first image. The above-mentioned target parameter prediction model is a convolutional neural network model.

[0067] In some embodiments, the transparency adjustment parameters include: background adjustment parameters, and / or, foreground adjustment parameters.

[0068] The embodiments of the present disclosure provide two opacity calculation methods.

[0069] Wherein, one method is: the opacity of the foreground image is equal to the product of the target difference and the global keying smoothness parameter, where the target difference is the difference between the central color distance of the pixel point and the global keying intensity parameter, and the above-mentioned global keying smoothness parameter can be understood as the transparency adjustment parameter in the embodiments of the present disclosure.

[0070] Another method is: the opacity of the foreground image is equal to the ratio of the first difference and the second difference. The first difference is the difference between the central color distance of the pixel point and the background adjustment parameter (hereinafter may be expressed as t1), and the second difference is the difference between the foreground adjustment parameter (hereinafter may be expressed as t2) and the background adjustment parameter. Wherein, the background adjustment parameter is the color limit range of the pure background, and the foreground adjustment parameter is the color limit range of the pure foreground.

[0071] As Figure 2As shown, it is a schematic diagram of t1, t2, and d in the UV coordinates provided by an embodiment of the present disclosure. The physical meaning of t1 is the projection of the UV coordinate vector of the background image pixel point on the UV coordinate vector of the synthesized image pixel point; the physical meaning of t2 is the projection of the UV coordinate vector of the foreground image pixel point on the UV coordinate vector of the synthesized image pixel point, and the central color distance is represented as d in Figure 2 and the central color distance is the modulus of the UV coordinate vector of the synthesized image pixel point.

[0072] t1 is equal to the ratio of the first vector inner product to the central color distance, and t2 is equal to the ratio of the second vector inner product to the central color distance;

[0073] wherein, the first vector inner product is the inner product of the UV coordinate vector of the background image pixel point and the UV coordinate vector of the synthesized image pixel point; the second vector inner product is the inner product of the UV coordinate vector of the foreground image pixel point and the UV coordinate vector of the synthesized image pixel point.

[0074] The above t1 and / or t2 may refer to the transparency adjustment parameter in the embodiment of the present disclosure.

[0075] To calculate the opacity map of the accurate foreground image, an accurate parameter map needs to be determined first. For example, an accurate parameter map for s needs to be determined, or an accurate parameter map for t1 and / or a parameter map for t2 needs to be determined.

[0076] In the embodiment of the present disclosure, an accurate parameter map for s can be obtained through the target parameter prediction model, or the target parameter prediction model can obtain an accurate parameter map for t1, and / or a parameter map for t2. The training process of the target parameter prediction model for obtaining the parameter map for t1 and the parameter map for t2 is taken as an example for illustration.

[0077] In some embodiments, before executing 102, the initial parameter prediction model can be trained based on the sample information to obtain the target parameter prediction model.

[0078] Wherein, the sample information includes a plurality of sample images and a first parameter map corresponding to each sample image. The sample images include a foreground image, a background image, and a random green screen image. The first parameter map is a parameter map determined according to the UV coordinate vector of the foreground image pixel point, the UV coordinate vector of the random green screen image pixel point, and the central color distance of the pixel point, and / or a parameter map determined according to the UV coordinate vector of the background image pixel point of the random green screen image, the UV coordinate vector of the random green screen image pixel point, and the central color distance of the pixel point; the random green screen image is obtained by fusing the color channels of the foreground image and the background image according to the opacity of the foreground image (i.e., the opacity map of the foreground image); the background image is obtained by superimposing random green on a real picture.

[0079] Through the above sample information, a target parameter prediction model that can predict the target parameter map of an image can be trained, which facilitates quickly obtaining the transparency adjustment parameters corresponding to at least some pixel points in the image during subsequent image processing.

[0080] In some embodiments, the parameter map determined according to the UV coordinate vector of the foreground image pixel points, the UV coordinate vector of the random green screen image pixel points, and the central color distance of the pixel points, that is, the parameter map for t2, and / or the parameter map determined according to the UV coordinate vector of the background image pixel points of the random green screen image, the UV coordinate vector of the random green screen image pixel points, and the central color distance of the pixel points, that is, the parameter map for t1.

[0081] When obtaining the random green screen image, the random green (i.e., the random green image) can be first used to overlay the real image to synthesize an uneven green screen image as the background image, and then an image is obtained as the foreground image. According to the opacity of the foreground image, the color channels of the foreground image and the background image are fused to obtain the synthesized random green screen image. For each synthesized random green screen image, t1 and t2 of at least some pixel points in the corresponding whole image can be set in advance to form the parameter map for t1 and the parameter map for t2 as labels, and training is performed in combination with alpha (the opacity map of the foreground image).

[0082] The above parameter map for t1 can be calculated in advance according to the pixel coordinates of the pixel points in the background image of the random green screen image (i.e., the above uneven green screen image) and the pixel coordinates of the pixel points in the random green screen image. The above parameter map for t2 can be calculated according to the pixel coordinates of the pixel points in the foreground image of the random green screen image and the pixel coordinates of the pixel points in the random green screen image.

[0083] Furthermore, training can be performed in combination with alpha (the opacity map of the foreground image).

[0084] For the initial parameter prediction model, an end-to-end training method can be adopted. During the training process of the initial parameter prediction model, the parameter map for t1 and the parameter map for t2 can be output.

[0085] In some embodiments, during the process of training the initial parameter prediction model based on the sample information to obtain the target parameter prediction model, the following steps (1) to (4) need to be executed at least once to obtain the target parameter prediction model:

[0086] (1) Obtain a target sample image from multiple sample images; wherein, the target sample image can be any one of the multiple sample images.

[0087] Each time the above step (1) is executed, the target sample image obtained from multiple sample images may be different from the sample image obtained last time.

[0088] (2) Input the target sample image into the initial parameter prediction model to obtain the output parameter map of the target sample image output by the initial image processing model.

[0089] (3) Determine the target loss function according to the output parameter map and the first parameter map corresponding to the target sample image.

[0090] In the case where the initial parameter prediction model outputs a parameter map for t1 and / or a parameter map for t2 during the training process:

[0091] The loss function of t1 can be calculated based on the parameter map for t1 output by the initial parameter prediction model and the parameter map for t1 as the label;

[0092] and / or

[0093] The loss function of t2 can be calculated based on the parameter map for t2 output by the initial parameter prediction model and the parameter map for t2 as the label.

[0094] (4) Modify the initial parameter prediction model based on the target loss function.

[0095] In some embodiments, the target loss function includes: the first loss function corresponding to the foreground adjustment parameter (i.e., the loss function of t2); and / or, the second loss function corresponding to the background adjustment parameter (i.e., the loss function based on t1). The initial parameter prediction model can be modified based on the loss function of t1 and / or the loss function of t2.

[0096] In the above embodiments, by calculating the target loss function in each training process to modify the initial parameter prediction model, the finally obtained target parameter prediction model can output an accurate target parameter map based on the image.

[0097] In some embodiments, the first opacity map of the foreground image of the target sample image can be determined based on the output parameter map; the second opacity map of the foreground image of the target sample image can be determined based on the first parameter map corresponding to the target sample image; the third loss function can be determined according to the first opacity map and the second opacity map; the initial parameter prediction model can be modified based on the target loss function and the third loss function.

[0098] Further, after the initial parameter prediction model outputs a parameter map for t1 and a parameter map for t2 during the training process, it can also be combined with the calculated predicted alpha, and this predicted alpha is the above-mentioned first opacity map, which can be expressed as alpha1.

[0099] The above second opacity map is an opacity map (which can be denoted as alpha2) set as a label for the foreground image of the target sample image. Based on alpha1 and alpha2, the alpha loss function (i.e., the above third loss function) can be obtained.

[0100] In some embodiments, the initial parameter prediction model can be corrected based on the third loss function, or the initial parameter prediction model can be corrected based on the target loss function and the third loss function.

[0101] Among them, when correcting the above initial parameter prediction model according to at least two of the above first loss function, second loss function, and third loss function, the weights of the loss functions can be set, and the weighted sum can be obtained based on the weights to obtain the total loss function. The initial parameter prediction model can be corrected based on the total loss function.

[0102] The weights of the above first loss function, second loss function, and third loss function can be set according to actual needs, and the embodiments of the present disclosure do not make limitations. Exemplarily, the weight ratio of the first loss function, second loss function, and third loss function can be 1:1:1.

[0103] In the above embodiments, the initial parameter prediction model can be corrected together by the target loss function and the third loss function, which can ensure that the finally obtained target parameter prediction model can output an accurate target parameter map based on the image, and an accurate opacity map of the foreground image can be calculated based on the output target parameter map.

[0104] In some embodiments, the target parameter map can be directly obtained through the target parameter prediction model. The implementation method is as follows: The first image can be input into the target parameter prediction model, and the target parameter map output by the target parameter preset model can be obtained.

[0105] As Figure 3 shown, it is a schematic diagram of a processing process for obtaining a target parameter map provided by an embodiment of the present disclosure. The obtained first image is an RGB three-channel image, and the RGB three-channel image can be mapped to the YUV space to obtain a Y, U, V three-channel image. Then, the Y, U, V three-channel image is input into the target parameter prediction model, and finally, the target parameter map is output through the target parameter prediction model, that is, Figure 3 the parameter map T1 for t1 and the parameter map T2 for t2 in

[0106] Under normal circumstances, the distribution of the transparency adjustment parameters in the target parameter map obtained for the image in the green screen scene is flat. Therefore, the target parameter map is not sensitive to scale scaling. Therefore, before inputting the image into the target parameter prediction model, the original image can be downsampled to obtain an image with a smaller size. Then, through the above-mentioned target parameter prediction model, a target parameter map with a smaller size (i.e., lower resolution) is first output, and through scale scaling processing, a target parameter map with a larger size (i.e., higher resolution) is obtained. Based on the target parameter map with a larger size, high-precision alpha is calculated. When the target parameter prediction model is a convolutional neural network, the calculation time consumption of the target parameter prediction model is strongly correlated with the scale scaling of the image. Therefore, by processing the image with a smaller size, smaller output and faster calculation can be achieved without losing obvious details.

[0107] It should be noted that the first image in the embodiments of the present disclosure can be the original image or the image after downsampling the original image.

[0108] 103. Determine the target opacity map of the foreground image in the first image according to the transparency adjustment parameters of at least some pixel points and the central color distance of at least some pixel points.

[0109] Among them, the central color distance is determined according to the color value of the pixel point and the color value of the central color, and the central color refers to green.

[0110] In the embodiments of the present disclosure, the target opacity map of the foreground image in the first image can be calculated according to the transparency adjustment parameters of at least some pixel points and the central color distance of at least some pixel points.

[0111] For each pixel point among at least some pixel points in the first image, the opacity value of the foreground image of each pixel point among at least some pixel points in the first image can be calculated according to the transparency adjustment parameters (t1 and / or t2) and the central color distance, that is, the target opacity map of the foreground image is obtained.

[0112] Such as Figure 4 shown, it is a schematic diagram of calculating the target opacity map of the foreground image provided by the embodiments of the present disclosure. As Figure 4 shown, in the process of calculating the alpha of the first image, first, according to the selected central color, the distance from each pixel in the first image to the central color in the UV space can be calculated to form a distance map d. The distance map d includes the central color distance of each pixel in the first image; then, according to the distance map d, the parameter map T1 for t1, and the parameter map T2 for t2, the target opacity map of the first image can be calculated. In Figure 4 the target opacity map of the first image is denoted as α.

[0113] In some embodiments, an initial opacity map of the foreground image in the first image is determined based on the transparency adjustment parameters of at least some pixel points and the central color distance of at least some pixel points. Then, by using the grayscale image of the first image as a guidance image, guided filtering is performed on the initial opacity map to obtain a target opacity map. Since there are usually problems such as missing high-frequency detail information and loss of compressed image quality in the UV channel in real scenes, after obtaining the target parameter map based on the model, the initial opacity map calculated based on the opacity parameters in the target parameter map will also have problems such as missing high-frequency detail information. Therefore, by performing guided filtering on the initial parameter map using the grayscale image of the first image, the grayscale image of the original image input to the target parameter prediction model can be used as a guidance image to increase the details of the opacity estimated based on the model and smooth the compression noise in the UV channel.

[0114] In the green screen matting method provided by the embodiments of the present disclosure, since the transparency adjustment parameters of at least some pixel points in the first image can be obtained first through the target parameter prediction model, even for images with uneven illumination and uneven green screens, because the local parameters (transparency adjustment parameters) are determined based on pixel points and the target opacity map of the foreground image is determined based on these local parameters, the determination of the target opacity map of the foreground image can be applied to any green screen matting scenario, and the calculation result is more accurate.

[0115] 104. Combine the target opacity map and the color values of the first image to calculate the foreground image.

[0116] In some embodiments, in the process of calculating the foreground image based on the target opacity map and the color values of the first image, first, the background in the first image (which can also be called the background image) can be determined according to the target opacity map, and the color mean value of each pixel point in the background image can be calculated to obtain the background color mean value. Then, the fusion opacity coefficient is calculated based on the background color mean value and the color value of the current pixel. Next, based on the fusion opacity coefficient, the color values of the first image, and the color values of the background image, the color values of the foreground image are calculated.

[0117] The color values of the first image include: the color values of the R channel, the color values of the G channel, and the color values of the B channel of the first image.

[0118] In some embodiments, the method of calculating the foreground image by combining the opacity map value of the foreground image and the color values of the first image can be implemented through the following steps 104a to 104c:

[0119] 104a. Obtain the fusion opacity coefficient.

[0120] For each pixel point in at least a part of the pixels in the first image, determining the fusion opacity coefficient includes the following two cases:

[0121] Case 1: When the color value in the G channel is less than or equal to the target color value, determine the fusion opacity coefficient to be 1;

[0122] Case 2: When the color value in the G channel is greater than the target color value, determine the fusion opacity coefficient according to the first color distance and the second color distance. The first color distance is the distance from the color value in the first image to the green limit boundary plane, and the second color distance is the distance from the color mean value in the background image to the green limit boundary plane.

[0123] Wherein, the target color value is half of the sum of the color value in the R channel and the color value in the B channel, and the green limit boundary plane is the plane determined when the color value in the G channel is equal to the target color value.

[0124] In the RGB color space, the limit boundary of the visually perceived green can usually be set as: G = (R + B) / 2;

[0125] Wherein, G can represent the color value in the G channel, R represents the color value in the R channel, and B represents the color value in the B channel.

[0126] Assume that alpha * represents the fusion opacity coefficient, the first color distance is denoted as D f , the second color distance is denoted as Db, then alpha * In the above two cases, it can be expressed as:

[0127]

[0128] As Figure 5 shown, it is a schematic diagram of a green limit boundary plane provided by an embodiment of the present disclosure. Figure 5 The cube in it can be regarded as the RGB color space. The limit boundary of the visually perceived green can usually be roughly set as G = (R + B) / 2. In the RGB color space, it can be expressed as the green limit boundary plane 51 shown in Figure 5 . The RGB coordinates of the 4 vertices of this green limit boundary plane are respectively (0, 0, 0), (0, 0.5, 1), (1, 0.5, 0) and (1, 1, 1).

[0129] 104b. Calculate the color value of the foreground image according to the fusion opacity coefficient, the color value of the first image, and the color value of the background image.

[0130] In some embodiments, calculating the color value of the foreground image based on the fusion opacity coefficient, the color value of the first image, and the color value of the background image may be: calculating the color value of the foreground image according to the fusion opacity coefficient, the color value of the first image, and the color value of the background image.

[0131] The color value of the first image is obtained by weighted summing the color value of the foreground image and the color value of the background image based on the fusion opacity coefficient.

[0132] Wherein, the color value of the first image is equal to the sum of a first product and a second product. The first product is the product of the fusion opacity coefficient and the color value of the foreground image, and the second product is the product of the difference obtained by subtracting the fusion opacity system from 1 and the color value of the background image.

[0133] In the above embodiments, in a green screen scenario, the semi-transparent area, the segmentation edge, and the green reflection area of the foreground object in the foreground image are considered, and the fusion opacity coefficient alpha * and its opacity alpha are different. Therefore, the foreground image is not directly calculated according to the opacity alpha, but the fusion opacity coefficient alpha is first calculated * , and then the foreground image is calculated according to the color value of the foreground image calculated based on alpha * and alpha. The foreground image obtained in this way removes the green color from the semi-transparent area, the segmentation edge, and the green reflection area of the foreground object, so as to obtain a more natural foreground extraction in terms of visual effect, so as to superimpose other backgrounds for material synthesis.

[0134] It should be noted that in the embodiments of the present disclosure, when calculating the target opacity map of the foreground image in the first image and the process of calculating the foreground image, pixel-level operations are performed. That is to say, the above operations are performed for each pixel point of at least some pixel points in the image, so as to obtain the opacity map of the entire foreground image in the first image and calculate the entire foreground image.

[0135] As Figure 6 shown, it is a structural block diagram of a green screen matting device provided by an embodiment of the present disclosure. The device includes:

[0136] An acquisition module 601, configured to acquire a first image;

[0137] A prediction module 602, configured to input the first image into a target parameter prediction model, and obtain a target parameter map based on the target parameter prediction model. The target parameter map includes transparency adjustment parameters of at least some pixel points in the first image;

[0138] A determination module 603, configured to determine a target opacity map of the foreground image in the first image according to the transparency adjustment parameter of the at least partial pixel points and the central color distance of the at least partial pixel points;

[0139] A matte extraction module 604, configured to calculate the foreground image by combining the target opacity map and the color values of the first image.

[0140] Wherein, the central color distance is determined according to the color value of the pixel point and the color value of the central color.

[0141] As an optional implementation manner of an embodiment of the present disclosure, the transparency adjustment parameter includes: a foreground adjustment parameter and / or a background adjustment parameter.

[0142] As an optional implementation manner of an embodiment of the present disclosure, the determination module 603 is specifically configured to: determine an initial opacity map of the foreground image in the first image according to the transparency adjustment parameter of the at least partial pixel points and the central color distance of the at least partial pixel points; use the grayscale image of the first image as a guidance image, and perform guided filtering on the initial opacity map to obtain the target opacity map.

[0143] As an optional implementation manner of an embodiment of the present disclosure,

[0144] The matte extraction module 604 is specifically configured to:

[0145] Obtain a fusion opacity coefficient;

[0146] Calculate the color value of the foreground image according to the fusion opacity coefficient, the color value of the first image, and the color value of the background image;

[0147] Calculate the foreground image according to the color value of the foreground image and the target opacity map.

[0148] As an optional implementation manner of an embodiment of the present disclosure, the determination module 603 is specifically configured to:

[0149] When the color value of the G channel in the first image is less than or equal to the target color value, determine that the fusion opacity coefficient is 1,

[0150] When the color value of the G channel in the first image is greater than the target color value, determine the fusion opacity coefficient according to a first color distance and a second color distance, where the first color distance is the distance from the color value in the first image to the green limit boundary plane, and the second color distance is the distance from the background color mean value to the green limit boundary plane;

[0151] Wherein, the target color value is half of the sum of the color values of the R channel and the B channel, and the green limit boundary plane is the plane determined when the color value of the G channel is equal to the target color value.

[0152] As an optional implementation manner of the embodiments of the present disclosure, the prediction module 602 is further configured to:

[0153] Before inputting the first image into the target parameter prediction model and obtaining the target parameter map based on the target parameter prediction model, train the initial parameter prediction model based on the sample information to obtain the target parameter prediction model;

[0154] The sample information includes multiple sample images and a first parameter map corresponding to each sample image. The sample images include a foreground image, a background image, and a random green screen image. The first parameter map is a parameter map determined according to the UV coordinate vectors of the foreground image pixel points, the UV coordinate vectors of the random green screen image pixel points, and the central color distance of the pixel points, and / or a parameter map determined according to the UV coordinate vectors of the background image pixel points of the random green screen image, the UV coordinate vectors of the random green screen image pixel points, and the central color distance of the pixel points; the random green screen image is obtained by fusing the color channels of the foreground image and the background image according to the opacity of the foreground image; the background image is obtained by superimposing random green on a real picture.

[0155] As an optional implementation manner of the embodiments of the present disclosure, the prediction module 602 is specifically configured to:

[0156] Execute the following steps at least once to obtain the target parameter prediction model:

[0157] Obtain a target sample image from the multiple sample images, input the target sample image into the initial parameter prediction model, and obtain the output parameter map of the target sample image output by the initial image processing model;

[0158] Determine a target loss function according to the output parameter map and the first parameter map corresponding to the target sample image;

[0159] Modify the initial parameter prediction model based on the target loss function;

[0160] Wherein, the target loss function includes:

[0161] The first loss function corresponding to the foreground adjustment parameter;

[0162] And / or,

[0163] The second loss function corresponding to the background adjustment parameter.

[0164] As an optional implementation manner of the embodiments of the present disclosure, the prediction module 602 is specifically configured to:

[0165] Based on the output parameter map, determine a first opacity map of the foreground image of the target sample image;

[0166] Based on the first parameter map corresponding to the target sample image, determine a second opacity map of the foreground image of the target sample image;

[0167] According to the first opacity map and the second opacity map, determine a third loss function;

[0168] Based on the target loss function and the third loss function, correct the initial parameter prediction model.

[0169] Figure 7 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure, which is used to exemplarily illustrate the electronic device for implementing the green screen matting method in the embodiments of the present disclosure, and should not be construed as a specific limitation to the embodiments of the present disclosure.

[0170] As Figure 7 shown, the electronic device 700 may include a processor (such as a central processing unit, a graphics processing unit, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0171] Generally, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or wiredly to exchange data. Although the electronic device 700 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be alternatively implemented or had.

[0172] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processor 701, the functions defined in the green screen keying method provided by the embodiment of the present disclosure can be executed.

[0173] An embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, each process of determining the green screen keying method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0174] Among them, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disc, etc.

[0175] An embodiment of the present invention provides a computer program product. The computer program product stores a computer program. When the computer program is executed by a processor, each process of determining the green screen keying method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0176] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0177] In the present disclosure, the processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0178] In the present disclosure, the memory may include non-permanent memory in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0179] In the present disclosure, the computer-readable medium includes permanent and non-permanent, removable and non-removable storage media. The storage media can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, Phase Change Memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0180] It should be noted that, in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0181] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A green screen keying method, characterized in that Including: Obtain a first image; Input the first image into a target parameter prediction model, and obtain a target parameter map based on the target parameter prediction model, where the target parameter map includes transparency adjustment parameters of at least some pixel points in the first image; Determine a target opacity map of the foreground image of the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points; Combine the target opacity map and the color values of the first image to calculate the foreground image; Wherein, the combining the target opacity map and the color values of the first image to calculate the foreground image includes: Obtain a fusion opacity coefficient; Calculate the color values of the foreground image according to the fusion opacity coefficient, the color values of the first image, and the color values of the background image; The obtaining the fusion opacity coefficient includes: When the color value of the G channel in the first image is less than or equal to a target color value, determine the fusion opacity coefficient to be 1; When the color value of the G channel in the first image is greater than the target color value, determine the fusion opacity coefficient according to a first color distance and a second color distance, where the first color distance is the distance from the color value in the first image to the green limit boundary plane, and the second color distance is the distance from the background color mean value to the green limit boundary plane; Wherein, the target color value is half of the sum of the color values of the R channel and the B channel, and the green limit boundary plane is the plane determined when the color value of the G channel is equal to the target color value.

2. The method according to claim 1, wherein The transparency adjustment parameters include: foreground adjustment parameters and / or background adjustment parameters.

3. The method according to claim 1, characterized in that The determining the target opacity map of the foreground image in the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points includes: Determine an initial opacity map of the foreground image in the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points; Use the grayscale image of the first image as a guiding image, and perform guided filtering on the initial opacity map to obtain the target opacity map.

4. The method according to claim 1, wherein Before inputting the first image into the target parameter prediction model and obtaining the target parameter map based on the target parameter prediction model, the method further includes: Train an initial parameter prediction model based on sample information to obtain the target parameter prediction model; The sample information includes a plurality of sample images and a first parameter map corresponding to each sample image. The sample images include a foreground image, a background image, and a random green screen image. The first parameter map is a parameter map determined based on the UV coordinate vectors of the pixel points of the foreground image, the UV coordinate vectors of the pixel points of the random green screen image, and the central color distance of the pixel points, and / or a parameter map determined based on the UV coordinate vectors of the background image pixel points of the random green screen image, the UV coordinate vectors of the pixel points of the random green screen image, and the central color distance of the pixel points. The random green screen image is obtained by fusing the color channels of the foreground image and the background image according to the opacity of the foreground image. The background image is obtained by superimposing random green on a real picture.

5. The method according to claim 4, wherein Training the initial parameter prediction model based on the sample information to obtain the target parameter prediction model includes: Performing the following steps at least once to obtain the target parameter prediction model: Obtaining a target sample image from the plurality of sample images, inputting the target sample image into the initial parameter prediction model, and obtaining an output parameter map of the target sample image output by the initial image processing model; Determining a target loss function according to the output parameter map and the first parameter map corresponding to the target sample image; Modifying the initial parameter prediction model based on the target loss function; Wherein, the target loss function includes: A first loss function corresponding to the foreground adjustment parameter; And / or A second loss function corresponding to the background adjustment parameter.

6. A green screen keying device, characterized in that, Including: An acquisition module for acquiring a first image; A prediction module for inputting the first image into the target parameter prediction model and obtaining a target parameter map based on the target parameter prediction model. The target parameter map includes transparency adjustment parameters of at least some pixel points in the first image; A determination module for determining a target opacity map of the foreground image in the first image according to the transparency adjustment parameters of the at least some pixel points and the central color distance of the at least some pixel points; A matting module for calculating the foreground image by combining the target opacity map and the color values of the first image; Wherein, the calculating the foreground image by combining the target opacity map and the color values of the first image includes: Obtaining a fusion opacity coefficient; Calculating the color value of the foreground image according to the fusion opacity coefficient, the color value of the first image, and the color value of the background image; The obtaining the fusion opacity coefficient includes: When the color value of the G channel in the first image is less than or equal to the target color value, determining the fusion opacity coefficient to be 1; When the color value of the G channel in the first image is greater than the target color value, determining the fusion opacity coefficient according to a first color distance and a second color distance. The first color distance is the distance from the color value in the first image to the green limit boundary plane, and the second color distance is the distance from the background color mean to the green limit boundary plane. Wherein, the target color value is half of the sum of the color values of the R channel and the B channel, and the green limit boundary plane is the plane determined when the color value of the G channel is equal to the target color value.

7. An electronic device, characterized in that, Comprising: A processor, a memory, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, it implements the green screen keying method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Comprising: A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the green screen keying method according to any one of claims 1 to 5.

Images