Image processing method and apparatus, and device and storage medium

Abstract

Provided in the embodiments of the present description are an image processing method and apparatus, and a device and a storage medium. The method comprises: acquiring an image to be matted which has a pure-color background; determining a ternary graph corresponding to the image to be matted; selecting a target image from an A-channel image and a B-channel image, in a LAB color space, of the image to be matted, wherein the color of the target image is closer to the background color of the image to be matted; by taking the target image as a guide image, performing guided filtering processing on the ternary graph to obtain an Alpha image; and on the basis of the Alpha image, performing matting to obtain a foreground area from the image to be matted. By means of the solution, details of an edge part of a foreground area can be retained to the greatest extent; in addition, the processing speed of a matting process can be increased.

Description

Image processing method, device, equipment and storage medium Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Art

[0002] Cutout technology is widely used in various fields. Take green screen cutout as an example. In scenes such as film and television shooting and live broadcast shooting, a solid color screen of a specific color (for example, green, blue, red, etc.) is usually used as the background to shoot the subject to obtain the original image. Then, the background part of the specific color is identified from the original image and replaced with the background the user wants, so that some special effects images or images in specific scenes can be obtained without the need for additional special scenes. In order to achieve a realistic effect, it is often required that when the background area of ​​the image is replaced with a new background, the edge parts of the foreground area and the background area in the image are better processed, so that the filtering of the foreground area and the background area is more natural and there is no sense of disobedience. However, the image obtained by using the existing cutout algorithm for cutout processing after replacing the new background often has poor effect on the edge parts of the foreground area and the background area, the edge details are not retained, the transition is not natural enough, and the display effect of the image is seriously affected.

[0003] Summary of the Invention

[0004] Based on this, this specification provides an image processing method, apparatus, device and storage medium.

[0005] According to a first aspect of the embodiments of this specification, there is provided an image processing method, the method comprising:

[0006] Get the image to be cut out with a solid color background;

[0007] Determine a ternary image corresponding to the image to be cutout;

[0008] Selecting a target image whose color is closer to the background color of the image to be cut out from the A channel image and the B channel image of the image to be cut out in the LAB color space;

[0009] Using the target image as a guide image, performing guided filtering on the ternary image to obtain an Alpha image;

[0010] A foreground area is cut out from the image to be cut out based on the Alpha image.

[0011] According to a second aspect of the embodiments of this specification, there is provided an image processing apparatus, the apparatus comprising:

[0012] An acquisition module is used to obtain an image to be cut out with a solid color background;

[0013] A ternary image determination module, which determines the ternary image corresponding to the image to be cut out;

[0014] a filtering module configured to select a target image whose color is closer to the background color of the image to be cut out from the A-channel image and the B-channel image of the image to be cut out in the LAB color space; and perform guided filtering on the ternary image using the target image as a guide image to obtain an Alpha image;

[0015] The cutout module is used to cut out a foreground area from the image to be cutout based on the Alpha image.

[0016] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory for execution by the processor, wherein the processor implements the method mentioned in the first aspect when executing the computer program.

[0017] According to a fourth aspect of the embodiments of this specification, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and when the computer program is executed, the method mentioned in the first aspect is implemented.

[0018] When applying the embodiment of the present specification, when performing a cutout process on an image to be cutout, the ternary map of the image to be cutout can be determined first. When obtaining the Alpha image corresponding to the image to be cutout based on the ternary map, in order to retain the edge detail information of the foreground area and the background area as much as possible, the ternary map can be filtered by guided filtering to obtain the Alpha image. In order to retain the image details to the greatest extent, when filtering the ternary map, a target image that is closer to the background color can be selected from the A channel image and the B channel image of the image to be cutout in the LAB color space based on the background color of the image to be cutout, and then the target image is used as a guide image to perform a guided filtering process on the ternary map, so that the filtered ternary map can retain the texture of the target image (i.e., the background area of ​​the image to be cutout), that is, the details of the edge part can be better retained to obtain a more refined Alpha image. The Alpha image can then be used to deduct the foreground area from the image to be cutout for subsequent synthesis. In addition, since the target image selected from the LAB color space that is closer to the background color is a single-channel image, when the single-channel image is used as a guide image to perform guided filtering on the ternary image, compared with the related art of using an RGB three-channel image as a guide image to perform guided filtering on the ternary image, the amount of calculation can be reduced and the processing speed in the filtering process can be greatly improved.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0021] FIG1 is a schematic diagram of an image to be cut out, a ternary image, and an Alpha image.

[0022] FIG2 is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0023] FIG3 is a flow chart of an image processing method according to an embodiment of the present disclosure.

[0024] 4-6 are schematic diagrams of image processing methods according to various embodiments of this specification.

[0025] FIG7 is a schematic diagram of a green screen cutout method according to an embodiment of the present disclosure.

[0026] FIG8 is a schematic diagram of an image processing device according to an embodiment of this specification.

[0027] FIG9 is a schematic diagram of an electronic device according to an embodiment of this specification. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0029] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0031] Cutout technology is widely used in various fields. For example, in film and television shooting, live broadcasting, and other scenes, a solid-color screen of a specific color (such as green, blue, or red) is often used as a background to capture the subject and obtain the original image. The background portion of the specific color is then identified from the original image and replaced with the user's desired background. This allows for the creation of special effects images or images of specific scenes without the need for additional dedicated scenes.

[0032] The key to the cutout technology is to accurately identify the foreground and background areas from the image to be cutout. For any image to be cutout I, the subject part can be called the foreground F, and the rest is the background B. Then the image to be cutout I can be regarded as a weighted fusion of F and B: I = Alpha*F+(1-Alpha)*B, where Alpha is a continuous value between [0, 1], which can be understood as the probability that each pixel belongs to the foreground, or the opacity of the foreground image (that is, the higher the probability that the pixel belongs to the foreground, the more opaque it is). The task of cutout is to find the appropriate weight Alpha for each pixel, that is, to obtain an Alpha image corresponding to the image to be cutout. The pixel value of each pixel in the Alpha image represents the probability that the pixel at the corresponding pixel position of the image to be cutout is the foreground. Then, the foreground area can be cut out from the image to be cutout based on the Alpha image.

[0033] When determining the Alpha image, the ternary map (Trimap) corresponding to the image to be cut out can usually be determined. The ternary map is an image of the same size as the image to be cut out. The ternary map contains only three pixel values. For example, a pixel with a pixel value of 0 is a background pixel that has been determined, and a pixel with a pixel value of 1 is a foreground pixel that has been determined. A pixel value between 0 and 1 is used to represent the edge of the foreground and background. This part is an unknown area, which requires further refinement to determine the probability of it belonging to the foreground or background. After obtaining the ternary map, the ternary map can usually be processed, that is, the unknown area in the ternary map is finely divided to determine the probability of it belonging to the foreground or background, and then the Alpha image can be obtained. As shown in Figure 1, the three images on the left, middle, and right are the image to be cut out, the ternary map, and the Alpha image, respectively.

[0034] In related technologies, when obtaining an Alpha image based on a ternary image, it is often impossible to retain the details of the edge of the foreground area well and obtain a more refined Alpha image, so that the edge of the foreground area in the subsequent image synthesized based on the foreground area and the new background often has a poor effect, the edge details are not retained, and the transition is not natural enough, which seriously affects the effect of the final synthesized image.

[0035] Based on this, an embodiment of the present application provides an image processing method, as shown in FIG2 , when performing a cutout process on an image to be cutout, a ternary map of the image to be cutout can be first determined, and when obtaining an Alpha image corresponding to the image to be cutout based on the ternary map, in order to retain the edge detail information of the foreground area and the background area as much as possible, the ternary map can be filtered by using a guided filtering method to obtain the Alpha image. In order to retain the image details to the greatest extent, when filtering the ternary map, a target image that is closer to the background color can be selected from the A channel image and the B channel image of the image to be cutout in the LAB color space based on the background color of the image to be cutout, and then the target image is used as a guide image to perform a guided filtering process on the ternary map, so that the filtered ternary map can retain the texture of the target image (i.e., the background area of ​​the image to be cutout), that is, the details of the edge part can be better retained to obtain a finer Alpha image. The Alpha image can then be used to deduct the foreground area from the image to be cutout for subsequent synthesis.

[0036] Through the cutout scheme provided by the embodiment of the present application, on the one hand, a target image that is close to the background color is selected from the A-channel image and the B-channel image of the image to be cutout in the LAB color space as a guide image, and the ternary image is subjected to guided filtering processing, so that the Alpha image obtained by filtering can better retain the image details at the foreground and background edges of the image to be cutout, and is more refined, so that the foreground edge effect in the subsequent synthesized image is also better, and the transition is more natural. On the other hand, since the target image that is closer to the background color selected from the LAB color space is a single-channel image, when the single-channel image is used as the guide image to perform guided filtering processing on the ternary image, compared with the related art of using the RGB three-channel image as the guide image to perform guided filtering processing on the ternary image, the amount of calculation can be reduced and the processing speed in the filtering process can be greatly improved, so that the cutout processing scheme can be applied to some scenes with high real-time requirements, such as live video shooting (for example, replacing the background of the live video and then pushing it to the stream).

[0037] The image processing method provided in the embodiments of the present application can be executed by various electronic devices, such as mobile phones, computers, various live broadcast devices, or cloud servers, etc. For example, the image processing method can be integrated into a certain cutout APP or cutout SDK. As long as the cutout APP or cutout SDK is installed on the device, the above image processing method can be implemented.

[0038] As shown in FIG3 , the image processing method may include the following steps:

[0039] S302, obtaining an image to be cut out with a solid color background;

[0040] In step S302, an image to be cut out with a solid color background may be obtained, wherein the solid color background may be a green background, a blue background, a red background, etc. The image to be cut out may be an image directly output by a camera, for example, an image in NV12 format, or an image after a certain format conversion processing is performed on the image output by the camera, for example, an image in NV12 format is converted into RGB format or LAB format, and this embodiment of the application does not impose any limitation.

[0041] S304, determining a ternary image corresponding to the image to be cut out;

[0042] In step S304, after obtaining the image to be cut out, a ternary map corresponding to the image to be cut out can be determined. For example, the foreground area and background area of ​​the image to be cut out can be distinguished by algorithms such as color recognition and foreground and background segmentation to obtain the ternary map.

[0043] S306, selecting a target image whose color is closer to the background color of the image to be cut out from the A channel image and the B channel image of the image to be cut out in the LAB color space;

[0044] In step S306, the A-channel image and B-channel image of the image to be cut out in the LAB color space can be determined. If the image to be cut out is an image in the LAB format, the A-channel image and B-channel image in the LAB color space can be determined directly based on the LAB format image. If the image to be cut out is an image in a non-LAB format, such as an image in the RGB, YUV, or HSV format, the image to be cut out can be first converted to an image in the LAB format, and then the A-channel image and B-channel image in the LAB color space can be determined.

[0045] Among them, in the LAB color space, each image can include three channels: L, A, and B. The L channel represents the brightness of the image, and the A and B channels represent the color of the image. The A channel represents from green to gray to magenta, and the B channel represents from blue to gray to yellow.

[0046] Then, a target image that is closer to the background color of the image to be cut out can be selected from the above A-channel image and B-channel image. For example, if the background of the image to be cut out is green, the A-channel image representing green can be selected as the target image.

[0047] S308, using the target image as a guide image, performing guided filtering on the ternary image to obtain an Alpha image;

[0048] In step S308, after determining the target image, the target image can be used as a guide image to perform guided filtering on the ternary image to obtain an Alpha image. Guided filtering is a filtering method with a good edge-preserving effect. When processing an image, it can keep the edges from being smoothed to the greatest extent possible, that is, retain the original gradient in the image. Guided filtering usually includes a guide image and an input image. In the process of processing the input image using the guide image, the purpose of guided filtering is to make the gradient of the filtered image (i.e., the output image) as similar as possible to the guide image, and at the same time make the grayscale (brightness) of the output image similar to the input image, so as to preserve the edges and filter out noise.

[0049] Since the one that is closer to the background color of the image to be cut out is selected from the A channel image and the B channel image in the LAB color space of the image to be cut out as the target image, that is, the gradient of the target image actually also reflects the texture of the edge of the background area of ​​the image to be cut out (that is, the texture of the edge of the foreground area). With the target image as the guide image, the ternary image is subjected to guided filtering, that is, the Alpha image obtained by the filtering process also retains the texture of the edge of the foreground area of ​​the image to be cut out to a certain extent. Therefore, after the image to be cut out is cut out using the Alpha image, the details of the edge of the foreground area in the image to be cut out can be better retained.

[0050] In addition, since the guide image is a color channel image selected from the LAB color space, it is a single-channel image. Compared with the related art that directly uses three-channel images such as RGB and HSV as the guide image for filtering processing, it can greatly reduce the amount of calculation and improve the processing speed.

[0051] S310: Extract a foreground area from the image to be cut out based on the Alpha image.

[0052] In step S310, after the Alpha image corresponding to the image to be cut out is determined, the foreground area can be subtracted from the image to be cut out based on the Alpha image.

[0053] In green screen cutout scenarios, the background is often green or blue. Therefore, in some embodiments, if the background of the image to be cutout is green, the target image selected as the guide image is the A channel image of the image to be cutout in the LAB color space. If the background of the image to be cutout is blue, the target image selected as the guide image is the B channel image of the image to be cutout in the LAB color space.

[0054] In the related art, when performing cutout processing on a cutout image, most processes are performed directly on the YUV format output by the camera, or convert it into an RGB or HSV format before processing. Compared to LAB format images, YUV, RGB, and HSV format images have a narrower color gamut, resulting in lower segmentation accuracy when segmenting the foreground and background regions of the cutout image. For example, shadows in the background are often difficult to identify and are more likely to be identified as foreground regions, leading to inaccurate segmentation of the foreground and background regions. Furthermore, RGB format images represent a single color using the colors of the three R, G, and B channels. This representation method cannot strictly segment green, and is therefore prone to misidentification when segmenting the foreground and background regions of the cutout image. Since the foreground and background regions cannot be accurately identified during segmentation, the alpha image determined based on the segmentation results is also inaccurate. As a result, the cutout foreground region may also contain background components, resulting in poor image quality when subsequently synthesizing the foreground region and the new background.

[0055] In order to more accurately segment the foreground area and background area of ​​the image to be cut out, in some embodiments, as shown in Figure 4, if the format of the image to be cut out is any one of RGB format, YUV format, and HSV format, when determining the ternary graph corresponding to the image to be cut out, the image to be cut out can be first converted into an image in LAB format, and then the image in LAB format can be segmented into the foreground area and the background area to obtain a binary image. Since the color gamut of the image in LAB format is wider, after the image to be cut out is converted into an image in LAB format, segmentation processing can be performed, and the foreground area and the background area can be more accurately segmented. Wherein, when the image in LAB format is segmented, the Otsu threshold algorithm can be adopted to segment the image to determine the foreground area and the background area, or other algorithms with similar functions can be adopted, and the embodiments of the present application are not limited. After the image in LAB format is segmented, a binary image can be obtained. Wherein, the binary image includes two pixel values, for example, 1 is used to represent the foreground and 0 is used to represent the background. After obtaining the binary image, feathering can be performed on the edge regions of the foreground and background regions in the binary image, thereby obtaining a ternary image of the image to be cut out. The feathering process involves blurring the edges of the foreground and background. For example, the edge regions can be Gaussian blurred using a Gaussian blur algorithm. Of course, other blurring methods may also be used, and this is not a limitation of the present embodiment.

[0056] In some embodiments, in order to reduce the amount of computation required when segmenting the foreground and background, when segmenting the foreground and background areas of an image in LAB format to obtain a binary image, a target image that is closer to the background color of the image to be cut out can be selected from the A-channel image and the B-channel image of the LAB format image, and then the foreground and background areas of the target image can be segmented to obtain a binary image. For example, if the background of the image to be cut out is green, the A-channel image can be segmented to separate the foreground and background areas. If the background of the image to be cut out is blue, the B-channel image can be segmented to separate the foreground and background areas. Since the green color is all in the A-channel image and the blue color is all in the B-channel image in the LAB format, only the A-channel image or the B-channel image that is closer to the background color is segmented, and the background part can be accurately segmented out, and the rest is the foreground. Therefore, only the single-channel image needs to be segmented, which reduces the amount of computation and improves processing efficiency. It is more suitable for scenes with higher real-time requirements such as live broadcast cutouts.

[0057] In the related art, when performing cutout processing, users are typically required to enter certain cutout parameters, such as feather radius, contrast, brightness, opacity, and so on. The image is then processed based on these user-entered cutout parameters, which is rather cumbersome. Taking the feather radius as an example, users are required to professionally evaluate the feather radius and enter an appropriate feather radius. Feathering processing is then performed based on this input. This approach is rather cumbersome and requires a high level of user expertise. The larger the feather radius, the larger the edge area to be blurred. Since users are required to set the feather radius based on the actual conditions of the image to be cutout, this is cumbersome and inconvenient. To avoid this problem, in some embodiments, when feathering a binary image to obtain a ternary image, the feather radius can be automatically determined based on attribute information of the image to be cutout. This attribute information can include the resolution of the image to be cutout, whether the image to be cutout contains moving objects, and the brightness information of the image to be cutout. For example, if the resolution of the image to be cutout is low, the feather radius can be set larger; otherwise, it can be set smaller. If the image to be cut out includes a moving object, the feathering radius can be set larger, otherwise smaller. After the feathering radius is automatically determined based on the attribute information of the image to be cut out, the edge areas of the foreground and background areas in the binary image can be feathered based on the feathering radius to obtain the above-mentioned ternary image.

[0058] In the embodiment of the present application, the feathering radius adapted to the image to be cut out is automatically determined based on the attribute information of the image to be cut out, and the user does not need to set the feathering radius. Similarly, the other cutout parameters are similar. In the embodiment of the present application, the relevant cutout parameters can be automatically determined based on the characteristics of the image to be cut out, and the user does not need to make additional input, thereby realizing one-click cutout, which is more convenient and quick.

[0059] During the image formation process, background reflections often cause some background colors to be sandwiched between the foreground area. Taking an image captured against a green screen as an example, there will often be green reflections at the edges of the foreground and background, or some areas of the foreground will also have a green cast. This can cause green reflections or green casts in some areas of the fused image after the foreground area is fused with the new background image, affecting the image fusion effect. To alleviate the above problem, in some embodiments, as shown in FIG5 , after the foreground area is extracted from the image to be cut out based on the alpha image, background color suppression processing can be performed on the foreground area. For example, a target pixel can be determined from the pixels in the foreground area. The target pixel is a pixel whose pixel value in a target color channel is higher than a preset threshold. The target color channel is the color channel of the R, G, and B color channels that is closer to the background color of the image to be cut out. For example, if the background of the image to be cut out is green, the target color channel is the G channel. If the background of the image to be cut out is blue, the target color channel is the B channel.

[0060] Then, the pixel value of the target color channel of the target pixel point can be suppressed to reduce the pixel value of the target color channel, thereby obtaining a foreground area after removing the background color. By identifying target pixels in the foreground area whose color components close to the background color are greater than a preset threshold, and suppressing the color components of these target pixels close to the background color, the interference of the background color in the foreground area can be eliminated.

[0061] For example, taking the background of the image to be cut out as a green background, target pixels whose G channel pixel values ​​are greater than a preset threshold can be determined from the foreground image, where the preset threshold can be flexibly set based on the actual application scenario. The G channel pixel values ​​of these target pixels can then be suppressed, for example, by reducing them by a certain proportion or subtracting a preset value to reduce the G channel pixel values. Through the above-mentioned green removal processing steps, the greenish cast and green reflection in the foreground area can be eliminated as much as possible.

[0062] Typically, after extracting the foreground region from the image to be cut out, the purpose is to fuse it with a new background image to synthesize a new image. Therefore, in some embodiments, after extracting the foreground region from the image to be cut out based on the alpha image, the foreground region can be fused with the target background image to obtain a fused image.

[0063] Considering that the foreground area and the target background image are images captured under different lighting conditions, their color temperature and color gamut may be inconsistent. If the two are directly fused, the resulting fused image may not be natural enough and may have a certain sense of incongruity. In order to alleviate the above problem, in some embodiments, as shown in Figure 6, after the foreground area and the target background image are fused to obtain a fused image, the white balance correction coefficient corresponding to the target color channel of the fused image can be determined, wherein the target color channel is the color channel of the three color channels R, G, and B that is closer to the background color of the image to be cut out. For example, if the background of the image to be cut out is a green background, the target color channel is the G channel, and if the background of the image to be cut out is a blue background, the target color channel is the B channel. The pixel value of the target color channel of the fused image can then be corrected based on the white balance correction coefficient. In particular, when determining the white balance correction coefficient corresponding to the target color channel of the fused image, a grayscale world algorithm can be used, or other algorithms with similar functions can be used.

[0064] For example, after obtaining a fused image, the white balance correction coefficients corresponding to the R, G, and B color channels of the fused image can be determined based on the grayscale world algorithm. The grayscale world algorithm is based on the grayscale world assumption, which states that for an image with significant color variation, the average values ​​of the R, G, and B color components tend to the same grayscale value, K. Physically speaking, the grayscale world method assumes that the average reflectance of light from natural scenes is generally a constant, approximately "gray." The color balance algorithm enforces this assumption on the image being processed, eliminating the effects of ambient light and retrieving the original scene image.

[0065] When determining the white balance correction coefficients corresponding to the three color channels R, G, and B based on the grayscale world algorithm, the K value can be determined first. The K value can be a fixed value. For example, assuming that the pixel values ​​of the fused image are 0-255, the K value can be 128. Alternatively, the K value can be the average value of the three color channels R, G, and B of the fused image, for example, K = (R+G+B) / 3.

[0066] After determining the K value, the white balance correction coefficient corresponding to each color channel can be obtained based on the K value:

[0067] For example, the white balance correction coefficient of the R channel gainR=K / R;

[0068] White balance correction coefficient of G channel gainG=K / G;

[0069] White balance correction coefficient of B channel gainB=K / B;

[0070] Among them, if the background of the image to be cut out is green, the pixel value of the G channel of each pixel point of the fused image can be corrected based on the white balance correction coefficient gainG of the G channel. If the background of the image to be cut out is blue, the pixel value of the B channel of each pixel point of the fused image can be corrected based on the white balance correction coefficient gainB of the B channel, thereby reducing the sense of incongruity of the fused image caused by the difference in lighting conditions between the foreground area and the target background image.

[0071] In some embodiments, if the white balance correction coefficient is determined based on a grayscale world algorithm, given that the grayscale world algorithm assumes the presence of rich colors in the image, if the fused image includes large areas of monochrome, the white balance correction coefficient determined based on the grayscale world algorithm may also be inaccurate. Therefore, before correcting the pixel values ​​of the target color channel of the fused image, it can be determined whether the fused image includes large areas of monochrome. If so, no correction is performed. If not, the pixel values ​​of the target color channel of the fused image are corrected based on the white balance correction coefficient determined by the grayscale world algorithm to ensure the corrected image is displayed properly. Specifically, when determining whether the fused image includes large areas of monochrome, the proportions of the color components of the R, G, and B color channels in the fused image can be calculated. For example, the proportion of the R channel is the ratio of the sum of the color components of the R channel of each pixel to the sum of the color components of the R, G, and B color channels of each pixel. The same applies to the other two channels. After determining the proportions of the three color channels, it can be determined whether the proportion of any one of the color channels is greater than a preset proportion. If so, it indicates the presence of a large area of ​​monochrome. If the proportion of any color channel is less than the preset proportion, it means that there is no large monochrome area. At this time, it means that the white balance correction coefficient determined based on the gray world algorithm is accurate, and the white balance correction coefficient can be used to correct the pixel value of the target color channel of the fused image.

[0072] In order to further explain the image processing method of the present application, a specific embodiment is provided below for explanation.

[0073] During film and television shooting and live broadcasting, considering that the human eye or camera is more sensitive to green, a green screen is usually used as the background for shooting. The green screen cutout algorithm is then used to extract the foreground area from the captured image and fuse it with the target background image to obtain the image after the background is replaced.

[0074] Some existing green screen cutout algorithms produce fused images with incomplete details at the edges of the foreground and background, unnatural transitions, and green reflections and green casts in the foreground. Furthermore, in poor lighting conditions, shadows within the green background are often difficult to accurately identify, leading to inaccurate foreground and background segmentation. Consequently, these algorithms require strict lighting conditions during the shooting process. Furthermore, cutout parameters such as the feather radius must be manually configured, which is cumbersome and lacks intelligent control.

[0075] In order to solve the above problem, this embodiment provides a green screen cutout algorithm, as shown in FIG7 . The specific processing process of the algorithm is as follows:

[0076] 1. Obtain the image to be cut out in NV12 format captured by the camera, and then convert the image to be cut out in NV12 format into an RGB format image and a LAB format image respectively.

[0077] 2. Segment the foreground and background regions of the LAB image (for example, using the Otsu thresholding algorithm) to obtain a binary image. Compared to NV12 or RGB formats, LAB images have a wider color gamut, allowing for more accurate foreground and background segmentation.

[0078] 3. Automatically determine the feathering radius based on the resolution of the image to be cut out, whether it contains moving objects, brightness and other information, and feather the edge part of the foreground area in the binary image based on the feathering radius to obtain the ternary image corresponding to the image to be cut out.

[0079] 4. From the color channel images of the LAB format image (i.e., the A channel image and the B channel image), obtain the A channel image that is closer to the green background, use the A channel image as the guide image, and perform guided filtering on the ternary image to obtain the Alpha image corresponding to the image to be cut out. By selecting the A channel image as the guide image and performing guided filtering on the ternary image, the filtered ternary image can retain the texture of the edge of the background area (i.e., the edge of the foreground area), that is, the edge details of the foreground area can be retained to the greatest extent. Moreover, the guide image is a single-channel image, which can reduce the amount of calculation and greatly improve the processing speed during the filtering process.

[0080] 5. Extract the foreground area from the image to be cut out based on the Alpha image.

[0081] 6. Obtain the RGB format image corresponding to the foreground area from the RGB format image of the image to be cut out, and select the target pixels whose G channel pixel values ​​are greater than the preset threshold from the pixels in the foreground area. Suppress the G channel pixel values ​​of these target pixels to reduce the pixel values ​​of the G channel, that is, perform green removal processing to avoid green reflections, greening, etc. in the foreground image.

[0082] 7. Fuse the processed foreground area and the new background image to obtain a fused image.

[0083] 8. Use the grayscale world algorithm to determine the white balance correction coefficient corresponding to the G channel in the fused image. Use this white balance correction coefficient to correct the G channel pixel value of the pixel point in the fused image to eliminate the sense of incongruity between the foreground area and the new background image due to different lighting conditions.

[0084] The above-mentioned green screen cutout algorithm can preserve the details of the edge of the foreground area to a great extent, and the green edge removal effect is obvious. By removing the green in the foreground area, the original true color of the object is restored to a great extent, avoiding the problem of green cast and green reflection in the foreground area. And because the foreground and background are segmented in the LAB color space, more accurate segmentation results can be obtained. It is not demanding on the lighting scenes of various live broadcasts and film and television shooting scenes, which makes it convenient for users to easily and quickly set up live broadcast environments. The algorithm adaptively finds the optimal cutout parameters under the current imaging conditions (for example, the feathering eight scenes), eliminating the need for professionals to repeatedly adjust the parameters, and achieving one-click fool cutout.

[0085] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0086] Corresponding to the image processing method embodiment provided in the embodiment of the present application, the embodiment of the present application further provides an image processing device, as shown in FIG8 , wherein the device 80 includes:

[0087] An acquisition module 82 is used to acquire an image to be cut out with a solid color background;

[0088] A ternary image determination module 84 is used to determine the ternary image corresponding to the image to be cut out;

[0089] A filtering module 86 is configured to select a target image whose color is closer to the background color of the image to be cut out from the A-channel image and the B-channel image of the image to be cut out in the LAB color space; perform guided filtering on the ternary image using the target image as a guide image to obtain an Alpha image;

[0090] The cutout module 88 is configured to cut out a foreground area from the image to be cutout based on the Alpha image.

[0091] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned image processing method, and will not be repeated here.

[0092] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0093] From a hardware perspective, Figure 9 shows a hardware structure diagram of a device that includes an image processing apparatus according to an embodiment of this specification. In addition to the processor 92 and memory 94 shown in Figure 9 , the device may also include other hardware, such as a forwarding chip responsible for message processing. From a hardware perspective, the device may also be distributed, potentially including multiple interface cards to enable hardware-level message processing expansion. The memory 94 stores computer instructions, and when the processor 92 executes these computer instructions, it implements the image processing method described in any of the aforementioned embodiments.

[0094] Since the portion of the embodiment of this specification that contributes to the prior art or the entire or partial technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes a number of instructions for causing a terminal device to execute all or part of the steps of each method of the embodiment of this specification. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0095] Those skilled in the art will readily recognize other implementations of the embodiments of the present invention after considering the specification and practicing the instructions disclosed herein. The embodiments of the present invention are intended to cover any variations, uses, or adaptations of the embodiments of the present invention that follow the general principles of the embodiments of the present invention and include common knowledge or customary techniques in the art not disclosed in the embodiments of the present invention. The description and examples are to be considered as exemplary only, and the true scope and spirit of the embodiments of the present invention are indicated by the following claims.

[0096] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

[0097] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be cut out with a solid color background; Determine a ternary image corresponding to the image to be cutout; Selecting a target image whose color is closer to the background color of the image to be cut out from the A channel image and the B channel image of the image to be cut out in the LAB color space; Using the target image as a guide image, performing guided filtering on the ternary image to obtain an Alpha image; A foreground area is cut out from the image to be cut out based on the Alpha image.

2. The method according to claim 1, characterized in that If the background of the image to be cut out is a green background, the target image is the A channel image; if the background of the image to be cut out is a blue background, the target image is the B channel image.

3. The method according to claim 1, characterized in that: The format of the image to be cut out includes any one of the following: RGB format, YUV format, and HSV format. The determining of the ternary image corresponding to the image to be cut out includes: Convert the image to be cutout into an image in LAB format; Performing segmentation processing on the LAB format image into a foreground area and a background area to obtain a binary image; The edge regions of the foreground region and the background region in the binary image are subjected to feathering processing to obtain the ternary image.

4. The method according to claim 3, characterized in that The LAB format image is segmented into a foreground area and a background area to obtain a binary image, including: Selecting a target image whose color is closer to the background color of the image to be cut out from the A channel image and the B channel image of the LAB format image; The foreground area and the background area of ​​the target image are segmented to obtain the binary image. picture.

5. The method according to claim 3, characterized in that: The edge regions of the foreground region and the background region in the binary image are feathered to obtain the ternary image, including: Determining a feathering radius based on attribute information of the image to be cut out, wherein the attribute information includes one or more of the following: a resolution of the image to be cut out, whether the image to be cut out includes a moving object, and brightness of the image to be cut out; Based on the feathering radius, feathering processing is performed on the edge areas of the foreground area and the background area in the image to be cut out to obtain the ternary image.

6. The method according to claim 1, characterized in that After extracting the foreground area from the image to be cut out based on the Alpha image, the method further includes: Determine a target pixel point from the pixel points of the foreground area, the pixel value of the target color channel of the target pixel point is higher than a preset threshold, and the target color channel is a color channel of the three color channels R, G, and B that is closer to the background color of the image to be cut out; The pixel value of the target color channel of the target pixel point is suppressed to reduce the pixel value of the target color channel.

7. The method according to any one of claims 1 to 6, characterized in that: After extracting the foreground image from the image to be cut out based on the Alpha image, the method further includes: The foreground area is fused with the target background image to obtain a fused image.

8. The method according to claim 7, characterized in that The method further comprises: Determine a white balance correction coefficient corresponding to a target color channel of the fused image, wherein the target color channel is a color channel of the three color channels R, G, and B that is closer in color to a background color of the image to be cut out; Correction processing is performed on the pixel values ​​of the target color channel of the fused image based on the white balance correction coefficient.

9. The method according to claim 6 or 8, characterized in that: If the background of the image to be cut out is a green background, the target color channel is the G channel; if the background of the image to be cut out is a blue background, the target image is the B channel.

10. The method according to claim 8, characterized in that The white balance correction coefficient is determined based on a gray world algorithm. Before the pixel value of the target color channel of the fused image is corrected based on the white balance correction coefficient, the method further includes: Determine the respective proportions of color components of the three color channels R, G, and B in the fused image; If the proportions of the color components of the three color channels are all less than the preset proportions, correction processing is performed on the pixel values ​​of the target color channel of the fused image based on the white balance correction coefficient.

11. An image processing device, characterized in that: The device comprises: An acquisition module is used to acquire the image to be cut out with a pure color background; A ternary image determination module, which determines the ternary image corresponding to the image to be cut out; A filtering module is used to select a target image whose color is closer to the background color of the image to be cut out from the A channel image and the B channel image of the image to be cut out in the LAB color space; using the target image as a guide image, performing guided filtering processing on the ternary image to obtain an Alpha image; The cutout module is used to cut out a foreground area from the image to be cutout based on the Alpha image.

12. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, and the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 10 is implemented.

Images