Region matting method, electronic device, and storage medium

Abstract

The application relates to the field of image processing, and provides a region matting method, an electronic device and a storage medium. The method comprises the following steps: determining a target region of a to-be-matted image, generating a mask image of the target region; performing matting on the to-be-matted image to obtain a first alpha channel corresponding to a first foreground image; updating the first alpha channel image by using the mask image to obtain a second alpha channel; and segmenting the to-be-matted image by using the second alpha channel to obtain a second foreground image. The method can protect the target region during matting, so that a complete foreground image is obtained, and the matting effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a region matting method, electronic device, and storage medium. Background Technology

[0002] Background removal is a common operation in image processing, used to extract the desired portion of an image from the entire frame, preparing it for later compositing. However, when the image to be removed is taken against a solid-color background of a specific color, the areas of the desired color are often treated as background and removed, resulting in a less than ideal removal effect. For example, if the image to be removed is a green screen image from a live-streaming event selling agricultural products, removing the green agricultural products would often treat them as background and remove them. Summary of the Invention

[0003] In view of the above, it is necessary to propose a region matting method, electronic device and storage medium that can protect the specified region during matting, thereby obtaining a complete foreground image and improving the matting effect.

[0004] Embodiments of this application provide a region matting method, the method comprising: determining a target region of an image to be matted, generating a mask image of the target region; matting the image to be matted to obtain a first alpha channel corresponding to a first foreground image; updating the first alpha channel image using the mask image to obtain a second alpha channel; and segmenting the image to be matted using the second alpha channel to obtain a second foreground image.

[0005] In one embodiment, determining the target region of the image to be cut out includes: determining an initial region corresponding to the target object in the image to be cut out; clustering the pixels in the initial region and selecting the category with the most pixels in the clustering result as the target category; when the main color of the pixels in the target category is the target color, calculating the ratio of a first number of pixels in the target category to a second number of pixels in the initial region; when the ratio is greater than a first preset threshold, using the initial region as the target region.

[0006] In one embodiment, when the image to be cut out comprises multiple images that are temporally related, a target tracking algorithm is used to track and locate the target region in each image to be cut out.

[0007] In one embodiment, determining the initial region corresponding to the target object in the image to be cut out includes: performing feature detection on the target object in the image to be cut out, generating multiple candidate boxes for the target object, and using the region formed by the multiple candidate boxes as the initial region.

[0008] In one embodiment, clustering the pixels in the initial region includes: clustering the pixels of the target image based on the color of the pixels in the target image.

[0009] In one embodiment, the method further includes: calculating the average color of all pixels in the target category, and calculating the cosine distance between the average color and the target color; when the cosine distance is less than a second preset threshold, determining the main color of the pixels in the target category as the target color.

[0010] In one embodiment, the mask image has the same size as the image to be cut out, the grayscale value of the target region in the mask image is 1, and the grayscale value of the area outside the target region in the mask image is 0.

[0011] In one embodiment, updating the first alpha channel image using the mask image includes: performing an OR operation between the mask image and the first alpha channel to update the grayscale value of the target region in the first alpha channel to 1.

[0012] This application provides a region matting device, comprising: a determining module, a matting module, an updating module, and a segmentation module; the determining module is used to determine a target region of an image to be matted and generate a mask image of the target region; the matting module is used to matte the image to be matted to obtain a first alpha channel corresponding to a first foreground image; the updating module is used to update the first alpha channel image using the mask image to obtain a second alpha channel; and the segmentation module is used to segment the image to be matted using the second alpha channel to obtain a second foreground image.

[0013] An embodiment of this application provides an electronic device, which includes a processor and a memory, wherein the processor is used to implement the region matting method when executing a computer program stored in the memory.

[0014] Embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the region matting method.

[0015] In summary, the region matting method, electronic device, and storage medium described in this application can determine the target region that needs to be retained in the image to be matted, protect the target region during matting, thereby obtaining a complete foreground image and improving the matting effect. Attached Figure Description

[0016] Figure 1 This is a structural diagram of the electronic device provided in the embodiments of this application.

[0017] Figure 2 This is a flowchart of the region matting method provided in the embodiments of this application.

[0018] Figure 3 This is a flowchart of the process for determining the target area provided in an embodiment of this application.

[0019] Figure 4 This is a structural diagram of the region matting device provided in the embodiments of this application. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing an embodiment in one instance only and is not intended to be limiting of the application.

[0022] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0024] In one embodiment, image matting is a common operation in image processing, referring to the extraction of the desired portion of an image from the entire frame to prepare for later compositing. When the image to be matted is taken against a solid-color background of a specific color, the specific color area that needs to be retained is often treated as background and removed, resulting in a matting effect that does not meet expectations. For example, if the image to be matted is a green screen image from a live-streaming event selling agricultural products, the green agricultural products would be treated as background and removed when matting the green screen image.

[0025] To address the aforementioned issues, this application provides a region matting method that can determine the target region to be retained in the image to be matted, protect the target region during matting, thereby obtaining a complete foreground image and improving the matting effect.

[0026] For example Figure 1 The diagram shown is a structural diagram of an electronic device provided in an embodiment of this application. The region matting method provided in this embodiment is executed by an electronic device. The electronic device integrates the region matting function provided by the method in this embodiment, or runs in the electronic device in the form of a software development kit (SDK). The electronic device can be a computer, server, laptop, mobile terminal, or other devices.

[0027] In this embodiment of the application, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0028] Those skilled in the art should understand that Figure 1 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0029] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include other external devices, such as input / output devices like keyboards, mice, remote controls, displays, touchpads, or voice-activated devices.

[0030] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0031] Figure 2This is a flowchart of a region matting method provided in an embodiment of this application. The region matting method is applied in electronic devices, for example... Figure 1 The electronic device 1 in the flowchart specifically includes the following steps. Depending on different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0032] S11, determine the target region of the image to be cut out, and generate a mask image of the target region.

[0033] In one embodiment, the image to be cut out may include one or more images. When the image to be cut out includes multiple images, the images to be cut out may be multiple video frames belonging to the same video and having a temporal relationship, wherein the temporal relationship may be the order in which each image to be cut out was captured.

[0034] The image to be cut out can also be an image taken against a solid color background of a specific color. For example, the image to be cut out can be an image taken in a green screen environment, or it can be a green screen image used for live-streaming sales of agricultural products.

[0035] In one embodiment, the electronic device can receive an image to be cut out input by a user. When the user inputs a video, the electronic device can simultaneously receive the video and extract video frames from it for subsequent image cutout based on the video frames. Furthermore, the electronic device can also be electrically or communicatively connected to a camera device that captures the image to be cut out, thereby acquiring the image to be cut out.

[0036] In another embodiment, the electronic device may also include a camera (e.g., a webcam) to directly capture the image to be cut out.

[0037] In one embodiment, the image to be cut out is processed by extracting the foreground image from the image to be cut out. For example, various objects photographed in different environments, or the target object (e.g., human body, agricultural product, etc.) photographed in a green screen environment, are used as the background in the green screen area, and the image of the area where the target object is located outside the green screen area is the foreground image of the image to be cut out.

[0038] While obtaining the foreground image, the corresponding alpha channel (transparency channel image) can also be obtained. Based on the foreground image and its corresponding alpha channel, the foreground image can be composited onto any new background to obtain a composite image. In the alpha channel of the foreground image, the area containing the target object is opaque, while the remaining areas are transparent. Grayscale values ​​can be used to describe the transparency of the areas in the alpha channel: a grayscale value of 1 indicates opacity, a grayscale value greater than 0 and less than 1 indicates semi-transparency (the closer the grayscale value is to 1, the lower the transparency), and a grayscale value of 0 indicates complete transparency.

[0039] In one embodiment of this application, the image matting principle can be explained by the following formula:

[0040] img foreground =alpha*img,

[0041] Among them, img foreground The image represents the foreground image, alpha is the alpha channel corresponding to the foreground image, and img represents the image to be cut out.

[0042] The principle of image synthesis can be explained by the following formula:

[0043] img′=alpha*img foreground +(1-alpha)*img background ,

[0044] Where img′ is the synthesized image, img background This indicates a new background.

[0045] In one embodiment, since the target object may include a protected object that is the same color as the background and needs to be retained during the cutout process, such as green agricultural products in a green screen environment, it is necessary to determine the target area where the protected object is located and protect the target area in order to prevent the area where the protected object is located from being removed during the cutout process.

[0046] The specific process for determining the target region of the image to be cut out can be found in the following text. Figure 3 Detailed explanation of the process shown.

[0047] In one embodiment, once the target area to be protected is determined, a mask image of the target area can be generated to protect the target area. Specifically, the mask image has the same size as the image to be cut out, the grayscale value of the target area in the mask image is 1, and the grayscale value of the area outside the target area in the mask image is 0. The method for generating the mask image is a common method in the art and will not be described in detail here.

[0048] S12, perform image cutout on the image to be cut out to obtain the first alpha channel corresponding to the first foreground image.

[0049] In one embodiment, referring to the image matting principle in step S11, a preset image matting tool (such as the Keylight plugin in After Effects software) can be used to matte the image to be matted, or a pre-trained image matting model can be used. For example, the image matting model can adopt an encoder plus decoder structure, wherein the encoder is used to extract foreground features from the image to be matted, and the decoder is used to decode the foreground features extracted by the encoder into a foreground image.

[0050] In one embodiment, since the target area containing the protected object may be removed as background during the image matting process, the target area in the first foreground image may be a blank area, and the target area in the first alpha channel may be a transparent area (grayscale of 0). If the first foreground image and the first alpha channel are directly used to synthesize a new background, the synthesized image will lack the image of the protected object in the target area, resulting in an unsatisfactory synthesis effect. Therefore, the first foreground image cannot be directly used as the foreground image during synthesis, and the first alpha channel must be updated.

[0051] S13, update the first alpha channel image using the mask image to obtain the second alpha channel.

[0052] In one embodiment, updating the first alpha channel image using the mask image includes: performing an OR operation between the mask image and the first alpha channel to update the grayscale value of the target region in the first alpha channel to 1.

[0053] In one embodiment, since the mask image and the image to be cut out are the same size, the mask image and the first alpha channel are also the same size. After performing an OR operation between the mask image and the first alpha channel, the grayscale value of the target region in the first alpha channel can be updated to be the same as the grayscale value of the target region in the mask image, that is, the grayscale value of the target region in the first alpha channel is updated to 1, while the grayscale values ​​outside the target region in the first alpha channel remain unchanged, thereby making the target region in the second alpha channel an opaque region.

[0054] S14, the image to be cut out is segmented using the second alpha channel to obtain a second foreground image.

[0055] In one embodiment, the second alpha channel is the same size as the image to be cut out. The image to be cut out is segmented using the boundaries of the opaque regions in the second alpha channel, thus achieving the segmentation of the foreground image in the image to be cut out. Furthermore, since the target region in the second alpha channel is an opaque region, the region in the image to be cut out corresponding to the opaque region in the second alpha channel is the final second foreground image that meets the user's requirements.

[0056] In one embodiment, a user can use the second alpha channel, a second foreground image, and a new background to obtain a composite image of a new background. This composite image will include an image of the target area corresponding to the protected object, thereby meeting the user's needs and improving the synthesis effect.

[0057] The region matting method provided in this application can determine the target region that needs to be retained in the image to be matted, and protect the target region during matting to obtain a complete foreground image and improve the matting effect.

[0058] In one embodiment, such as Figure 3 The flowchart shown is for determining the target area according to an embodiment of this application, and specifically includes the following steps:

[0059] S111, determine the initial region corresponding to the target object in the image to be cut out.

[0060] In one embodiment, determining the initial region corresponding to the target object in the image to be cut out includes: performing feature detection on the target object in the image to be cut out, generating multiple candidate boxes for the target object, and using the region formed by the multiple candidate boxes as the initial region.

[0061] In one embodiment, the algorithm for feature detection of a target object (e.g., a human body, green agricultural products) includes commonly used object detection algorithms, such as the sliding window algorithm and the YOLO (You Only Look Once) algorithm. During target object detection, multiple candidate boxes (or predicted boxes) are generated for the region where the target object is located. These candidate boxes have overlapping and non-overlapping areas, and the union of all candidate boxes is used as the initial region.

[0062] S112, cluster the pixels in the initial region, and select the category with the most pixels in the clustering result as the target category.

[0063] In one embodiment, clustering the pixels in the initial region includes: clustering the pixels of the target image based on the color of the pixels in the target image.

[0064] In one embodiment, since determining the target region is to identify the area containing the protected object that is the same color as the background, pixel clustering is performed based on the color of the pixels in the initial region. Specifically, the algorithm for clustering pixels includes, but is not limited to, the K-means clustering algorithm.

[0065] S113, when the main color of the pixels of the target category is the target color, calculate the ratio of the first number of pixels of the target category to the second number of pixels contained in the initial region.

[0066] In one embodiment, the method further includes: calculating the average color of all pixels in the target category, and calculating the cosine distance between the average color and the target color (e.g., green), and determining the main color of the pixels in the target category as the target color when the cosine distance is less than a second preset threshold (e.g., 0.95).

[0067] In one embodiment, when the main color of the pixels of the target category is the target color, it is necessary to determine the proportion of pixels of the target color included in the initial region within the entire initial region. When the main color of the pixels of the target category is not the target color, the initial region is directly used as the background region without protection.

[0068] S114, when the ratio is greater than the first preset threshold, the initial region is taken as the target region.

[0069] In one embodiment, when the proportion of pixels of the target color included in the initial region within the entire initial region is greater than a first preset threshold (e.g., 0.5), the initial region can be protected as a target region. When the proportion of pixels of the target color included in the initial region within the entire initial region is less than or equal to the first preset threshold (e.g., 0.5), the initial region can be treated as a background region and no protection is required.

[0070] In one embodiment, in addition to the method of automatically identifying the target area described above, the area selected by the user can also be used as the target area in response to the user's operation. For example, the user can use a mouse or other operating device to directly select the object to be protected in the image to be cut out using various shapes (such as rectangles, circles, etc.). The electronic device responds to the user's selection operation and uses the area selected by the user as the target area.

[0071] In one embodiment, in addition to the method for determining the target region in a single image to be cut out as described above, the method further includes: when the image to be cut out comprises multiple images that are temporally related, using a target tracking algorithm to track and locate the target region in each image to be cut out. The target tracking algorithm includes, but is not limited to, the YOLO algorithm.

[0072] In one embodiment, using a target tracking algorithm can improve the efficiency of image matting in video streams by eliminating the need to identify the target region in each of multiple temporally related images.

[0073] In one embodiment, the region matting method provided in this application can automatically identify the target region to be protected, and can also interact with the user, using a user-defined region as the target region, thereby achieving accurate protection of the target region. Furthermore, in addition to identifying the target region of a static protected object in a single image to be matted, it can also track dynamic protected objects in a video stream image, thereby improving the efficiency of target region identification and matting.

[0074] Figure 4 This is a structural diagram of the region matting device provided in the embodiments of this application.

[0075] In some embodiments, the region matting device 20 may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the region matting device 20 may be stored in the memory of an electronic device and executed by at least one processor to perform (see details). Figure 2 (Description) The function of region cutout.

[0076] In this embodiment, the region matting device 20 can be divided into multiple functional modules according to its functions. These functional modules may include: a determination module 201, a matting module 202, an update module 203, and a segmentation module 204. The term "module" in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory. In this embodiment, the functional implementation of each module in the region matting device 20 can be found in the above description of the region matting method, and will not be repeated here.

[0077] The determining module 201 is used to determine the target region of the image to be cut out and generate a mask image of the target region.

[0078] The image matting module 202 is used to perform image matting on the image to be matted to obtain the first alpha channel corresponding to the first foreground image.

[0079] The update module 203 is used to update the first alpha channel image using the mask image to obtain the second alpha channel.

[0080] The segmentation module 204 is used to segment the image to be cut out using the second alpha channel to obtain a second foreground image.

[0081] Continuing from the previous text Figure 1 As described above, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the region matting method as described. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0082] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application required for a function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0083] In one embodiment of this application, a computer program is stored on the computer-readable storage medium, which, when executed by the processor 32, implements the following: Figure 2 The process shown. Alternatively, when the computer program is executed by the processor, it implements as follows: Figure 4 The functions of each module / unit in the area matting device shown are as follows: Figure 4 Modules 201-204 in the document.

[0084] In some embodiments, the at least one processor 32 is the control unit of the electronic device 3, connecting various components of the electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data. For example, when the at least one processor 32 executes a computer program stored in the memory, it implements all or part of the steps of the region matting method described in this application embodiment; or it implements all or part of the functions of the region matting device. The at least one processor 32 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0085] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0086] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 32 through a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, camera devices, etc., which will not be described in detail here.

[0087] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of this application.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0089] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0091] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method of region matting, characterized in that, The method includes: Determine the target region of the image to be cut out, and generate a mask image of the target region; The image to be cut out is cut out to obtain the first alpha channel corresponding to the first foreground image; The first alpha channel is updated using the mask image to obtain the second alpha channel; The image to be cut out is segmented using the second alpha channel to obtain a second foreground image.

2. The method of claim 1, wherein, The process of determining the target region of the image to be cut out includes: Determine the initial region corresponding to the target object in the image to be cut out; Cluster the pixels in the initial region and select the category with the most pixels in the clustering result as the target category; When the main color of the pixels of the target category is the target color, calculate the ratio of the first number of pixels of the target category to the second number of pixels contained in the initial region; When the ratio is greater than a first preset threshold, the initial region is taken as the target region.

3. The method of claim 1, wherein, The method further includes: When the image to be cut out includes multiple images that are time-related, a target tracking algorithm is used to track and locate the target region in each image to be cut out.

4. The method of claim 2, wherein, Determining the initial region corresponding to the target object in the image to be cut out includes: Feature detection is performed on the target object in the image to be cut out, multiple candidate boxes of the target object are generated, and the region formed by the multiple candidate boxes is used as the initial region.

5. The region matting method according to claim 2, characterized in that, The clustering of pixels in the initial region includes: clustering the pixels in the initial region based on their colors.

6. The region matting method according to claim 2, characterized in that, The method further includes: Calculate the average color of all pixels in the target category, and calculate the cosine distance between the average color and the target color. When the cosine distance is less than a second preset threshold, determine the main color of the pixels in the target category as the target color.

7. The region matting method according to claim 1, characterized in that, The mask image has the same size as the image to be cut out. The grayscale value of the target area in the mask image is 1, and the grayscale value of the area outside the target area in the mask image is 0.

8. The region matting method according to claim 1, characterized in that, Updating the first alpha channel using the mask image includes: The mask image is ORed with the first alpha channel to update the grayscale value of the target region in the first alpha channel to 1.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to implement the region matting method as described in any one of claims 1 to 8 when executing a computer program stored in the memory.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the region matting method as described in any one of claims 1 to 8.

Images