A method, device and electronic equipment for obtaining a foreground mask

Abstract

The application relates to the technical field of image processing, in particular to a method and device for obtaining a foreground mask and electronic equipment. The method for obtaining the foreground mask comprises the following steps: acquiring a color image and a depth image which comprise a foreground and are aligned; performing foreground segmentation on the color image and the depth image to obtain a corresponding first foreground mask and a second foreground mask; and performing morphological fusion on the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain a third foreground mask. According to the method, the depth image and the color image are combined, and a stable and refined foreground mask is obtained.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for obtaining a foreground mask. Background Technology

[0002] With the rise of self-media, live-streaming e-commerce, video conferencing, and online education, image matting has also become popular. Current matting techniques typically involve first obtaining a detailed foreground mask, and then using it to extract the foreground from a given image or video. Therefore, obtaining a detailed foreground mask is crucial for image matting technology.

[0003] Existing methods for obtaining foreground masks are generally based on foreground segmentation of pure RGB images. However, this method relies on visible light. In scenes where the foreground and background colors are similar, or where there is reflection or glare, the foreground mask obtained by this method is prone to failure or is coarse. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for obtaining a foreground mask, which can solve at least one technical problem in the related art.

[0005] According to an embodiment of this application, a method for obtaining a foreground mask is provided, comprising: obtaining a color image and a depth image that include a foreground and are aligned; performing foreground segmentation on the color image and the depth image to obtain corresponding first foreground mask and second foreground mask respectively; and performing morphological fusion on the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain a third foreground mask.

[0006] According to an embodiment of this application, an apparatus for obtaining a foreground mask is provided, comprising: an acquisition module for acquiring a color image and a depth image that include a foreground and are aligned; a foreground segmentation module for performing foreground segmentation on the color image and the depth image to obtain corresponding first foreground masks and second foreground masks respectively; and a fusion module for performing morphological fusion on the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain a third foreground mask.

[0007] According to an embodiment of this application, an electronic device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for obtaining a foreground mask as described in any of the above embodiments.

[0008] According to an embodiment of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the method for obtaining a foreground mask as described above.

[0009] According to an embodiment of this application, a computer program product is provided that, when the computer program product is run on an electronic device, causes the electronic device to perform the method for obtaining a foreground mask as described above.

[0010] The beneficial effects of this application are as follows: By using depth images to assist color images in distinguishing foreground and background, compared to using only RGB image information, it is not affected by interference from similar colors, is more stable, and can solve the problem of not being able to distinguish the foreground when the foreground and background colors are similar. In addition, due to the existence of depth information, users can easily block scenes at any depth distance, thus protecting privacy. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating the implementation flow of a method for obtaining a foreground mask according to an embodiment of this application;

[0013] Figure 2 This is a schematic diagram illustrating the implementation process of a method for obtaining a foreground mask according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the implementation process of step S130 in a method for obtaining a foreground mask provided in an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the implementation flow of step S130 in a method for obtaining a foreground mask provided in another embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an apparatus for obtaining a foreground mask according to an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of a blending module in an apparatus for obtaining a foreground mask according to an embodiment of this application;

[0018] Figure 7 This is a schematic diagram of the structure of a blending module in an apparatus for obtaining a foreground mask according to another embodiment of this application;

[0019] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0022] The terms "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] Furthermore, in the description of this application, "a plurality of" means two or more. The terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] Figure 1 This is a schematic diagram illustrating the implementation flow of a method for obtaining a foreground mask according to an embodiment of this application. The method includes:

[0025] S110, acquire a color image and a depth image that are aligned and include the foreground.

[0026] The color image can be, for example, an RGB image using the RGB model or a YUV image using the YUV model. This application does not specifically limit the color model applied to the color image.

[0027] In one embodiment, taking a human body as an example, a color image of the same scene including the human body and its corresponding depth image can be directly acquired by a camera. Further, after acquiring the color image and depth image containing the human body, the images need to be aligned, that is, the pixels in each image are aligned one-to-one, so that every pixel in the color image can find a corresponding pixel in the depth image.

[0028] S120, perform foreground segmentation on the color image and depth image to obtain the corresponding first foreground mask and second foreground mask respectively.

[0029] Specifically, a refined first foreground mask is obtained based on a color image, and a coarser second foreground mask is obtained based on a depth image. Due to the image characteristics of color and depth images, the first foreground mask has higher refinement than the second foreground mask, while the second foreground mask has higher stability than the first foreground mask.

[0030] It should be noted that in this embodiment, the first foreground mask of the color image can be obtained first, and then the second foreground mask corresponding to the depth image can be obtained according to the alignment relationship between the color image and the depth image; in another embodiment, the foreground mask images corresponding to the color image and the depth image can also be obtained simultaneously using different algorithms, such as... Figure 2 As shown, the foreground of the color image and the depth image are extracted using a preset image matting algorithm model and a preset depth segmentation algorithm model, respectively, to obtain the corresponding refined first foreground mask and the second foreground mask with a stable foreground. No restrictions are imposed here.

[0031] In one embodiment, the preset matting algorithm model and the preset depth segmentation algorithm model can be foreground segmentation methods such as edge detection and background segmentation, or deep learning-based neural network models such as ResNet or VGG. Preferably, the preset matting algorithm model and the preset depth segmentation algorithm model are lightweight neural network models, such as MobileNet or SqueezeNet, to reduce the computational complexity and storage size of the model and lower the deployment requirements of the matting algorithm model.

[0032] It should be noted that the neural network structures that can be used for the image matting algorithm model and the depth segmentation algorithm model in this application embodiment can be the same or different. When using the same neural network model, the parameters of each layer of the network need to be adjusted because images with different characteristics are used. This is not limited here.

[0033] S130, based on the alignment relationship between the color image and the depth image, morphological fusion is performed using the first foreground mask and the second foreground mask to obtain the third foreground mask.

[0034] More specifically, a refined first foreground mask obtained from a color image and a stable second foreground mask obtained from a depth image are subjected to morphological operations. While ensuring the extraction of fine edges from the first foreground mask, the second foreground mask is merged into the first foreground mask according to the alignment relationship between the color image and the depth image, resulting in a third foreground mask that is both refined and has a stable foreground. This process requires low computational power and can quickly obtain the third foreground mask. Furthermore, after quickly obtaining the third foreground mask, the foreground in the color image is extracted using the third foreground mask and composited with an arbitrary background image, thereby achieving a rapid completion of the image extraction and compositing process.

[0035] In one embodiment, such as Figure 3 As shown, S130 more specifically includes steps S311, S312, and S313:

[0036] S311, Erosion and dilation are performed on the second foreground mask, and the eroded and dilated second foreground masks are merged to generate a three-part map including the foreground, background and unknown areas.

[0037] In one embodiment, a second foreground mask is subjected to one or more dilation operations to obtain a dilated second foreground mask. The eroded second foreground mask and the dilated second foreground mask are then merged to generate a three-part image including the foreground, background, and unknown region. Dilation and erosion of the second foreground mask image improves the accuracy of the three-part image, thereby enhancing the accuracy of the foreground mask. It should be noted that this application does not limit the number of dilation and erosion operations, but three operations are preferred, as this ensures both data processing efficiency and result accuracy.

[0038] In one embodiment, the erosion operation on the second foreground mask more specifically includes: normalizing the pixels of the second foreground mask to [0,1], selecting a 3×3 structure matrix, wherein all elements in the matrix are 1, scanning each pixel in the second foreground mask with a step size of 1, performing a logical AND operation between the structure matrix and the second foreground mask, and if the values ​​of the structure matrix and the second foreground mask are both 1, then the pixel value of that point in the output image is 1, otherwise the pixel value of the output image is 0.

[0039] In another embodiment, the dilation operation on the second foreground mask more specifically includes: normalizing the pixels of the second foreground mask to [0,1], selecting a 3×3 structure matrix, wherein all elements in the matrix are 1, performing a logical AND operation between the structure matrix and the second foreground mask, and if the values ​​of the structure matrix and the second foreground mask are both 0, then the pixel value of that point in the output image is 0, otherwise the pixel value of the output image is 1.

[0040] Furthermore, when the pixel values ​​of both the eroded and dilated second foreground mask images are 1, this pixel constitutes the foreground region in the tripartite image; when the pixel values ​​of both the eroded and dilated second foreground mask images are 0, this pixel constitutes the background region in the tripartite image; the rest are unknown regions in the tripartite image.

[0041] In one embodiment, fusing the eroded second foreground mask and the dilated second foreground mask to obtain a three-part image more specifically includes: standardizing the unknown region and foreground region constituting the three-part image, and merging the standardized foreground region, unknown region, and background region to obtain the three-part image. More specifically, assuming the pixel matrix corresponding to the foreground region is A, the pixel matrix corresponding to the unknown region is B, and the pixel value of the background region is 0 (which can be omitted from calculation), preferably, the foreground region, unknown region, and background region are standardized to the range [0, 255] to obtain the three-part image, which can be generated by A*128+B*255. Thus, in the three-part image, the pixel value of the foreground region is 255, the pixel value of the unknown region is 128, and the pixel value of the background region is 0.

[0042] It should be noted that this application does not limit the standardization range of pixel values ​​for each region, and is not limited to [0,255]. The design can be made according to the actual situation, and no restrictions are imposed here.

[0043] S312, based on the alignment relationship between the color image and the depth image, the intersection region of the unknown region of the tripartite image and the first foreground mask is obtained to obtain the edge image.

[0044] S313, determine the union region of the edge image and the eroded second foreground mask to obtain the third foreground mask.

[0045] In another embodiment, such as Figure 4 As shown, step S130 more specifically includes steps S411 and S412:

[0046] S411, perform edge detection on the first foreground mask to extract the edge region, and perform an erosion operation on the second foreground mask to obtain the eroded second foreground mask.

[0047] In one embodiment, the second foreground mask is etched once or multiple times to obtain the etched second foreground mask, preferably three times, which not only ensures data processing efficiency but also ensures the accuracy of the results.

[0048] S412, based on the alignment relationship between the depth image and the color image, calculate the union region of the edge region and the eroded second foreground mask to obtain the third foreground mask.

[0049] It should be noted that, compared with the previous embodiment, the method used in this embodiment has lower requirements for processor computing power and is faster.

[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0051] An embodiment of this application also provides an apparatus for acquiring a foreground mask. For details not described in the foreground mask acquisition apparatus, please refer to the descriptions in the foregoing embodiments of the method for acquiring a foreground mask.

[0052] Figure 5 This is a schematic block diagram of an apparatus for acquiring a foreground mask according to an embodiment of this application. The apparatus for acquiring the foreground mask includes: an acquisition module 51, a foreground segmentation module 52, and a blending module 53.

[0053] The acquisition module 51 is used to acquire a color image and a depth image that are aligned and include a foreground.

[0054] The foreground segmentation module 52 is used to perform foreground segmentation on the color image and the depth image to obtain the corresponding first foreground mask and second foreground mask, respectively.

[0055] The fusion module 53 is used to perform morphological fusion of the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain the third foreground mask.

[0056] In one embodiment, such as Figure 6 As shown, the fusion module 53 includes a first submodule 531, a second submodule 532 and a third submodule 533.

[0057] The first submodule 531 is used to erode and dilate the second foreground mask, and generate a three-part map including the foreground, background and unknown areas based on the eroded and dilated second foreground mask.

[0058] The second submodule 532 is used to obtain the edge image by finding the intersection area of ​​the unknown region of the triangulation image and the first foreground mask based on the alignment relationship between the color image and the depth image.

[0059] The third submodule 533 is used to determine the union region of the edge image and the eroded second foreground mask to obtain the third foreground mask.

[0060] In another embodiment, such as Figure 7 As shown, the fusion module 53 includes a fourth submodule 534, a fifth submodule 535, and a sixth submodule 536.

[0061] The fourth submodule 534 is used to perform edge detection on the first foreground mask to extract edge regions.

[0062] The fifth submodule 535 is used to perform an erosion operation on the second foreground mask to obtain an eroded second foreground mask.

[0063] The sixth submodule 536 is used to obtain the third foreground mask by calculating the union region of the edge region and the eroded second foreground mask based on the alignment relationship between the depth image and the color image.

[0064] Figure 8 An electronic device according to an embodiment of this application may include one or more processors 100. Figure 8 (Only one is shown in the diagram), memory 101, and a computer program 102 stored in memory 101 and executable on one or more processors 100, such as a program for acquiring a foreground mask. When one or more processors 100 execute computer program 102, they can implement the various steps in the method embodiment for acquiring a foreground mask. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in the apparatus embodiment for acquiring a foreground mask, which is not limited here.

[0065] In some embodiments, the electronic device further includes an acquisition device, which includes a color camera and a depth camera, for acquiring color images and depth images respectively and transmitting them to the processor 100 to obtain a foreground mask. It should be noted that the color camera and the depth camera can be set independently or as an integrated unit. The depth camera can be a structured light camera, a TOF camera, a binocular camera, or a multi-view camera, etc., and there is no limitation here.

[0066] Those skilled in the art will understand that Figure 8 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. Electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, etc.

[0067] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0068] In one embodiment, memory 101 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 101 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 101 may include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the electronic device. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0070] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the various steps in the method embodiment for obtaining a foreground mask.

[0071] One embodiment of this application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps in the method embodiment for acquiring a foreground mask.

[0072] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can also be implemented by a computer program instructing related hardware, and a computer program can be stored in a computer-readable storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0078] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of obtaining a foreground mask, characterized by, The method comprises the following steps: obtaining a color image and a depth image including a foreground and being aligned; performing foreground segmentation on the color image and the depth image to obtain a first foreground mask and a second foreground mask respectively; performing morphological fusion on the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain a third foreground mask, comprising: performing erosion and dilation on the second foreground mask, and fusing the eroded second foreground mask and the dilated second foreground mask to generate a trimap including a foreground, a background and an unknown region; obtaining an intersection region of the unknown region of the trimap and the first foreground mask according to the alignment relationship between the color image and the depth image to obtain an edge image; determining a union region of the edge image and the eroded second foreground mask to obtain a third foreground mask; or, performing edge detection on the first foreground mask to extract an edge region, and performing erosion on the second foreground mask to obtain an eroded second foreground mask; obtaining a union region of the edge region and the eroded second foreground mask according to the alignment relationship between the depth image and the color image to obtain a third foreground mask.

2. The method of claim 1, wherein, The erosion on the second foreground mask comprises: normalizing pixels of the second foreground mask, selecting a 3*3 structure matrix, elements in the structure matrix are all 1, performing logical AND operation on the structure matrix and the normalized second foreground mask, if values of the structure matrix and the normalized second foreground mask are both 1, then outputting a point pixel value of a to-be-calculated pixel point of an image as 1, otherwise outputting a pixel value of the to-be-calculated pixel point of the image as 0.

3. The method of claim 1, wherein, The dilation on the second foreground mask comprises: normalizing pixels of the second foreground mask, selecting a 3*3 structure matrix, elements in the structure matrix are all 1, performing logical AND operation on the structure matrix and the normalized second foreground mask, if values of the structure matrix and the normalized second foreground mask are both 0, then outputting a point pixel value of a to-be-calculated pixel point of an image as 0, otherwise outputting a pixel value of the to-be-calculated pixel point of the image as 1.

4. The method of claim 1, wherein, The fusion of the eroded second foreground mask and the dilated second foreground mask to generate a trimap including a foreground, a background and an unknown region comprises: performing standardization processing on the unknown region and the foreground region constituting the trimap, and merging the standardized foreground region, unknown region and background region to obtain the trimap.

5. An apparatus for obtaining a foreground mask, the apparatus comprising: The method comprises the following steps: an acquisition module, configured to acquire a color image and a depth image including a foreground and being aligned; a foreground segmentation module, configured to perform foreground segmentation on the color image and the depth image to obtain a first foreground mask and a second foreground mask respectively; a fusion module, configured to perform morphological fusion on the first foreground mask and the second foreground mask according to the alignment relationship between the color image and the depth image to obtain a third foreground mask, comprising: eroding and dilating the second foreground mask, fusing the eroded second foreground mask and the dilated second foreground mask to generate a trimap including foreground, background and unknown regions; obtaining an intersection region of the unknown region of the trimap and the first foreground mask according to the alignment relationship between the color image and the depth image, to obtain an edge image; determining a union region of the edge image and the eroded second foreground mask to obtain a third foreground mask; or, performing edge detection on the first foreground mask to extract an edge region, and eroding the second foreground mask to obtain an eroded second foreground mask; obtaining a third foreground mask according to a union region of the edge region and the eroded second foreground mask according to the alignment relationship between the depth image and the color image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method for obtaining a foreground mask according to any one of claims 1 to 4.

7. The electronic device of claim 6, wherein, The acquisition device includes a color camera and a depth camera, and is configured to acquire a color image and a depth image respectively and transmit the color image and the depth image to the processor.

8. A computer storage medium storing a computer program, the computer program comprising instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 7. The computer program is executed by the processor to implement the method for obtaining a foreground mask according to any one of claims 1 to 4. The acquisition device includes a color camera and a depth camera, and is configured to acquire a color image and a depth image respectively and transmit the color image and the depth image to the processor.

Images