A foreground cutout method, device and electronic device

Abstract

The present application relates to the field of computer vision technology, and in particular to a foreground cutout method, device and electronic device. The foreground cutout method includes: obtaining a current frame color image and a depth image corresponding to the current frame, a historical frame color image and its corresponding transparency map; performing multiple corrosion and dilation processes on the depth image to generate a current frame trisection map; and splicing the historical frame color image and its corresponding transparency map to generate a historical frame fusion map; obtaining a coded feature map based on the current frame color image, the historical frame fusion map and the current frame trisection map; using the coded feature map to obtain a transparency map corresponding to the current frame, and cutting out the foreground based on the transparency map. The embodiments of the present application improve the accuracy of the foreground segmentation results and the stability of the edges.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a foreground cutout method, device and electronic device. Background Art

[0002] In the field of computer vision technology, image cutout is a commonly used technique. With the rapid development of deep learning, the development of image cutout algorithms has also been greatly promoted.

[0003] Among the deep learning-based cutout algorithms, the trimap-based methods have been widely studied. Although the trimap-based methods have high accuracy, in some solutions, the trimap is manually generated by the user, and the given image needs to be manually annotated to add additional constraints to the cutout problem, which is very user-unfriendly and has poor practicality. In other solutions, the trimap is generated by a depth image based on the time-of-flight principle or the structured light principle. The stability of the depth image in the edge of the object, hair and other areas is poor, which will lead to poor stability of the final cutout.

[0004] In recent years, portrait-based semantic segmentation methods have received a lot of attention. Since foreground cutout tasks usually require the cutout of people and the objects they hold, portrait-based semantic segmentation methods have good results for portrait segmentation but poor results for object segmentation. The effect deteriorates when dealing with scenes of people holding objects or non-humans, so this method is not suitable for foreground cutout.

[0005] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present application. It does not necessarily belong to the prior art of the present application. In the absence of clear evidence that the above content has been disclosed before the filing date of the present application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a foreground cutout method, device and electronic device, which can solve at least one technical problem in the related art.

[0007] In a first aspect, an embodiment of the present application provides a foreground cutout method, comprising:

[0008] Get the current frame color image and the depth image corresponding to the current frame, the historical frame color image and its corresponding transparency map;

[0009] Performing multiple corrosion and expansion processes on the depth image corresponding to the current frame to generate a current frame trisection image; and splicing the historical frame color image and its corresponding transparency image to generate a historical frame fusion image;

[0010] Acquire a coding feature map according to the current frame color image, the historical frame fusion map and the current frame trisection map;

[0011] The encoding feature map is used to obtain a transparency map corresponding to the current frame, and the foreground is extracted according to the transparency map.

[0012] In this embodiment, on the one hand, the current frame information and historical frame information are used as information sources for foreground segmentation prediction, thereby improving the accuracy of the segmentation result and the stability of the edge; on the other hand, in addition to considering the historical frame color image, the historical frame information also takes the historical frame transparency map as additional information, thereby further improving the stability of the foreground segmentation edge.

[0013] As an implementation of the first aspect, acquiring the encoding feature map according to the current frame color image, the historical frame fusion map and the current frame tri-map includes:

[0014] Performing feature extraction on the current frame color image, the historical frame fusion image, and the current frame tri-image, to obtain a first feature image of the current frame color image, a second feature image of the historical frame fusion image, and a third feature image of the current frame tri-image, respectively;

[0015] The first feature map and the second feature map are fused to obtain a first feature fusion map; the first feature map and the third feature map are fused to obtain a second feature fusion map;

[0016] The first feature map, the first feature fusion map and the second feature fusion map are concatenated to obtain a coding feature map.

[0017] As an implementation of the first aspect, obtaining a transparency map corresponding to the current frame according to the encoding feature map includes:

[0018] The encoded feature map is decoded and a transparency map corresponding to the current frame is output.

[0019] As an implementation of the first aspect, the step of splicing the first feature map, the first feature fusion map, and the second feature fusion map to obtain a coding feature map includes:

[0020] The first feature map, the first feature fusion map and the second feature fusion map are concatenated using a feature combination module including a convolution layer, a batch normalization layer and an activation function layer to obtain a coding feature map.

[0021] As an implementation of the first aspect, the extracting features of the current frame color image, the historical frame fusion image, and the current frame tri-image includes:

[0022] A preset feature extraction network is used to perform feature extraction on the current frame color image, the historical frame fusion image and the current frame tri-image. The preset feature extraction network includes three convolution modules, and each convolution module includes a convolution layer, a batch normalization layer and an activation function layer.

[0023] As an implementation of the first aspect, performing multiple erosion and dilation processes on the depth image corresponding to the current frame to generate a current frame trisection map includes:

[0024] Segmenting the depth image corresponding to the current frame to obtain a foreground soft segmentation map of the current frame;

[0025] The foreground soft segmentation map of the current frame is subjected to multiple erosion processes to obtain a foreground area of ​​the current frame trisection map; the foreground soft segmentation map of the current frame is subjected to multiple dilation processes and then the foreground area is subtracted to obtain an unknown area of ​​the current frame trisection map; the part of the current frame soft segmentation map except the foreground area and the unknown area is the background area of ​​the current frame trisection map.

[0026] As an implementation of the first aspect, segmenting the depth image corresponding to the current frame to obtain a foreground soft segmentation map of the current frame includes:

[0027] Obtain a depth threshold; traverse the pixels in the depth image corresponding to the current frame, set pixel values ​​greater than zero and less than the depth threshold to 1, and set pixel values ​​greater than or equal to the depth threshold to 0, to obtain a foreground soft segmentation map of the current frame; or

[0028] Obtain a valid depth range; traverse the pixels in the depth image corresponding to the current frame, set the pixel values ​​belonging to the valid depth range to 1, and set the pixel values ​​not belonging to the valid depth range to 0, to obtain a foreground soft segmentation map of the current frame.

[0029] As an implementation of the first aspect, when the transparency map corresponding to the historical frame does not exist, the transparency map corresponding to the historical frame is replaced by the current frame trisection map.

[0030] In a second aspect, an embodiment of the present application provides a foreground cutout device, comprising:

[0031] The first acquisition module is used to acquire the current frame color image and the depth image corresponding to the current frame, the historical frame color image and the corresponding transparency map;

[0032] A generation module is used to perform multiple corrosion and expansion processes on the depth image corresponding to the current frame to generate a current frame trisection image; and to splice the historical frame color image and its corresponding transparency image to generate a historical frame fusion image;

[0033] A second acquisition module is used to acquire a coding feature map according to the current frame color image, the historical frame fusion image and the current frame tri-image;

[0034] The third acquisition module is used to use the encoded feature map to obtain a transparency map corresponding to the current frame, and to extract the foreground according to the transparency map.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising an RGBD camera, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the RGBD camera is used to capture a color image of a current frame and a depth image corresponding to the current frame, and when the processor executes the computer program, the foreground cutout method as described in the first aspect or any implementation method of the first aspect is implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the foreground cutout method as described in the first aspect or any implementation of the first aspect is implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the foreground cutout method as described in the first aspect or any implementation of the first aspect.

[0038] It should be understood that the beneficial effects of the second to fifth aspects can be found in the relevant description of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 This is a schematic diagram of an implementation flow of a foreground cutout method provided in an embodiment of the present application;

[0041] Figure 2 This is a schematic diagram of the implementation flow of steps S130 and S140 of a foreground cutout method provided in one embodiment of the present application;

[0042] Figure 3 It is a structural schematic diagram of a cutout model provided in one embodiment of the present application;

[0043] Figure 4It is a structural schematic diagram of a foreground cutout network device provided in one embodiment of the present application;

[0044] Figure 5 It is a structural schematic diagram of a second acquisition module in a foreground cutout device provided in an embodiment of the present application;

[0045] Figure 6 It is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0047] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] "One embodiment" or "some embodiments" described in the specification of this application means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0049] In addition, in the description of the present application, "plurality" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0050] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0051] Figure 1 1 is a schematic diagram of the implementation process of a foreground cutout method provided by an embodiment of the present application. The foreground cutout method in this embodiment can be executed by an electronic device. The electronic device includes but is not limited to a computer, a tablet computer, a server, a mobile phone, a camera or a wearable device, etc. Among them, the server includes but is not limited to an independent server or a cloud server, etc. Figure 1 As shown, the foreground cutout method may include steps S110 to S140.

[0052] S110, obtaining a current frame color image and a depth image corresponding to the current frame, a historical frame color image and a corresponding transparency map.

[0053] Specifically, the historical frame color image and the transparency map corresponding to the historical frame can be the previous or preceding frame color image and its corresponding transparency map. The historical frame color image and the transparency map corresponding to the historical frame can be stored in the memory of the electronic device. It should be noted that the transparency (Alpha) map in the present application is composed of the transparency of each pixel in the image, which can be specifically represented by the Alpha channel.

[0054] In some implementations, a color image and a depth image of a target object entering a scene are collected by an RGBD camera. Specifically, when a target object enters a scene, the RGBD camera is turned on to collect a color image of a current frame and a depth image corresponding to the current frame.

[0055] S120, performing multiple erosion and dilation processing on the depth image corresponding to the current frame to generate a current frame trisection image; and splicing the historical frame color image and its corresponding transparency image to generate a historical frame fusion image.

[0056] In some embodiments, the depth image corresponding to the current frame is segmented to generate a trimap of the current frame, where the trimap includes a foreground area, an unknown area, and a background area, thereby obtaining an initial segmentation result.

[0057] In some embodiments, the effective depth range or depth threshold is first obtained, and then the depth image corresponding to the current frame is segmented according to the effective depth range or depth threshold to obtain the foreground soft segmentation image of the current frame; the foreground soft segmentation map of the current frame is eroded multiple times to obtain the foreground area of ​​the current frame trisection map; the foreground soft segmentation map of the current frame is dilated multiple times and then the foreground area is subtracted to obtain the unknown area of ​​the current frame trisection map; the part of the current frame other than the foreground area and the unknown area is the background area of ​​the current frame trisection map. As a non-limiting example, the number of erosion processes is 3 times, and the number of dilation processes is 3 times. By first segmenting the depth image corresponding to the current frame according to the depth value to generate the foreground soft segmentation image of the current frame, and then processing the foreground soft segmentation image of the current frame to obtain the trisection map of the current frame, on the one hand, the accuracy of the trisection map of the current frame is improved, thereby improving the accuracy of the foreground segmentation prediction, and on the other hand, using the trisection map of the current frame as an input can achieve lightweight cutout model.

[0058] In one embodiment, a 3×3 structure matrix is ​​selected, in which all elements in the matrix are 1, and each pixel in the foreground soft segmentation map is scanned with a step size of 1. The structure matrix and the foreground soft segmentation map are used to perform a logical AND operation. If the values ​​of the structure matrix and the foreground soft segmentation map are both 1, the pixel value of the point in the output image is 1, and in other cases the pixel value of the output image is 0. This process is called an erosion process, and the foreground soft segmentation map is reduced by one circle through the erosion process.

[0059] In one embodiment, a 3×3 structure matrix is ​​selected, and all elements in the matrix are 1. The structure matrix is ​​used to perform a logical AND operation with the foreground soft segmentation image. If the values ​​of the structure matrix and the foreground soft segmentation image are both 0, the pixel value of the point in the output image is 0. In other cases, the pixel value of the output image is 1. This process is called an expansion process, and the foreground soft segmentation image is expanded by one circle through the expansion process.

[0060] For ease of understanding, the effective depth range or depth threshold is briefly described. The effective depth range or depth threshold can be understood as extracting all objects within a specified distance. When traversing the pixels in the depth image corresponding to the current frame, if the depth value of a pixel on the depth image is within the effective depth range or less than the depth threshold and greater than zero, the pixel is a valid pixel and the pixel value is set to 1; otherwise, the pixel is an invalid pixel and the pixel value is set to 0.

[0061] More specifically, in some embodiments, the effective depth range or depth threshold can be customized by the user, for example, input by the user, and the electronic device can obtain the effective depth range or depth threshold input by the user; in some other embodiments, the effective depth range or depth threshold can be set by the system by default. The effective depth range or depth threshold can be designed according to actual needs. For example, if the user sets the effective shooting range of the depth camera to 1 to 3 meters, the pixels with a depth between 1 and 3 meters on the depth image are considered to be valid pixels, and the rest are invalid pixels. The effective depth range is used to segment the depth image of the current frame to obtain a foreground soft segmentation map. For another example, if the user sets the depth threshold of the depth camera to 3 meters, the pixels with a depth between 0 and 3 meters on the depth image are considered to be valid pixels, and the rest are invalid pixels. The depth threshold is used to segment the depth image of the current frame to obtain a foreground soft segmentation map.

[0062] Specifically, the historical frame color image and the transparency map corresponding to the historical frame are spliced ​​together from the channel dimension to obtain the historical frame fusion map pre_color_alpha.

[0063] Here, on the one hand, not only the information of the current frame is considered, but also the transparency of the historical frame (that is, the result of the historical frame); on the other hand, because the transparency of the historical frame may have the possibility of prediction errors, the color image of the historical frame is added as additional information, which can greatly improve the stability of the cutout, that is to say, the stability of the foreground segmentation edge can be guaranteed.

[0064] S130, generating a coding feature map according to the current frame color image, the historical frame fusion map, and the current frame tri-map.

[0065] Specifically, this embodiment prepares three inputs in total: the current frame color image Color t , historical frame color image Color t-1 Alpha corresponding to the historical frame t-1 The spliced ​​historical frame fusion map pre_color_alpha and the current frame trimap t , obtain the encoded feature map based on these three inputs.

[0066] S140, using the encoded feature map to obtain a transparency map corresponding to the current frame, and extracting the foreground according to the transparency map.

[0067] Specifically, the encoded feature map is decoded to obtain the transparency map corresponding to the current frame, denoted as Alpha t . Further, pixel alignment is performed with the color image of the current frame according to the transparency map corresponding to the current frame, and the foreground part of the color image of the current frame is extracted. Thus, the foreground segmentation prediction of the color image of the current frame is completed.

[0068] Since the encoded feature map fuses the information of the current frame and the historical frames, a transparency map of the current frame with more stable edges can be obtained.

[0069] It should be noted that whenever the RGBD camera collects new data, that is, collects a new frame of color image and a corresponding depth image, steps S110 to S140 need to be re-executed to generate a new foreground transparency map.

[0070] It should also be noted that, based on any of the foregoing embodiments, if there is no transparency map corresponding to the previous frame, the transparency map corresponding to the previous frame may be replaced by the current frame trisection map.

[0071] In some embodiments, if prediction starts from the second frame data collected by the RGBD camera, there is no transparency map corresponding to the previous frame, that is, the first frame. At this time, the transparency map corresponding to the first frame needs to be replaced by the current frame tripartite map.

[0072] In some other embodiments, if the prediction starts from the first frame data collected by the RGBD camera, there is no color image and corresponding transparency map of the previous frame. At this time, it is necessary to replace the color image of the previous frame with the color image of the current frame, and replace the transparency map of the previous frame with the current frame triplicate map.

[0073] Figure 2 FIG. 1 is a schematic diagram of the implementation flow of steps S130 and S140 in a foreground cutout method provided in an embodiment of the present application. Figure 2 As shown, step S130 may include steps S131 to S133, and step S140 may include step S141.

[0074] S131, performing feature extraction on the current frame color image, the historical frame fusion image and the current frame tri-image, and obtaining a first feature image of the current frame color image, a second feature image of the historical frame fusion image and a third feature image of the current frame tri-image respectively.

[0075] Specifically, in this embodiment, a cutout model is pre-built and deployed in the electronic device in advance. Figure 3 As shown in Figure 1, the cutout model includes three feature extraction networks in parallel. The three feature extraction networks respectively extract the color image of the current frame. t , historical frame fusion map pre_color_alpha, current frame trimap t Perform feature extraction to obtain the feature maps of the three.

[0076] For example, the current frame color image Color t Input the first feature extraction network, input the historical frame fusion map pre_color_alpha into the second feature extraction network, and input the current frame trimap t Input the third feature extraction network and output the current frame color image Color t The first feature map, the second feature map of the historical frame fusion map pre_color_alpha, and the current frame tripogram Trimap t The third characteristic diagram of .

[0077] In some embodiments, the feature extraction network may be designed based on a residual network (Resnet). The three feature extraction networks may use the same network or different networks.

[0078] In one embodiment, the feature extraction network may include three convolution modules, each of which includes a convolution layer, a batch normalization layer, and a Relu activation function layer.

[0079] S132, fusing the first feature map and the second feature map to obtain a first feature fusion map; fusing the first feature map and the third feature map to obtain a second feature fusion map.

[0080] Specifically, Figure 3 As shown, the cutout model pre-built in this embodiment also includes a feature combination module connected to the output of the feature extraction network. The feature combination module is configured to fuse the first feature map and the second feature map to obtain a first feature fusion map, and to fuse the first feature map and the third feature map to obtain a second feature fusion map. The feature combination module includes at least one fusion layer.

[0081] The first feature map and the second feature map are subjected to feature fusion through the first fusion layer to obtain a first feature fusion map; the first feature map and the third feature map are subjected to feature fusion through the second fusion layer to obtain a second feature fusion map. The first fusion layer and the second fusion layer can be the same fusion layer or two fusion layers.

[0082] It should be noted that using the fusion layer to fuse feature images refers to splicing two feature images in the channel dimension. The spliced ​​feature map contains rich semantics after multiple convolution operations.

[0083] S133, concatenating the first feature map, the first feature fusion map, and the second feature fusion map to obtain a coding feature map.

[0084] Specifically, Figure 3 As shown, the cutout model pre-built in this embodiment also includes a feature combination module connected to the output of the feature combination module. The feature combination module is used to splice the first feature map, the first feature fusion map, and the second feature fusion map. The first feature map, the first feature fusion map, and the second feature fusion map are input to the feature combination module, and feature maps of multiple channels are generated by the feature combination module.

[0085] In one embodiment, a feature combination module consisting of a convolutional layer, a batch normalization layer, and a ReLU activation function layer is used to concatenate the first feature map, the first feature fusion map, and the second feature fusion map to fuse semantic features of different modalities, obtain an encoded feature map, and transmit it to a decoding module.

[0086] S141, decode the encoded feature map and output a transparency map corresponding to the current frame.

[0087] Specifically, Figure 3 As shown, the pre-built cutout model of this embodiment also includes a decoding module connected to the output of the feature combination module.

[0088] In one embodiment, a decoder composed of multiple deconvolution layers, or a decoding module composed of upsampling layers and convolution layers is used to decode the encoded feature map, and the detail information retained by the encoded feature map is output, and the detail information includes a transparency map corresponding to the current frame, that is, a transparency matrix, thereby completing a high-quality cutout operation.

[0089] S142, extracting the target foreground in the color image of the current frame according to the transparency map corresponding to the current frame.

[0090] Furthermore, after obtaining the transparency map corresponding to the current frame, the color image of the current frame and the transparency map corresponding to the current frame can be pixel-aligned to extract the target foreground in the color image of the current frame. The foreground can be synthesized with different background images to obtain multiple different synthesized images, thereby realizing the operation of extracting the foreground from the color image and synthesizing it with other background images to obtain multiple synthesized images of different backgrounds and the same foreground combination.

[0091] It should be noted that the cutout model of this embodiment is a trained model. The present application embodiment does not specifically limit the training method of the cutout model, and any training method for the model can be used in this application.

[0092] In some embodiments, real, continuous-frame color images and corresponding depth images may be collected to train the initial cutout model to obtain a trained cutout model.

[0093] In some other embodiments, a single frame of data, i.e., a single frame of color image and a corresponding depth image, can also be used to train the initial cutout model to obtain a trained cutout model. In these embodiments, the single frame of data needs to simulate the generation of the previous frame of data due to the lack of the previous frame of data. In some implementations, the color image of the current frame can be annotated, and the same translation, scaling, rotation, etc. can be performed to achieve data enhancement to simulate the color image and transparency image of the previous frame.

[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] An embodiment of the present application also provides a foreground cutout device. For details not described in the foreground cutout device, please refer to the description of the foreground cutout method embodiment above.

[0096] Figure 4 4 is a schematic block diagram of a foreground cutout device provided by an embodiment of the present application. The foreground cutout device comprises: a first acquisition module 41 , a generation module 42 , a second acquisition module 43 and a third acquisition module 44 .

[0097] The first acquisition module 41 is used to acquire the current frame color image and the depth image corresponding to the current frame, the historical frame color image and the corresponding transparency map;

[0098] The generating module 42 is used to perform multiple erosion and dilation processing on the depth image corresponding to the current frame to generate a current frame trisection image; and to splice the historical frame color image and its corresponding transparency image to generate a historical frame fusion image;

[0099] A second acquisition module 43 is used to acquire a coding feature map according to the current frame color image, the historical frame fusion map and the current frame tri-map;

[0100] The third acquisition module 44 is used to obtain a transparency map corresponding to the current frame using the encoded feature map, and extract the foreground according to the transparency map.

[0101] In some embodiments, Figure 5 As shown, the second acquisition module 43 specifically includes: an extraction execution module 431 , a fusion execution module 432 and a splicing execution module 433 .

[0102] The extraction execution module 431 is used to extract features from the current frame color image, the historical frame fusion image and the current frame tri-image, and obtain a first feature image of the current frame color image, a second feature image of the historical frame fusion image and a third feature image of the current frame tri-image respectively;

[0103] The fusion execution module 432 is used to fuse the first feature map and the second feature map to obtain a first feature fusion map; and fuse the first feature map and the third feature map to obtain a second feature fusion map;

[0104] The splicing execution module 433 is used to splice the first feature map, the first feature fusion map and the second feature fusion map to obtain a coding feature map.

[0105] In some embodiments, the third acquisition module 44 is specifically configured to:

[0106] Decode the encoded feature map and output the transparency map corresponding to the current frame.

[0107] In some embodiments, the splicing execution module 433 is specifically used to:

[0108] The first feature map, the first feature fusion map and the second feature fusion map are concatenated using a feature combination module including a convolution layer, a batch normalization layer and an activation function layer to obtain a coded feature map.

[0109] In some embodiments, the extraction execution module 431 is specifically used to:

[0110] The feature extraction network is used to extract features from the current frame color image, the historical frame fusion image and the current frame tri-image. The feature extraction network includes three convolution modules, each of which includes a convolution layer, a batch normalization layer and an activation function layer.

[0111] In some embodiments, the generating module 42 is specifically used to:

[0112] Segment the depth image corresponding to the current frame to obtain the foreground soft segmentation map of the current frame;

[0113] The foreground soft segmentation map of the current frame is eroded multiple times to obtain the foreground area of ​​the current frame trisection map; the foreground soft segmentation map of the current frame is expanded multiple times and then the foreground area is subtracted to obtain the unknown area of ​​the current frame trisection map; the part of the current frame soft segmentation map except the foreground area and the unknown area is the background area of ​​the current frame trisection map.

[0114] In some embodiments, segmenting the depth image corresponding to the current frame to obtain a foreground soft segmentation map of the current frame includes:

[0115] Obtain a depth threshold; traverse the pixels in the depth image corresponding to the current frame, set pixel values ​​greater than zero and less than the depth threshold to 1, and set pixel values ​​greater than or equal to the depth threshold to 0, to obtain a foreground soft segmentation map of the current frame; or

[0116] Get the valid depth range; traverse the pixels in the depth image corresponding to the current frame, set the pixel values ​​belonging to the valid depth range to 1, and set the pixel values ​​not belonging to the valid depth range to 0, to obtain the foreground soft segmentation map of the current frame.

[0117] In some embodiments, when the transparency map corresponding to the historical frame does not exist, the transparency map corresponding to the historical frame is replaced by the current frame trisection map.

[0118] In these embodiments, the generating module 42 is specifically used to: generate a current frame tripartite map based on a depth image corresponding to the current frame; and generate a historical frame fusion map based on the historical frame color image and the current frame tripartite map.

[0119] An embodiment of the present application also provides an electronic device, such as Figure 6 As shown, the electronic device may include an RGBD camera 63, one or more processors 60 ( Figure 6Only one is shown in the figure), a memory 61 and a computer program 62 stored in the memory 61 and executable on one or more processors 60, for example, a foreground cutout program. The RGBD camera 63 is used to capture the color image of the current frame and the depth image corresponding to the current frame. When one or more processors 60 execute the computer program 62, the various steps in the foreground cutout method embodiment can be implemented. Alternatively, when one or more processors 60 execute the computer program 62, the functions of the various modules / units in the foreground cutout device embodiment can be implemented, which is not limited here.

[0120] Exemplarily, the computer program 62 may be divided into one or more modules / units, one or more modules / units are stored in the memory 61 and executed by the processor 60 to complete the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the processing unit. For example, the computer program 62 may be divided into the following modules. The specific functions of each module are as follows:

[0121] The first acquisition module is used to acquire the current frame color image and the depth image corresponding to the current frame, the historical frame color image and the corresponding transparency map;

[0122] A generation module is used to perform multiple corrosion and expansion processes on the depth image corresponding to the current frame to generate a current frame trisection image; and to splice the historical frame color image and its corresponding transparency image to generate a historical frame fusion image;

[0123] The second acquisition module is used to acquire a coding feature map according to the current frame color image, the historical frame fusion image and the current frame tri-image;

[0124] The third acquisition module is used to obtain a transparency map corresponding to the current frame using the encoded feature map, and extract the foreground according to the transparency map.

[0125] Those skilled in the art will understand that Figure 6 The electronic device is merely an example of an electronic device and does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0126] In one embodiment, the processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] In one embodiment, the memory 61 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 61 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 61 may also include both an internal storage unit of the electronic device and an external storage device. The memory 61 is used to store computer programs and other programs and data required by the electronic device. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0128] An embodiment of the present application also provides another preferred embodiment of an electronic device. In this embodiment, the electronic device includes an RGBD camera and one or more processors. The RGBD camera is used to collect a color image of a current frame and a depth image corresponding to the current frame. The one or more processors are used to execute the following program modules stored in the memory:

[0129] The first acquisition module is used to acquire the current frame color image and the depth image corresponding to the current frame, the historical frame color image and the corresponding transparency map;

[0130] A generation module is used to perform multiple corrosion and expansion processes on the depth image corresponding to the current frame to generate a current frame trisection image; and to splice the historical frame color image and its corresponding transparency image to generate a historical frame fusion image;

[0131] The second acquisition module is used to acquire a coding feature map according to the current frame color image, the historical frame fusion image and the current frame tri-image;

[0132] The third acquisition module is used to obtain a transparency map corresponding to the current frame using the encoded feature map, and extract the foreground according to the transparency map.

[0133] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0134] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, each step in the foreground cutout method embodiment can be implemented.

[0135] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement each step in the foreground cutout method embodiment.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0142] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A foreground cutout method, characterized in that: include: Get the current frame color image and the depth image corresponding to the current frame, the historical frame color image and its corresponding transparency map; Performing multiple corrosion and expansion processes on the depth image to generate a current frame trisection image; and splicing the historical frame color image and its corresponding transparency image to generate a historical frame fusion image; According to the current frame color image, the historical frame fusion image and the current frame trisection image, a coding feature image is obtained; including: Performing feature extraction on the current frame color image, the historical frame fusion image, and the current frame tri-image, to obtain a first feature image of the current frame color image, a second feature image of the historical frame fusion image, and a third feature image of the current frame tri-image, respectively; The first feature map and the second feature map are fused to obtain a first feature fusion map; the first feature map and the third feature map are fused to obtain a second feature fusion map; splicing the first feature map, the first feature fusion map and the second feature fusion map to obtain the encoding feature map; The method comprises: obtaining a transparency map corresponding to the current frame by using the encoded feature map, and extracting the foreground according to the transparency map; comprising: The encoded feature map is decoded, and a transparency map corresponding to the current frame is output.

2. The foreground cutout method according to claim 1, characterized in that: The step of splicing the first feature map, the first feature fusion map, and the second feature fusion map to obtain a coding feature map includes: The first feature map, the first feature fusion map and the second feature fusion map are concatenated using a feature combination module including a convolution layer, a batch normalization layer and an activation function layer to obtain a coding feature map.

3. The foreground cutout method according to claim 2, characterized in that: The extracting features of the current frame color image, the historical frame fusion image and the current frame trisection image includes: A preset feature extraction network is used to perform feature extraction on the current frame color image, the historical frame fusion image and the current frame tri-image. The preset feature extraction network includes three convolution modules, and each convolution module includes a convolution layer, a batch normalization layer and an activation function layer.

4. The foreground cutout method according to any one of claims 1 to 2, characterized in that: The step of performing multiple erosion and dilation processes on the depth image corresponding to the current frame to generate a current frame trisection image includes: Segmenting the depth image corresponding to the current frame to obtain a foreground soft segmentation map of the current frame; The foreground soft segmentation map of the current frame is subjected to multiple erosion processes to obtain a foreground area of ​​the current frame trisection map; the foreground soft segmentation map of the current frame is subjected to multiple dilation processes and then the foreground area is subtracted to obtain an unknown area of ​​the current frame trisection map; the part of the current frame soft segmentation map except the foreground area and the unknown area is the background area of ​​the current frame trisection map.

5. The foreground cutout method according to claim 4, characterized in that: The depth image corresponding to the current frame is segmented to obtain a foreground soft segmentation map of the current frame, including: Obtain a depth threshold; traverse the pixels in the depth image corresponding to the current frame, set pixel values ​​greater than zero and less than the depth threshold to 1, and set pixel values ​​greater than or equal to the depth threshold to 0, to obtain a foreground soft segmentation map of the current frame; or Obtain a valid depth range; traverse the pixels in the depth image corresponding to the current frame, set the pixel values ​​belonging to the valid depth range to 1, and set the pixel values ​​not belonging to the valid depth range to 0, to obtain a foreground soft segmentation map of the current frame.

6. The foreground cutout method according to any one of claims 1 to 2, characterized in that: In the case that the transparency map corresponding to the historical frame does not exist, the transparency map corresponding to the historical frame is replaced by the current frame trisection map.

7. A foreground cutout device, characterized in that: include: The first acquisition module is used to acquire the current frame color image and the depth image corresponding to the current frame, the historical frame color image and the corresponding transparency map; A generation module is used to perform multiple corrosion and expansion processes on the depth image corresponding to the current frame to generate a current frame trisection image; and to splice the historical frame color image and its corresponding transparency image to generate a historical frame fusion image; The second acquisition module is used to acquire a coding feature map according to the current frame color image, the historical frame fusion map and the current frame tri-map; including: Performing feature extraction on the current frame color image, the historical frame fusion image, and the current frame tri-image, to obtain a first feature image of the current frame color image, a second feature image of the historical frame fusion image, and a third feature image of the current frame tri-image, respectively; The first feature map and the second feature map are fused to obtain a first feature fusion map; the first feature map and the third feature map are fused to obtain a second feature fusion map; splicing the first feature map, the first feature fusion map and the second feature fusion map to obtain the encoding feature map; The third acquisition module is used to obtain a transparency map corresponding to the current frame using the encoding feature map, and extract the foreground according to the transparency map; including: The encoded feature map is decoded, and a transparency map corresponding to the current frame is output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the foreground cutout method according to any one of claims 1 to 6 is implemented.

9. The electronic device according to claim 8, characterized in that: It also includes an RGBD camera, which is used to collect a current frame color image and a depth image corresponding to the current frame. The processor uses the current frame color image and the depth image corresponding to the current frame to implement the foreground cutout method as described in any one of claims 1 to 6.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the foreground cutout method according to any one of claims 1 to 6 is implemented.

Images