A cutout network training method, cutout method and device

Abstract

The present application relates to the field of image processing technology, and in particular to a cutout network training method, a cutout method and a device. The cutout network training method includes: obtaining several frames of color composite images and color images and depth images containing foregrounds and aligned; obtaining a mask in the color composite image, and simulating the depth characteristics of the mask to obtain a tripartite diagram corresponding to the color composite image; based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and simulating the depth characteristics of the soft segmentation image to obtain a tripartite diagram corresponding to the color image; inputting the color image and its corresponding tripartite diagram, and the color composite image and its corresponding tripartite diagram into the initial cutout network for iterative optimization to obtain a target cutout network. The embodiment of the present application no longer requires manual annotation of the tripartite diagram, which reduces labor costs and improves efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cutout network training method, a cutout method and a device. Background Art

[0002] Currently, the mainstream cutout algorithms can be divided into three types. One is the trimap-based method, which can obtain a foreground image with higher precision, but it requires manual annotation of the given image, which is very user-unfriendly, making this method less practical. The other is the trimap-free cutout method, which uses a pure RGB image to directly predict the foreground, but this method is sensitive to the scene, and the generalization performance of diversified scenes still needs to be improved.

[0003] In order to balance the advantages and disadvantages of the first two methods, the third method in the mainstream cutout algorithm is to use soft segmentation as the prior image for background segmentation and obtain the foreground image through the RGB image and the background image. However, this method requires the use of a complex segmentation network, which results in large amount of calculation and slow speed, making it difficult to achieve real-time performance. In addition, this method also requires taking an additional background image, which can easily lead to a poor user experience. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a cutout network training method, a cutout method and a device, which can solve at least one technical problem in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for training a cutout network, comprising: obtaining several frames of color composite images and color images and depth images containing foregrounds and aligned; obtaining a mask in the color composite image, and performing depth characteristic simulation on the mask to obtain a tripartite diagram corresponding to the color composite image; based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and performing depth characteristic simulation on the soft segmentation image to obtain a tripartite diagram corresponding to the color image; inputting the color image and its corresponding tripartite diagram, and the color composite image and its corresponding tripartite diagram into an initial cutout network for iterative optimization to obtain a target cutout network.

[0006] In this embodiment, the depth characteristics are simulated to obtain a three-division map, and manual annotation of the three-division map is no longer required, thereby reducing labor costs and improving efficiency.

[0007] In a second aspect, an embodiment of the present application provides a cutout method, comprising: obtaining a color image and a depth image that include a foreground and are aligned; based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and processing the soft segmentation image to obtain a tri-map corresponding to the color image; inputting the color image and its corresponding tri-map into the target cutout network obtained by the cutout network training method described in the first aspect to obtain a mask that only includes the foreground.

[0008] In the third aspect, an embodiment of the present application provides a cutout network training device, comprising: an acquisition unit, used to acquire several frames of color composite images and color images and depth images containing foregrounds and aligned; a preprocessing unit, used to acquire a mask in the color composite image, and perform depth characteristic simulation on the mask to obtain a tripartite map corresponding to the color composite image; based on the alignment relationship between the color image and the depth image, use the depth image to acquire a soft segmentation image and perform depth characteristic simulation on the soft segmentation image to obtain a tripartite map corresponding to the color image; a training execution unit, used to input the color image and its corresponding tripartite map, and the color composite image and its corresponding tripartite map into an initial cutout network for iterative optimization to obtain a target cutout network.

[0009] In the fourth aspect, an embodiment of the present application provides a cutout device, comprising: an acquisition unit, used to acquire a color image and a depth image that include a foreground and are aligned; a preprocessing unit, used to obtain a soft-segmented image using the depth image based on the alignment relationship between the color image and the depth image, and process the soft-segmented image to obtain a tri-image corresponding to the color image; a cutout execution unit inputs the color image and its corresponding tri-image into the target cutout network obtained by the cutout network training device described in the embodiment of the third aspect, to obtain a mask that only includes the foreground.

[0010] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cutout network training method as described in the embodiment of the first aspect is implemented; or the cutout method as described in the embodiment of the second aspect is implemented.

[0011] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cutout network training method as described in the embodiment of the first aspect is implemented; or the cutout method as described in the embodiment of the second aspect is implemented.

[0012] In the seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the cutout network training method as described in the embodiment of the first aspect; or executes the cutout method as described in the embodiment of the second aspect.

[0013] It should be understood that the beneficial effects of the embodiments of the second to seventh aspects can be found in the relevant description of the embodiment of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 It is a schematic diagram of the implementation process of a cutout network training method provided in one embodiment of the present application;

[0016] Figure 2 is a process schematic diagram of step S120 provided in one embodiment of the present application;

[0017] Figure 3 is a schematic diagram of four color composite images and their corresponding masks provided in an embodiment of the present application;

[0018] Figure 4 is a process schematic diagram of step S120 provided in another embodiment of the present application;

[0019] Figure 5 It is a structural schematic diagram of a cutout network provided by an embodiment of the present application;

[0020] Figure 6 This is a schematic diagram of an implementation process of a cutout method provided in an embodiment of the present application;

[0021] Figure 7 It is a structural schematic diagram of a cutout network training device provided in one embodiment of the present application;

[0022] Figure 8 It is a structural schematic diagram of a cutout device provided in one embodiment of the present application;

[0023] Fig. 9 It is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] "One embodiment" or "some embodiments" described in the specification of this application means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] In addition, in the description of the present application, "plurality" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0029] Figure 1 This is a schematic diagram of the implementation process of a cutout network training method provided in an embodiment of the present application. The cutout network training method in this embodiment can be executed by an electronic device. The electronic device includes but is not limited to a computer, a tablet computer, a server, a mobile phone, a camera or a wearable device. Among them, the server includes but is not limited to an independent server or a cloud server. Figure 1 As shown, the cutout network training method may include steps S110 to S130.

[0030] S110, obtaining a plurality of frames of color composite images and color images and depth images containing foreground and aligned.

[0031] In the embodiment of the present application, the acquired image data may include two parts, the first part is real image data, that is, image data directly obtained by acquisition; the other part is synthetic image data, including color synthetic images, that is, images synthesized by using color foreground images and color background images. Relative to real image data, synthetic image data can be understood as virtual image data, wherein the amount of real image data used in the present application is much less than the amount of synthetic image data, thereby reducing the labor cost of manually acquiring training images and also improving efficiency.

[0032] In some possible implementations, the foreground is taken as a human body for illustrative purposes. On the one hand, the camera can be used to directly capture a color image of the same scene including a human body and its corresponding depth image to obtain real image data. Furthermore, in this embodiment, after obtaining the color image and the depth image, the images need to be aligned, that is, the pixels in each image are aligned one by one. On the other hand, a crawling tool can be used to crawl multiple color foreground images and multiple color background images from the Internet, and then each color foreground image and any color background image are synthesized to obtain a color composite image.

[0033] S120, obtaining a mask in the color composite image, and performing depth characteristic simulation on the mask to obtain a tri-map corresponding to the color composite image; based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and performing depth characteristic simulation on the soft segmentation image to obtain a tri-map corresponding to the color image.

[0034] Step S120 is the data enhancement stage, which is divided into two parts, one for a small amount of real image data and the other for a large amount of synthetic image data. In this stage, the tri-map is no longer manually annotated, but obtained through data enhancement, which further reduces labor costs and improves efficiency.

[0035] For a small amount of real image data collected, in some embodiments, the depth image is first segmented using a distance threshold to obtain a soft segmentation image, and then a depth characteristic simulation is performed on the soft segmentation image to obtain a first image, and then a morphological erosion and dilation operation is used to generate a tri-map of the first image. In some other embodiments, such as Figure 2 As shown, the depth image is first segmented by a distance threshold to obtain a soft segmentation image, and then the intermediate three-dimensional image is generated by morphological erosion and dilation operations on the soft segmentation image, and then the depth characteristics of the intermediate three-dimensional image are simulated to obtain the three-dimensional image. The depth characteristic simulation includes randomly punching holes in the image to simulate the real depth occlusion problem to enhance the depth characteristics of the depth image. For example, random hole sampling is performed on the edge and / or inside of a half-length portrait, and the corresponding pixel value is assigned to 0.

[0036] It should be noted that, based on the alignment relationship between the color image and the depth image in step S110 , the tri-map obtained from the depth image can be converted into a tri-map corresponding to the color image.

[0037] For a large amount of color composite image data, it is necessary to first obtain the mask of the color composite image data, that is, the binary image including only the foreground, such as Figure 3 Further, after obtaining the mask of the color composite image data, the mask can be simulated for depth characteristics to obtain a second image, and then the second image can be triangulated using morphological erosion and dilation operations. In some other embodiments, Figure 4 As shown in the figure, the mask is firstly eroded and expanded by morphological operations to generate an intermediate three-dimensional image, and then the intermediate three-dimensional image is simulated by depth characteristics to obtain a three-dimensional image. The depth characteristics simulation includes randomly punching holes at the edge and inside of the foreground of the image to simulate the real depth occlusion problem. For example, randomly punch holes at the edge and inside of a half-length portrait and assign the corresponding pixel value to 0.

[0038] By simulating the depth characteristics of real image data and synthetic image data to enhance the depth characteristics of the image, and using it as the input image to train the cutout network model, the trained cutout network model can have the ability to complete the depth.

[0039] In some embodiments, a tripartite graph (or intermediate tripartite graph) is generated for an image (e.g., a soft segmented image or a mask, etc.) using morphological erosion and dilation operations, including: first performing multiple dilation operations on the image to obtain a dilated image, performing multiple erosion operations on the image to obtain a eroded image, and then obtaining a tripartite graph based on the dilated image and the eroded image. Specifically, the dilated image can be subtracted from the eroded image to obtain the tripartite graph. In a possible implementation, the number of dilation operations and erosion operations can be three times, which can ensure both data processing efficiency and the accuracy of the results. It should be noted that the number of dilation and erosion operations is not limited in this embodiment.

[0040] S130, iteratively optimizing the initial cutout network using the color image and its corresponding tri-plot, the color composite image and its corresponding tri-plot, to obtain a target cutout network.

[0041] The initial cutout network is pre-stored in the electronic device as a network model to be trained. The initial cutout network includes a set of network parameters to be learned. After the initial cutout network is trained, the target cutout network is obtained. The target cutout network has the same network structure as the initial cutout network, but the network parameters of the two are different.

[0042] In some embodiments, the initial cutout network (or target cutout network) may include a neural network model based on deep learning, such as a ResNet or VGG network.

[0043] In some embodiments, the network structure of the initial cutout network (or target cutout network) is as follows: Figure 5 As shown, it includes one stage network or multiple stage networks connected in series and is lightweight through specific network design. Furthermore, the initial cutout network uses a color image Image (i.e., a color image containing a foreground and a color composite image) and a three-part image corresponding to the simulated depth feature Depth as input, and outputs a refined foreground image step by step for multiple times, thereby achieving a more reliable foreground segmentation prediction.

[0044] In one embodiment, the stage includes a context combining module (CCM), a stem block, and a prediction module predictor. The CCM module is used to output low-order features and high-order features after feature exchange; the stem block preferably includes a feature fusion module (FFM) for fusing low-order features and high-order features based on an attention mechanism to obtain fused features, and the prediction module is used to output a prediction mask based on the fused features.

[0045] Specifically, the CCM module is used to exchange the features of two input images to generate two output features, a low-order feature F L and a high-order feature F H First, the low-level features corresponding to each input image are encoded separately as features F 1I =E 1I (Image), and F 1S =E 1S (Depth), and then concatenate them to produce an overall low-order feature F L =Cat(F 1I ,F 1S ), where Cat represents a concatenation operation. In the other branch, each low-order feature is further downsampled to a single high-order feature F 2I =E 2I (F 1I ), and F 2S =E 2S (F 1S ). By fusing two single high-level features, we get Cat(F 2I ,F 2S ), and use F H =C IS (Cat(F 2IB ,F2S ))Get the overall high-order feature F H In the network's main stem block, the encoder E 3 For high-order features F H Down-sampled, and then decoded by D 3 Upsample and then use the first feature fusion module FFM 1 Fusion is performed to obtain feature F 2 =FFM 1 (F H ,D 3 (E 3 (F H ))). Further through the second feature fusion module FFM 2 The low-order features F L Merge into the main block to obtain the fusion feature F 1 =FFM 2 (F L ,D 2 (F 2 )). Finally, use α=D 1 (F 1 ) to obtain a prediction mask so that the prediction mask can be used for image synthesis later.

[0046] The above process is called a stage. In order to predict finer structural details, in some embodiments of the present application, another stage can be used to further refine the output of the previous stage, using the prediction mask of the previous stage as the prior mask of the next stage, and the color image Image input by the previous stage continues to be the prior of the next stage. This process can be repeated multiple times to form a series network structure. Preferably, the number of stages is 2.

[0047] It should be noted that Figure 5 A lighter backbone network (i.e., stage) is provided, which only requires 3 downsamplings. In addition, the attention mechanism provided in this application enables the network model to extract features by using fewer weights; the stage network uses a CCM module to exchange the contextual information of the color image and the triplicate image to fully extract the boundary features. This example is expanded to 2 inputs. After the CCM module fuses a single input image, it further post-fuses the fused features, so that the feature information of different inputs can be more effectively matched and learned. In addition, the CCM module uses fewer convolutional layers to extract the features of a single input, and then splices them together; through this design, the CCM module is lighter than the corresponding ResNet module because fewer convolutional channels are introduced before concatenation.

[0048] In one embodiment, the FFM module responsible for feature fusion is used to replace the traditional splicing operation to achieve better fusion of encoder features and decoder features by utilizing the attention mechanism. In the encoder, the structural features are extracted layer by layer to form high-order semantic features, and the semantic features are conducive to the model to use a wider receptive field range to determine the position of the segmentation edge. For example, when the foreground and background colors are similar (such as a black human head and a black background), it is difficult to judge the boundary (the boundary between the head and the black background) directly based on the local structural information, but the use of high-order semantic information can assist in the determination of the segmentation position through the empirical information of image recognition (such as a human head is usually round). The FFM module converts the high-order semantic features of the encoder into a spatial attention mask to guide the restoration of structural information in the decoder. Since only high-order semantic features can provide accurate spatial attention, the low-order features from the encoder are not suitable for using the FFM module. Therefore, when designing the network, the FFM module is only applied to the internal skip connection (innerskip connection), and not to the external skip connection (outer skip connection).

[0049] It can be seen that the network is lightweight in three aspects. First, the network depth is shallow; unlike traditional backbone networks such as ResNet and GoogleNet, they downsample the input 5 times to extract rich semantic clues. Due to the use of soft segmentation prior, background cutout becomes a task that relies less on semantics but more on structural information, so the network does not need to be too deep; only downsampling the input 3 times, the semantic information obtained is sufficient, and rich structural clues are retained. Second, there are fewer channels, because background cutout is not a classification task, so one channel can serve multiple objects; in addition, the input triplicate map provides ready-made features, so fewer channels are required for information extraction. Third, the CCM module is lighter than its corresponding residual network.

[0050] In one embodiment, a mixed loss function including a variety of different loss functions, such as mean square error (MSE) loss and compositional loss (CL), is used to train the network. The MSE loss function is a conventional pixel regression loss. The CL loss function synthesizes a new color image based on the prediction mask, and then calculates the loss with the original color image; wherein, the new color image needs to obtain the predicted foreground by multiplying the prediction mask with the original color image, and synthesizes the predicted foreground and background image, and the background image is the same as the background in the original color image. Due to the use of a mixed loss function, the foreground and boundaries can be detected more accurately. In one embodiment, the weighted value of two different loss functions is used as a mixed loss function, or a joint loss function, which is defined as:

[0051] L=λ 1 L MSE +λ 2 L CL .

[0052] Among them, λ 1 ,λ 2 , are the weight coefficients of the two different loss functions. L MSE is the MSE loss, L MSE Defined as:

[0053]

[0054] Where H and W represent the image height and width respectively; α i,j and They represent the predicted mask and the prior mask respectively. CL is the composite loss, L CL Defined as:

[0055]

[0056] in, represents the i-th color image of the input, Represents the mask α predicted using the i-th color image n Synthesized color image, σ is a constant.

[0057] Another embodiment of the present application provides a cutout method. The cutout method can be applied to an electronic device, and a cutout network is deployed in advance in the electronic device. The cutout network can be trained using the cutout network training method of the above embodiment.

[0058] like Figure 6As shown, the cutout method may include steps S610 to S630. It should be noted that the cutout method embodiment is the same or similar to the above embodiment, and will not be described here in detail. Please refer to the above.

[0059] S610, acquiring a color image and a depth image that contain a foreground and are aligned.

[0060] In some embodiments, a color image and a depth image including a foreground may be collected by a collection device, and each pixel of the color image and the depth image may be further aligned one by one. It should be noted that the collection device may include a depth camera and a color camera, and the depth camera may be a structured light camera, a TOF camera, a binocular camera, etc., which is not limited here.

[0061] S620, based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and processing the soft segmentation image to obtain a tri-image corresponding to the color image.

[0062] In some embodiments, the depth image is segmented using a distance threshold to obtain a soft segmentation image, and then the soft segmentation image is eroded and expanded to generate a tripartite map.

[0063] S630, input the color image and its corresponding tri-image into a cutout network to obtain a mask containing only the foreground.

[0064] It should be understood that the process of using the cutout network to cut out images can refer to the relevant description of the aforementioned cutout network training process, which will not be repeated here.

[0065] The embodiment of the present application generates a trisection map through data enhancement based on the collected real data and synthetic data, which reduces manual annotation, reduces manual success, improves efficiency, and ensures model accuracy. In addition, a lightweight real-time background cutout network is proposed. The network is designed with a shallow structure, and two network modules are proposed. The FFM module can achieve better high-level feature fusion, and the CCM module is more lightweight than the corresponding traditional residual module, which is conducive to the fusion process of contextual information. Both modules have improved the accuracy to a certain extent. In order to achieve better boundary prediction and foreground prediction, a hybrid loss function is introduced, which combines the advantages of the two losses.

[0066] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0067] An embodiment of the present application also provides a cutout network training device. For details not described in the cutout network training device, please refer to the description of the cutout network training method embodiment above.

[0068] Figure 7 1 is a schematic block diagram of a cutout network training device provided in an embodiment of the present application. The cutout network training device comprises: an acquisition unit 71, a preprocessing unit 72 and a training execution unit 73.

[0069] The acquisition unit 71 is used to acquire a plurality of frames of color composite images and color images and depth images containing foreground and aligned.

[0070] The preprocessing unit 72 is used to obtain the mask in the color composite image, and simulate the depth characteristics of the mask to obtain a tri-map corresponding to the color composite image; based on the alignment relationship between the color image and the depth image, the depth image is used to obtain the soft segmentation image and the depth characteristics of the soft segmentation image are simulated to obtain the tri-map corresponding to the color image.

[0071] The training execution unit 73 is used to iteratively optimize the initial cutout network using the color image and its corresponding tri-plot, the color composite image and its corresponding tri-plot, to obtain a target cutout network.

[0072] In some embodiments, the preprocessing unit 72 is specifically used to randomly punch holes in the soft segmentation image to obtain a first image, perform erosion and dilation operations on the first image and generate a trigraph corresponding to the color image; or, perform erosion and dilation operations on the soft segmentation image and generate a first intermediate trigraph, randomly punch holes in the first intermediate trigraph to obtain a trigraph corresponding to the color image; and, randomly punch holes in the mask to obtain a second image, perform erosion and dilation operations on the second image and generate a trigraph corresponding to the color composite image; or, perform erosion and dilation operations on the mask and generate a second intermediate trigraph, randomly punch holes in the second intermediate trigraph to obtain a trigraph corresponding to the color composite image.

[0073] In some embodiments, the initial cutout network includes at least one stage network or multiple stage networks connected in series. Preferably, the stage network includes an environment combination module, a trunk block and a prediction module connected in series. The environment combination module is used to output low-order features and high-order features after feature exchange, the trunk block is used to fuse low-order features and high-order features based on the attention mechanism to obtain fused features, and the prediction module is used to output a predicted mask based on the fused features.

[0074] In some embodiments, the stage network comprises 3 times downsampling.

[0075] In some embodiments, the backbone block includes a feature fusion module based on an attention mechanism.

[0076] In some embodiments, the training execution module 73 adopts a hybrid loss function, which includes a mean square error loss and a synthesis loss.

[0077] An embodiment of the present application also provides a cutout device. For details not described in the cutout device, please refer to the description of the cutout method embodiment above.

[0078] Figure 8 8 is a schematic block diagram of a cutout device provided in an embodiment of the present application. The cutout device comprises: a collection unit 81, a pre-processing unit 82, and a cutout execution unit 83.

[0079] The acquisition unit 81 is used to acquire a color image and a depth image that contain a foreground and are aligned.

[0080] The preprocessing unit 82 is used to obtain a soft segmentation image using the depth image based on the alignment relationship between the color image and the depth image, and process the soft segmentation image to obtain a tri-image corresponding to the color image.

[0081] The cutout execution unit 83 inputs the color image and its corresponding tri-image into the cutout network, and outputs a mask containing only the foreground.

[0082] An embodiment of the present application also provides an electronic device, such as Fig. 9 As shown, the electronic device may include one or more processors 100 ( Fig. 9 Only one is shown), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a program for training a cutout network and / or a program for cutting out an image. When one or more processors 100 execute the computer program 102, the various steps in the cutout network training method and / or the cutout method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the cutout network training device and / or the cutout device embodiment can be implemented, which is not limited here.

[0083] Those skilled in the art will understand that Fig. 9 The electronic device is merely an example of an electronic device and does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0084] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0085] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the electronic device. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0086] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0087] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the cutout network training method and / or the cutout method embodiment can be implemented.

[0088] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the cutout network training method and / or the cutout method embodiment.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and the computer program that can be completed by instructing the relevant hardware through a computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0095] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for training a cutout network, characterized in that: include: Acquire several frames of color composite images and color images and depth images containing foreground and aligned; Acquire a mask in the color composite image, and perform depth characteristic simulation on the mask to obtain a trisection image corresponding to the color composite image; Based on the alignment relationship between the color image and the depth image, a soft segmentation image is acquired using the depth image and a depth characteristic simulation is performed on the soft segmentation image to obtain a trisection image corresponding to the color image; The color image and its corresponding tri-plot, the color composite image and its corresponding tri-plot are input into an initial cutout network for iterative optimization to obtain a target cutout network.

2. The image cutout network training method according to claim 1, characterized in that: The step of simulating the depth characteristic of the mask to obtain a trisection image corresponding to the color composite image includes: The mask is randomly punched with holes to obtain a second image, the second image is eroded and expanded to generate a trisection diagram corresponding to the color composite image; or, the mask is eroded and expanded to generate a second intermediate trisection diagram, the second intermediate trisection diagram is randomly punched with holes to obtain the trisection diagram corresponding to the color composite image.

3. The image cutout network training method according to claim 1, characterized in that: The step of obtaining a soft segmentation image based on the alignment relationship between the color image and the depth image using the depth image and simulating the depth characteristics of the soft segmentation image to obtain a trisection image corresponding to the color image includes: Using the depth image to obtain a soft segmentation image, randomly punching holes in the soft segmentation image to obtain a first image, performing erosion and dilation operations on the first image to generate a trisection map corresponding to the color image; or performing erosion and dilation operations on the soft segmentation image to generate a first intermediate trisection map, and randomly punching holes in the first intermediate trisection map to obtain a trisection map corresponding to the depth image; Based on the alignment relationship between the color image and the depth image, the tri-map corresponding to the depth image is converted into the tri-map of the color image.

4. The method for training a cutout network according to any one of claims 1 to 3, characterized in that: The initial cutout network is trained using a hybrid loss function for supervision, where the hybrid loss function includes a mean square error loss and a synthesis loss.

5. The method for training a cutout network according to any one of claims 1 to 3, characterized in that: The initial cutout network includes at least one stage network or multiple stage networks connected in series; the stage network includes an environment combination module, a trunk block and a prediction module connected in series, the environment combination module is used to output low-order features and high-order features after feature exchange, the trunk block is used to fuse the low-order features and the high-order features based on an attention mechanism to obtain a fused feature, and the prediction module is used to output a predicted mask according to the fused feature.

6. A method for cutting out a picture, characterized in that: include: Obtain a color image and a depth image that contain a foreground and are aligned; Based on the alignment relationship between the color image and the depth image, using the depth image to obtain a soft segmentation image and processing the soft segmentation image to obtain a trisection image corresponding to the color image; The color image and its corresponding trisection image are input into the target cutout network obtained by the cutout network training method according to any one of claims 1 to 5 to obtain a mask containing only the foreground.

7. A cutout network training device, characterized in that: include: An acquisition unit, used for acquiring a plurality of frames of color composite images and color images and depth images containing foreground and aligned; A preprocessing unit, configured to obtain a mask in the color composite image, and perform depth characteristic simulation on the mask to obtain a trisection map corresponding to the color composite image; based on an alignment relationship between the color image and the depth image, obtain a soft segmentation image using the depth image and perform depth characteristic simulation on the soft segmentation image to obtain a trisection map corresponding to the color image; The training execution unit is used to input the color image and its corresponding three-part image, the color composite image and its corresponding three-part image into the initial cutout network for iterative optimization to obtain the target cutout network.

8. A cutout device, characterized in that: include: An acquisition unit, used for acquiring a color image and a depth image including a foreground and aligned; A preprocessing unit, configured to obtain a soft segmentation image using the depth image based on an alignment relationship between the color image and the depth image, and process the soft segmentation image to obtain a trisection image corresponding to the color image; The cutout execution unit inputs the color image and its corresponding trisection image into the target cutout network obtained by the cutout network training device as claimed in claim 7 to obtain a mask containing only the foreground.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the cutout network training method as described in any one of claims 1 to 5, or implements the cutout method as described in claim 6.

10. The electronic device according to claim 9, characterized in that: It includes an acquisition device for acquiring color images and depth images for the processor to implement the cutout network training method as described in any one of claims 1 to 5, or to implement the cutout method as described in claim 6 when executing the computer program.

11. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the cutout network training method as described in any one of claims 1 to 5, or implements the cutout method as described in claim 6.

Images