Model processing method, device, storage medium and electronic device

By determining the sample training image and cutout error map in the initial object cutout model and performing model correction processing, the robustness and accuracy of object cutout in the traditional method in complex background and light source changes is solved, and the problem of the traditional method is achieved in High accuracy object cutting in complex environments.

CN115082724BActive Publication Date: 2024-11-12GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210325143.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-12
Estimated Expiration
2042-03-30

AI Technical Summary

Technical Problem

The traditional portrait cutout method based on edge and manual features is difficult to adapt to in complex backgrounds and light source changes, resulting in poor robustness and inaccurate object cutouts.

Method used

The sample cutout image and cutout error diagram corresponding to the sample training image are determined through the initial object cutout model. The initial model is modified based on these images to improve the robustness and accuracy of the model.

Benefits of technology

It realizes high accuracy and robust object cutouts in complex lighting environments, improving the accuracy of image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115082724B_ABST
    Figure CN115082724B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a model processing method, device, storage medium and electronic device, wherein the method comprises: determining a sample cutout image and a cutout error map corresponding to a sample training image through an initial object cutout model, and performing model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model. The embodiment of the present application can improve the accuracy of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model processing method, device, storage medium and electronic device. Background Art

[0002] Object recognition is an important research direction in the field of computer vision based on image segmentation technology. Image object recognition, such as video object recognition and picture object recognition, is an important part of image semantic understanding. Object recognition based on image segmentation technology refers to the process of identifying the corresponding object from the image. Object recognition can be applied to scenes such as beautifying objects in images, replacing backgrounds, and blurring backgrounds. In these scenes, the object cutout function can be used based on the object recognition process. Summary of the invention

[0003] The present application provides a model processing method, device, storage medium and electronic device, and the technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a model processing method, the method comprising:

[0005] Determine a sample matting image and a matting error map corresponding to the sample training image through an initial object matting model;

[0006] The initial object cutout model is subjected to model correction processing based on the sample cutout image and the cutout error map to obtain an object cutout model.

[0007] In a second aspect, an embodiment of the present application provides a model processing device, the device comprising:

[0008] A model training module, used to determine a sample matting image and a matting error map corresponding to a sample training image through an initial object matting model;

[0009] A model correction module, configured to perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model;

[0010] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0011] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0012] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0013] In one or more embodiments of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a flow chart of a model processing method provided in an embodiment of the present application;

[0016] Figure 2 It is a flow chart of a model processing method provided in an embodiment of the present application;

[0017] Figure 3 It is a model schematic diagram involving an initial object cutout model involved in the model processing method provided in an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of a semantic segmentation module involved in the model processing method provided in an embodiment of the present application;

[0019] Figure 5 It is a scene schematic diagram of an edge segmentation module involved in the model processing method provided in an embodiment of the present application;

[0020] Figure 6 It is a scene schematic diagram of an initial object cutout model involved in the model processing method provided in an embodiment of the present application;

[0021] Figure 7 It is a flowchart of another model processing method provided in an embodiment of the present application;

[0022] Figure 8 is a structural schematic diagram of a model processing device provided in an embodiment of the present application;

[0023] Fig. 9It is a structural diagram of a model training module provided in an embodiment of the present application;

[0024] Fig.10 It is a structural schematic diagram of a model correction module provided in an embodiment of the present application;

[0025] Fig.11 is a structural schematic diagram of another model processing device provided in an embodiment of the present application;

[0026] Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] Fig.13 It is a schematic diagram of the structure of the operating system and user space provided in the embodiment of the present application;

[0028] Fig.14 yes Fig.13 The architecture diagram of the Android operating system;

[0029] Fig.15 yes Fig.13 Architecture diagram of the IOS operating system. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0032] In related technologies, the core objective of the object cutout task is to generate a mask image α that only contains the object part (such as the object is a portrait). For pixel i in the image Imerge, there is the following expression:

[0033]

[0034] Among them, F represents the foreground image and B represents the background image. In life scenes, the background is complex and changeable. The changes in light sources and the movement of portraits will seriously affect the calculation of mask images. Traditional portrait cutout methods based on edges and manual features are difficult to adapt to the current object cutout tasks. Methods based on deep neural networks have gradually become more robust methods. However, although low-level semantic features such as color and texture can represent the approximate area of ​​the object, it is difficult to distinguish the edges of fine and complex objects such as hair. Based on this, object cutouts have problems such as poor robustness and inaccurate object cutouts.

[0035] The present application is described in detail below with reference to specific embodiments.

[0036] In one embodiment, Figure 1 As shown, a model processing method is proposed, which can be implemented by a computer program and can be run on a model processing device based on the von Neumann system. The computer program can be integrated in an application or run as an independent tool application. The model processing device can be an electronic device, including but not limited to: a personal computer, a tablet computer, a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem. In different networks, terminal devices can be called different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, 5G network or electronic device in future evolution network, etc.

[0037] Specifically, the model processing method includes:

[0038] S101: Determine a sample matting image and a matting error map corresponding to a sample training image through an initial object matting model;

[0039] The core goal of the object cutout task is to generate a mask image α that only contains the object part (such as the object is a portrait). For pixel i in the image Imerge, there is the following expression:

[0040]

[0041] Among them, F represents the foreground image and B represents the background image. In real life scenes, the background is complex and changeable. The change of light source and the movement of the portrait will seriously affect the calculation of the mask map. The traditional portrait cutout method based on edge and manual features is difficult to adapt to the current task. The method based on deep neural network has gradually become a more robust method. However, although low-level semantic features such as color and texture can represent the general area of ​​the portrait, it is difficult to distinguish the fine and complex edges of the portrait such as hair.

[0042] The initial object cutout model may be an initial model created based on a neural network. By creating the initial object cutout model and training the initial object cutout model, a trained object cutout model may be obtained. It is understandable that the trained object cutout model may realize object cutout of specific objects in the image to be identified, for example, semantic segmentation of corresponding categories of objects such as roads, walls, cars, people, animals, etc. in the image to be identified in the video may be performed to cut out the objects.

[0043] The initial object cutout model can be implemented by fitting one or more models such as a Convolutional Neural Network (CNN) model, a Deep Neural Network (DNN) model, a Recurrent Neural Networks (RNN) model, an embedding model, a Gradient Boosting Decision Tree (GBDT) model, and a Logistic Regression (LR) model.

[0044] It is understandable that a large number of sample training images can be acquired in advance as a model image training set, and the model image training set can also come from a customized model image collection and a publicly available data set.

[0045] In a feasible implementation, in life scenes, considering the complexity and variability of the background, the changes in the light source and the movement of the portrait will seriously affect the calculation of the mask image (that is, the cutout image). Based on this, in order to enhance the robustness of the model and improve the accuracy of the model in complex lighting environments, data enhancement processing can be performed on the data source after acquiring the training image. During the training process, in order to simulate the environmental influence of different lighting, random lighting enhancement is performed with a corresponding probability when the data is read, that is, the light source is simulated at a random position to generate an exposure map, which can be understood as generating multiple similar training images with different light intensities from one training image.

[0046] In a specific implementation, a training image for the initial object cutout model and a standard cutout image corresponding to the training image can be obtained; the training image and the standard cutout image corresponding to the training image can be obtained based on a custom model image set and a publicly available data set. Then, the training image is subjected to light source data enhancement processing based on the standard cutout image to obtain at least one sample training image for the training image, that is, a training image generates multiple similar training images with different light intensities, and these images are all used as sample training images.

[0047] When reading data, the design will filter with probability p, and the selected image will be superimposed with the original image (training image) and the mask image (the standard cutout image corresponding to the training image) (which can be understood as the fitting of pixel values, usually within 255).

[0048] Indicatively, first randomly select a center point between 1 / 4 and 3 / 4 of the length and width of the image; then calculate the pixel value of the pixel point using the full image exposure method based on the light attenuation coefficient:

[0049] The pixel value of a pixel = (the original pixel value of the training image + random initial illumination) * brightness attenuation coefficient,

[0050] In this way, in each round of data enhancement, the training image can contain an exposure block, and the exposure block of the training image is fitted with the mask image (the standard cutout image corresponding to the training image); it can be understood that the formula for full-image exposure can calculate multiple full-image illumination enhancement images.

[0051] Among them, the brightness attenuation coefficient can be simplified to model as follows:

[0052]

[0053] Among them, C is a constant, α and β are random coefficients; d is the distance from pixel i to the center point;

[0054] It can be understood that for the constant C, the exponential term and linear term coefficients of random α and β, the relationship between the brightness attenuation coefficient t of pixel i on the mask and the distance d to the light source can be simplified as follows. Schematically, the attenuation coefficient is adjusted by random numbers C, α, and beta to dynamically adjust the different illumination components of each image. C is 10, and α and β are random numbers between 0 and 1.

[0055] Among them, the model image training set includes sample training images and standard cutout images corresponding to the sample training images. The initial object cutout model is trained using the sample training images in the training set, that is, multiple images (such as video frame images) containing specific objects in the training set and the standard cutout images corresponding to these images. Therefore, in each round of image training, the sample cutout images and the cutout error map corresponding to the sample training images are first identified by the initial object cutout model, and then the initial object cutout model is corrected based on the sample cutout images and the cutout error map to adjust the model parameters and even the model architecture of the initial object cutout model. After completing the training of the initial object cutout model, the object cutout model can be obtained.

[0056] It is understandable that during the training phase of the initial object cutout model, it is usually necessary to determine a sample cutout image and a cutout error map predicted by the initial object cutout model. It should be noted that the initial object cutout model does not use a standard cutout image to compare with the identified sample cutout image when determining the cutout error map.

[0057] It can be understood that the predicted matting error map is the annotation of the error pixels that may be misidentified during each round of sample matting image processing by the initial object matting model. The set of these error pixels is represented in the form of a matting error map, that is, the pixel of any pixel point in the matting error map represents the matting confidence corresponding to the pixel point. It can be understood that the sample matting image and the matting error map are usually a mask map (mask), and the mask map is used to represent a grayscale image of transparency, and the white part represents the area of ​​interest and the black part represents the area of ​​no interest.

[0058] In one or more embodiments of the present specification, the pre-created initial object cutout model may include a first network part for determining a sample cutout image and a cutout error map and a second network part for adjusting model parameters based on the “sample cutout image and the cutout error map”.

[0059] Optionally, the pre-created initial object cutout model may include multiple modules, such as a module for semantic segmentation processing, a module for edge segmentation processing, a module for fusion processing, and a module for model fine-tuning processing. In the related art, the model training process of object cutouts such as portrait cutouts needs to rely on strong prior information of a ternary map (trimap) and / or a background image. In the present application, the stability of image semantic segmentation can be improved based on the semantic segmentation module and the edge segmentation module. The fusion module is used to fuse the feature information of the semantic segmentation module and the edge segmentation module. The fusion module and the fine-tuning module optimize the edge segmentation of objects such as portrait edges, eliminating the dependence on background information or even the ternary map, that is, there is no need to input the background or even the ternary map as supplementary information during prediction.

[0060] S102: performing model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model.

[0061] In a feasible implementation, the electronic device first trains the initial object cutout model based on the sample training image, so as to first identify the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to adjust the model parameters and even the model architecture of the initial object cutout model. After completing the training of the initial object cutout model, the object cutout model can be obtained.

[0062] Optionally, the pre-created initial object cutout model may include a first network part for determining a sample cutout image and a cutout error map, and a second network part for adjusting model parameters based on the “sample cutout image and the cutout error map”. The total model loss of the initial object cutout model is determined based on the total loss of the first network part and the second network part. The second network part can fine-tune and optimize the sample cutout image identified by the first network part based on the cutout error map, and can calculate the target cutout loss based on the fine-tuned optimized image, which is also the loss of the second network part, and merge the target cutout loss with the cutout loss of the first network part to obtain the total cutout loss (also referred to as the target cutout loss in some embodiments), and perform model correction processing on the initial object cutout model based on the total cutout loss to adjust the various model parameters and even the model architecture of the initial object cutout model. After completing the training of the initial object cutout model, the object cutout model can be obtained. In some implementations, during the training process, when the total cutout loss of the initial cutout network model decreases or approaches an expected value (such as 0), it is determined that the training of the initial object cutout model is completed.

[0063] It can be understood that by adding the error map prediction of the initial object cutout model to output the cutout error map predicted by the model while outputting the sample cutout image, the confidence of each pixel misclassification in the sample cutout image is predicted, and there is a better response to the erroneous part of the sample cutout image and the object edge in each round of training. Based on this, additional loss information can be provided for model error correction and self-supervised training.

[0064] In an embodiment of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing; and by performing light source data enhancement processing greater than the data source, it can adapt to complex environments and improve the accuracy of the model in complex lighting environments.

[0065] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of a model processing method proposed in this application. Specifically:

[0066] S201: Determine image hidden semantic features and a semantic segmentation map using an initial object cutout model based on the sample training image.

[0067] It can be understood that semantic segmentation is performed based on the sample training image using the initial object cutout model to obtain image hidden semantic features and a semantic segmentation map;

[0068] In some embodiments, the module for semantic segmentation processing of the initial object cutout model can be used to perform semantic segmentation processing on the sample training image to obtain hidden semantic features of the image and a semantic segmentation map.

[0069] For illustration, see Figure 3 , Figure 3 It is a schematic diagram of the model involved in the initial object cutout model. Figure 3In the embodiment of the present invention, the pre-created initial object cutout model may include a module for semantic segmentation processing (such as a semantic segmentation module), a module for edge segmentation processing (such as an edge segmentation module), a module for fusion processing (such as a fusion module), and a module for model fine-tuning processing (such as a fine-tuning module). In one or more embodiments of the present specification, the pre-created initial object cutout model may include a semantic segmentation module, an edge segmentation module, a fusion module, and a fine-tuning module. It should be noted that the names of the various modules may be set with different names based on different application environments, and are not specifically limited here. The stability of image semantic segmentation can be improved based on the semantic segmentation module and the edge segmentation module. The fusion module is used to fuse the feature information of the semantic segmentation module and the edge segmentation module. The fusion module and the fine-tuning module optimize the edge segmentation of objects such as portrait edges, eliminating the dependence on background information or even ternary maps, that is, there is no need to input background or even ternary maps as supplementary information during prediction.

[0070] In a feasible implementation, the module for semantic segmentation processing (such as a semantic segmentation module) can also be called a coarse semantic segmentation module. The coarse semantic segmentation module extracts image foreground feature information in the sample training image, and segments it to obtain an object position grayscale image of a corresponding size to feedback the approximate area of ​​the object (which can be understood as the foreground area), and the value is used as a transparency prediction value to obtain a semantic segmentation map for the semantic segmentation module; in the process of the semantic segmentation module performing semantic segmentation on the sample training image; in the multi-network layers (such as convolutional layers) corresponding to the semantic segmentation module, image features of different scales are also extracted as semantic information of different image levels to obtain hidden semantic features of the image in the processing process.

[0071] Schematically, the semantic segmentation module of the initial object cutout model may include an encoder-decoder, such as Figure 4 As shown, Figure 4 is a schematic diagram of a semantic segmentation module. Figure 4The semantic segmentation module shown includes a neural network 10 (such as a convolutional network composed of multiple convolutional layers), an encoder 20, and a channel attention part (SE) 30. The neural network and encoder in the semantic segmentation module can perform downsampling semantic segmentation on the sample training image layer by layer, such as the semantic features corresponding to the foreground area (such as the portrait foreground area) of 1 / 16 of the original image size, so as to reduce the model training cost. The channel attention mechanism (SE) in the semantic segmentation module re-empowers the semantic features, increases the weight of the semantic features related to the object position (such as the portrait position), and suppresses irrelevant semantic features. The semantic segmentation module directly uses the downsampling of the true value of the mask image for constraint, so as to obtain a semantic segmentation map containing the semantic features of the object. In addition, the encoder in the semantic segmentation module extracts image features of different scales as semantic information of different image levels through the feature pyramid (SPP), and obtains the hidden semantic features of the image in the processing process. The hidden semantic features of the image can also be understood as hidden features through the feature pyramid (SPP) (output).

[0072] For example, the encoder in the semantic segmentation module uses the feature pyramid (SPP) to gradually downsample the sample training image to 1, 1 / 2, 1 / 4, 1 / 8, and 1 / 16 (such as the original Figure 1 / 2*1 / 2 size) downsampled image results for semantic feature extraction. Different from the usual practice of only retaining the last layer of features, the semantic features extracted corresponding to the 1, 1 / 2, 1 / 4, and 1 / 8 sizes are used for feature fitting as the intermediate features extracted by the semantic segmentation module, namely the aforementioned image hidden semantic features. The image hidden semantic features are obtained by model calculation inside the encoder of the semantic segmentation module.

[0073] It can be understood that the input of the semantic segmentation module is a sample training image, the intermediate output is the hidden semantic features of the image, and the final output is a semantic segmentation map. Each pixel in the semantic segmentation map is the confidence level of the prediction belonging to the object foreground (such as the portrait foreground).

[0074] S202: Determine a high-frequency image feature map using the initial object cutout model based on the sample training image, the image hidden semantic features, and the semantic segmentation map.

[0075] It can be understood that the high-frequency image feature map can be determined by performing edge segmentation processing based on the sample training image, the image hidden semantic features and the semantic segmentation map through the initial cutout model, such as by using an edge segmentation module for edge segmentation processing based on the initial cutout model;

[0076] Specifically, during the model training process, the area obtained by the semantic segmentation map output by the semantic segmentation module is used as a mask to perform fine-grained segmentation on the edge of the object in the image, extract high-frequency features, and output a high-frequency image feature map. The high-frequency features in the high-frequency image feature map can be understood as high-resolution image features, such as edge feature points, edge feature patterns, etc., which are not limited here.

[0077] It can be understood that the input of the semantic segmentation map is the sample training image, the intermediate output image hidden semantic features of the semantic segmentation module, and the final output semantic segmentation map of the semantic segmentation module, and the output of the semantic segmentation map is a high-frequency image feature map. In some embodiments, the size of the high-frequency image feature map is usually 1 / 16 * 1 / 16 of the original image.

[0078] Schematically, the edge segmentation processing of the initial object cutout model, such as an edge segmentation module, may include a decoder based on a convolutional layer, such as Figure 5 As shown, Figure 5 is a schematic diagram of a scene involving an edge segmentation module. Figure 5 In the embodiment, the edge segmentation module can use a model based on a simplified convolutional neural network (CNN). For example, the edge segmentation module can be a neural network composed of multiple convolutional layers. The coarse segmentation map sc predicted by semantic segmentation and the image hidden semantic features of SPP (spatial feature pyramid) are used to model the easily confused areas in the trimap (ternary map). During the processing, the edge segmentation module first connects and fuses the sample training image and the intermediate output of the semantic segmentation - the image hidden semantic features to obtain the first feature and then processes it; then the first feature is cascaded with the final output of the semantic segmentation - the semantic segmentation map to obtain the second feature (not shown in the figure) and then processed, and up-sampled in the decoder 40 based on the convolutional layer structure contained in the edge segmentation module, and the transition area around the foreground object in the original image (such as around the foreground portrait) is processed, and the size of the original image (sample training image) is used for calculation to obtain the final high-frequency image feature map.

[0079] S203: Determine a sample cutout image and a cutout error map using the initial object cutout model based on the semantic segmentation map and the high-frequency image feature map.

[0080] It is understandable that the initial object cutout model can be used for fusion processing based on the semantic segmentation map and the high-frequency image feature map, and the sample cutout image and the cutout error map can be determined during the fusion processing, such as using a fusion module for fusion processing.

[0081] It can be understood that the module for fusion processing (such as the fusion module) can also use a CNN model based on multiple convolutional layers to fuse features, and obtain a sample cutout image by fusing the result of the semantic segmentation module - the semantic segmentation map, the result of the edge segmentation - the high-frequency image feature map. The sample cutout image is usually a mask image. Schematically, as Figure 6 As shown, Figure 6 It is a scene diagram of an initial object cutout model. Figure 6 The initial object cutout model shown includes a fusion module, which fuses the result of the semantic segmentation module - the semantic segmentation map, the result of the edge segmentation - the high-frequency image feature map to obtain a sample cutout image, and on the other hand predicts a cutout error map for the sample cutout image.

[0082] It can be understood that the fusion module will also directly predict the error map for the sample matting image as the matting error map as an additional input to the fine-tuning module.

[0083] Among them, the cutout error map reflects where the model output may be wrong during the prediction process. The object cutout task can be roughly understood as a binary classification task of object foreground (such as portrait foreground) segmentation, and the transparency of the sample cutout image can be understood as the probability of belonging to the object foreground (such as portrait foreground). The cutout error map can be understood as the confidence of the model predicting the segmentation result of each pixel for the input sample cutout image. By evaluating the confidence of the prediction result, the additional information brought by the background can be compensated to eliminate the dependence of the object cutout model on the background image during the training process. In other words, only the sample training image is needed, and there is no need to introduce the ternary map corresponding to the sample training image or even the background image when inputting the model training.

[0084] As can be understood, the fusion module combines semantic features and edge detail features. Usually, the semantic segmentation map can be upsampled to be consistent with the size of the high-frequency image feature map. Then the fusion module connects the semantic features of the semantic segmentation map and the edge detail features of the high-frequency image feature map to predict the final sample cutout image.

[0085] S204: Determine a reference cutout loss for the initial object cutout model.

[0086] In one or more embodiments, the initial object cutout model includes a semantic segmentation module, an edge segmentation module, a fusion module and a fine-tuning module. The reference cutout loss can be understood as the total loss of the semantic segmentation module, the edge segmentation module and the fusion module.

[0087] In a feasible implementation, the semantic segmentation loss, edge segmentation loss and image fusion loss for the initial object cutout model may be obtained; schematically, the semantic segmentation loss for the semantic segmentation module, the edge segmentation loss for the edge segmentation module and the image fusion loss for the fusion module may be obtained;

[0088] It can be understood that the model loss in the process of the semantic segmentation module determining the semantic segmentation map is used as the semantic segmentation loss.

[0089] Schematically, when calculating the semantic segmentation loss, first obtain the standard cutout image corresponding to the sample training image, and use the first loss calculation formula based on the semantic segmentation image and the standard cutout image to obtain the semantic segmentation loss;

[0090] Indicatively, the first loss calculation formula satisfies the following formula:

[0091] L c =1 / 2||s c -D(s gt )||2

[0092] Among them, the L c is the semantic segmentation loss, the S c is the semantic segmentation map, the S gt is a standard cutout image, and D() is a downsampling function;

[0093] Optionally, in this application, downsampling can be performed directly without Gaussian blur processing to save model computing resources, and D(x) is used to represent the downsampling of the preset scale. For example, D(x) can be the original Figure 1 / 16 downsampling.

[0094] It can be understood that the model loss in the process of the edge segmentation module determining the high-frequency image feature map is used as the edge segmentation loss.

[0095] Schematically, when calculating the edge segmentation loss, the ternary image corresponding to the sample training image and the target mask image corresponding to the target color area in the ternary image are first determined.

[0096] The target mask image corresponding to the target color area in the trimap can be understood as the mask image of the easily confused area in the trimap. The trimap uses black to represent the background, white to represent the foreground, and gray to represent the easily confused area (the area to be solved for the cutout). That is to say, the target mask image can be obtained based on the mask image of the gray area in the trimap.

[0097] The ternary graph may be obtained by expert annotation or by a corresponding image segmentation algorithm in related technologies, such as automatically generating a ternary graph by pixel expansion and strongly connected component algorithms.

[0098] A second loss calculation formula is used based on the target mask image, the high-frequency image feature map, and the standard cutout image to obtain an edge segmentation loss;

[0099] The second loss calculation formula satisfies the following formula: L d =m d ||s d -s gt ||1

[0100] Among them, among them, the L d is the edge segmentation loss, the m d is the target mask image, the S d is the high-frequency image feature map, the S gt is a standard cutout image;

[0101] For example, md can be expressed as a mask of the confusing area in the trimap (ternary map), S d represents the predicted high-frequency image feature map, for example, the high-frequency image feature map may be a portrait edge distribution; the S gt is a standard cutout image, which can usually be understood as the true value of the mask image. In this application, it is not necessary to input the ternary map corresponding to the original image in the model training stage. Only the easily confused part of the trimap (ternary map) is considered in the loss function processing link. Usually S d Contains additional texture information that can be passed to the fusion module for further processing.

[0102] It can be understood that the model loss in the process of the fusion module determining the sample cutout image is used as the image fusion loss.

[0103] Indicatively, a reference cutout error map for a sample cutout image may be first obtained, and a third loss calculation formula may be used based on the sample cutout image, the standard cutout image, the reference cutout error map, and the cutout error map to obtain an image fusion loss;

[0104] Specifically, the electronic device obtains a reference cutout error map for the sample cutout image. The reference cutout error map can be understood as a standard cutout error map for the predicted cutout error map, which can provide a reference for the cutout error map predicted by each round of model to achieve model supervision. Generally speaking, the cutout error map predicted by the model in each round can tend to the reference cutout error map to achieve better cutout segmentation effect.

[0105] Specifically, it may be: by acquiring a standard cutout image corresponding to the sample training image, based on the standard cutout image and the sample cutout image, a reference cutout error map is determined by the initial object cutout model.

[0106] Indicatively, the fusion module of the initial object cutout model can be calculated using the absolute value error formula: abs(a,b), where the standard cutout image is taken as a, the sample cutout image is taken as b, and the absolute value of the calculation result is taken to obtain a reference cutout error map.

[0107] In the present application, the predicted matting error map is the annotation of the error pixels that may be misidentified during each round of sample matting image processing by the initial object matting model. The set of these error pixels is represented in the form of a matting error map, that is, the pixel of any pixel point in the matting error map represents the matting confidence corresponding to the pixel point. It can be understood that the sample matting image and the matting error map are usually a mask map (mask), which is used to represent a grayscale image of transparency, and the white part represents the area of ​​interest and the black part represents the area of ​​no interest.

[0108] The third loss calculation formula satisfies the following formula: L a =||s p -s gt ||1+||s pe -s gte ||1

[0109] Among them, the L a is the image fusion loss, the S p is the sample cutout image, the S gt is a standard cutout image, the S pe is the reference clipping error map, the S gte is the error map of the clipping image. It can also be understood as the true value of the error map of the predicted clipping image, that is, the true value of the error map.

[0110] In a feasible implementation manner, based on the semantic segmentation loss, the edge segmentation loss and the image fusion loss, a reference cutout loss for the initial object cutout model is obtained.

[0111] Optionally, loss weights may be set for the semantic segmentation loss, the edge segmentation loss, and the image fusion loss, respectively, and weighted fusion may be performed to obtain a reference cutout loss for the initial object cutout model. Schematically, the reference cutout loss may be obtained based on the following calculation formula, as follows:

[0112] Lx=λ c L c +λ d L d +λa L a

[0113] Where Lx represents the reference cutout loss, λ c is the loss weight of semantic segmentation loss (which can also be understood as a weight factor), λ d is the loss weight of edge segmentation loss, λ a is the loss weight of image fusion loss;

[0114] S205: Obtaining a reference cutout loss for the initial object cutout model;

[0115] S206: performing error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image.

[0116] It can be understood that the initial object cutout model can be used to perform error correction processing based on the sample cutout image and the cutout error map to obtain a cutout corrected image, such as by using a fine-tuning module for error correction processing;

[0117] Schematically, the fine-tuning module performs cropping and convolution-based feature correction according to the uncertainty map. Cropping: Take the top k pixels with confidence values ​​in the predicted cutout error map, where the size of the predicted cutout error map can be 1 / 16 of the original image, and cut a block of a preset size such as 8x8 pixels at the corresponding position of the original image with it as the center. Convolution correction can be understood as using a convolution-based network model structure to learn the high-resolution part of the image. The whole process can provide additional weight terms for the loss function to correct the misclassification model of the portrait edge. This method can improve the background misclassification and model false positives caused by excessive smoothing of the model's edges.

[0118] Schematically, based on the pixel value of each pixel in the cutout error map, the fine-tuning module can be used to determine at least one reference pixel, usually the pixel value is used to represent the cutout confidence corresponding to the pixel, and then the fine-tuning module is used to obtain the pixel blocks corresponding to each reference pixel from the cutout error map, and the sample cutout image is fine-tuned based on the pixel blocks to obtain a cutout corrected image.

[0119] Optionally, before determining the reference pixel point, the cutout error map may be connected on the basis of the sample cutout image to obtain a connected sample cutout image.

[0120] It can be understood that the fine-tuning module performs overlapping cutting according to the image edge of the sample cutout image, and the top k pixels in the predicted cutout error map (that is, the first k pixels are determined as reference pixels according to the pixel value of each pixel in the cutout error map), and then cuts the preset size (such as 8x8 specifications) based on these reference pixels; in some embodiments, based on the fine-tuning module, these switches are processed through at least one layer of convolution and effective filling processing, batch normalization processing and linear processing, the dimension of the cut is reduced and then up-sampled to restore to the preset size to obtain the processed cut; each cut is superimposed on the sample cutout image to form a new image (usually an irregular mask image mask) for semantic segmentation, and the semantic segmentation can be performed based on the convolution feature correction core to obtain a regular mask image mask as the correction result, that is, the correction result composed of the focus block (the aforementioned cut) is output, and the final correction result can be regarded as the cutout correction image obtained after the error fine-tuning processing. Then, the loss Lft generated by the re-segmentation is predicted according to the cutout correction image to adjust the optimization process.

[0121] Schematically, schematically, the fine-tuning module of the initial object cutout model may include a cropper and a refiner based on the composition of a convolutional layer. The fine-tuning module may use a model based on a simplified convolutional neural network (CNN) as a cropper (crop), and may use a model based on a simplified convolutional neural network (CNN) as a refiner (refiner).

[0122] S207: Determine a target cutout loss for the initial object cutout model based on the cutout correction image and the sample cutout image;

[0123] It can be understood that the sample cutout image is processed by an operator operation to obtain a first operation result; the standard cutout image is processed by an operator operation to obtain a second operation result;

[0124] The operator transport processing may be performed by using a sobel operator operation function, which is specifically determined based on an actual application scenario.

[0125] It can be understood that a fourth loss calculation formula may be used based on the matting correction image, the standard matting image, the first operation result, and the second operation result to determine a target matting loss for the initial object matting model;

[0126] Optionally, the fourth loss calculation formula satisfies the following formula:

[0127] L ft =||s ft -s gt ||2+λ edge ||sobel(s p)-sobel(s gt )||2

[0128] Among them, the L ft is the target cutout loss, the S ft To correct the image by cutting out, the S gt is a standard cutout image, sobel() is a sobel operator function, and S p is the sample cutout image, the λ edge is the weight factor.

[0129] For example, λedge represents the weight of the sobel loss and can be equal to 1 during training.

[0130] S208: Performing model correction processing on the initial object cutout model based on the target cutout loss and the reference cutout loss to obtain an object cutout model.

[0131] It can be understood that the target cutout loss can be determined by performing loss fusion based on the target cutout loss and the reference cutout loss;

[0132] Schematically, the reference cutout loss includes semantic segmentation loss, the edge segmentation loss, and the image fusion loss. That is, weighted fusion processing can be performed based on the semantic segmentation loss, the edge segmentation loss, the image fusion loss, and the reference cutout loss to obtain the target cutout loss.

[0133] Optionally, the target cutout loss can be calculated based on the following formula:

[0134] L=λ c L c +λ d L d +λ a L a +λ ft L ft

[0135] Where Lx represents the reference cutout loss, λ c is the loss weight of semantic segmentation loss (which can also be understood as a weight factor), λ d is the loss weight of edge segmentation loss, λ a is the loss weight of image fusion loss; ft is the target image loss L ft The loss weight of

[0136] It is understandable that after obtaining the target cutout loss, the initial object cutout model can be subjected to model correction processing based on the target cutout loss by using a back propagation training method to obtain an object cutout model. In a back propagation process corresponding to a back propagation training method: the model parameters and even the model architecture of the initial object cutout model are continuously adjusted, and after completing the training of the initial object cutout model, the object cutout model can be obtained. In some embodiments, during the training process, when the total cutout loss of the initial cutout network model decreases or approaches an expected value (such as 0), it is determined that the training of the initial object cutout model is completed.

[0137] It can be understood that by adding the error map prediction of the initial object cutout model to output the cutout error map predicted by the model while outputting the sample cutout image, the confidence of each pixel misclassification in the sample cutout image is predicted, and there is a better response to the erroneous part of the sample cutout image and the object edge in each round of training. Based on this, additional loss information can be provided for model error correction and self-supervised training.

[0138] In an embodiment of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing; and by performing light source data enhancement processing greater than the data source, it can adapt to complex environments and improve the accuracy of the model in complex lighting environments.

[0139] See also Figure 7 , Figure 7 This is a flow chart of another embodiment of a model processing method proposed in this application. Specifically:

[0140] S301: Determine a sample matting image and a matting error map corresponding to a sample training image through an initial object matting model.

[0141] S302: performing model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model.

[0142] S303: Acquire a target image.

[0143] It is understandable that the target image can be any picture, and the picture can be an image containing the corresponding object; in some embodiments, the target image can be a picture containing a portrait, or a picture not containing a portrait. For example, a pre-taken portrait picture, at least one frame of portrait image obtained from a video, etc. When the target image is a portrait image to be processed, the portrait contained in the portrait image to be processed can be one or more. The pre-trained object cutout model has multiple modules, including but not limited to a semantic segmentation module, an edge segmentation module, and a fusion module, and can also include an output semantic segmentation module, an edge segmentation module, a fusion module, and a fine-tuning module of the object cutout model. The object cutout model performs object cutout on the target image.

[0144] S304: Input the target image into the object cutout model, and output a target cutout image corresponding to the target image.

[0145] In a feasible implementation manner, the target image is input into the object cutout model, and the target cutout image is output through the output semantic segmentation module, edge segmentation module and fusion module of the object cutout model.

[0146] Schematically, the target image is input into the object cutout model, and the hidden semantic features of the target image and the target semantic segmentation map are determined by the semantic segmentation module; then the target image, the hidden semantic features of the target image and the target semantic segmentation map are subjected to edge semantic segmentation by the edge segmentation module to obtain a target high-frequency image feature map; finally, the target semantic segmentation map and the target high-frequency image feature map are semantically fused by the fusion module to determine a target cutout image for the target image, and the target cutout image is used as the output of the object cutout model.

[0147] In a feasible implementation manner, the target image may be input into the object cutout model, and the target cutout image may be output through the output semantic segmentation module, edge segmentation module, fusion module and fine-tuning module of the object cutout model.

[0148] Schematically, the target image is input into the object cutout model, and the semantic segmentation module determines the hidden semantic features of the target image and the target semantic segmentation map; then the edge segmentation module performs edge semantic segmentation on the target image, the hidden semantic features of the target image and the target semantic segmentation map to obtain a target high-frequency image feature map; then the fusion module performs semantic fusion on the target semantic segmentation map and the target high-frequency image feature map to determine an initial target cutout image and a target cutout error map for the target image; then the fine-tuning module performs error correction processing on the initial target cutout image and the target cutout error map to obtain the target cutout image after error correction.

[0149] In an embodiment of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing; and using a priori and rule-based method to correct parts of the cutout error map, the quality of object cutout can be further improved under the premise of ensuring real-time performance and without providing the original image background.

[0150] The following will be combined Figure 8 , the model processing device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 8 The model processing device shown is used to execute the present application Figure 1 to Figure 7 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 1 to Figure 7 The embodiment shown.

[0151] See also Figure 8 , which shows a schematic diagram of the structure of the model processing device of an embodiment of the present application. The model processing device 1 can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the model processing device 1 includes a model training module 11 and a model correction module 12, which are specifically used to:

[0152] A model training module 11, used to determine a sample matting image and a matting error map corresponding to a sample training image through an initial object matting model;

[0153] The model correction module 12 is used to perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model.

[0154] Optionally, the initial object cutout model includes a semantic segmentation module, an edge segmentation module and a fusion module, such as Fig. 9 As shown, the model training module 11 includes:

[0155] A semantic segmentation unit 111 is used to determine image hidden semantic features and a semantic segmentation map based on the sample training image using an initial object cutout model;

[0156] An edge segmentation unit 112 is used to determine a high-frequency image feature map using the initial object cutout model based on the sample training image, the image hidden semantic features and the semantic segmentation map;

[0157] The semantic fusion unit 113 is used to determine a sample cutout image and a cutout error map by using the initial object cutout model based on the semantic segmentation map and the high-frequency image feature map.

[0158] Optional, such as Fig.10 As shown, the model correction module 12 includes:

[0159] The error correction unit 121 is used to obtain a reference cutout loss for the initial object cutout model; perform error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image;

[0160] A loss determining unit 122, configured to determine a target cutout loss for the initial object cutout model based on the cutout-corrected image and the sample cutout image;

[0161] The model correction unit 123 is used to perform model correction processing on the initial object cutout model based on the target cutout loss and the reference cutout loss to obtain an object cutout model.

[0162] Optionally, the error correction unit 121 is further used to:

[0163] A reference matting loss for the initial object matting model is determined.

[0164] Optionally, the error correction unit 121 is further used to:

[0165] Obtaining a semantic segmentation loss for the semantic segmentation module, an edge segmentation loss for the edge segmentation module, and an image fusion loss for the fusion module;

[0166] Based on the semantic segmentation loss, the edge segmentation loss and the image fusion loss, a reference matting loss for an initial object matting model is obtained.

[0167] Optionally, the error correction unit 121 is specifically configured to:

[0168] Obtain a standard cutout image corresponding to the sample training image, and obtain a semantic segmentation loss by using a first loss calculation formula based on the semantic segmentation map and the standard cutout image;

[0169] Determine a ternary image corresponding to the sample training image and a target mask image corresponding to a target color region in the ternary image, and obtain an edge segmentation loss by using a second loss calculation formula based on the target mask image, the high-frequency image feature map, and the standard matting image;

[0170] Obtain a reference clipping error map for the sample clipping image, and obtain an image fusion loss by using a third loss calculation formula based on the sample clipping image, the standard clipping image, the reference clipping error map, and the clipping error map;

[0171] The first loss calculation formula satisfies the following formula: L c =1 / 2||s c -D(s gt )||2;

[0172] Among them, the L c is the semantic segmentation loss, the S c is the semantic segmentation map, the S gt is a standard cutout image, and D() is a downsampling function;

[0173] The second loss calculation formula satisfies the following formula: L d =m d ||s d -s gt ||1;

[0174] Among them, the L d is the edge segmentation loss, the m d is the target mask image, the S d is the high-frequency image feature map, the S gt is a standard cutout image;

[0175] The third loss calculation formula satisfies the following formula: L a =||s p -s gt ||1+||s pe -s gte ||1

[0176] Among them, the L a is the image fusion loss, the S p is the sample cutout image, the S gt is a standard cutout image, the S pe is the clipping error map, the S gte This is the reference clipping error map.

[0177] Optionally, the error correction unit 121 is further used to:

[0178] Obtaining a standard cutout image corresponding to the sample training image;

[0179] Based on the standard matting image and the sample matting image, a reference matting error map is determined by using the initial object matting model.

[0180] Optionally, the error correction unit 121 is specifically configured to:

[0181] Based on the pixel value of each pixel in the matting error map, the fine-tuning module is used to determine at least one reference pixel, wherein the pixel value is used to represent the matting confidence corresponding to the pixel;

[0182] The fine-tuning module is used to obtain pixel blocks corresponding to each of the reference pixel points from the cutout error map, and error fine-tuning processing is performed on the sample cutout image based on the pixel blocks to obtain a cutout corrected image.

[0183] Optionally, the error correction unit 121 is specifically configured to:

[0184] Performing an operator operation on the sample cutout image to obtain a first operation result; performing an operator operation on the standard cutout image to obtain a second operation result;

[0185] Determine a target cutout loss for the initial object cutout model using a fourth loss calculation formula based on the cutout correction image, the standard cutout image, the first operation result, and the second operation result;

[0186] The fourth loss calculation formula satisfies the following formula:

[0187] L ft =||s ft -s gt ||2+λ edge ||sobel(s p )-sobel(s gt )||2

[0188] Among them, the L ft is the target cutout loss, the S ft To correct the image by cutting out, the S gt is a standard cutout image, sobel() is a sobel operator function, and S p is the sample cutout image, the λ edge is the weight factor.

[0189] Optionally, the model correction unit 123 is specifically used to:

[0190] Determining a target cutout loss based on the target cutout loss and the reference cutout loss;

[0191] Based on the target cutout loss, a back propagation training method is adopted to perform model correction processing on the initial object cutout model to obtain an object cutout model.

[0192] Optionally, the model correction unit 123 is specifically used to:

[0193] The determining of the target cutout loss based on the target cutout loss and the reference cutout loss includes:

[0194] A weighted fusion process is performed based on the semantic segmentation loss, the edge segmentation loss, the image fusion loss and the reference cutout loss to obtain a target cutout loss.

[0195] Optionally, the device 1 is further used for:

[0196] Acquire a training image for an initial object cutout model and a standard cutout image corresponding to the training image;

[0197] The light source data of the training image is enhanced based on the standard cutout image to obtain at least one sample training image for the training image.

[0198] Optional, such as Fig.11 As shown, the device 1 further includes:

[0199] The object cutout module 13 is used to obtain a target image, input the target image into the object cutout model, and output a target cutout image corresponding to the target image.

[0200] It should be noted that the model processing device provided in the above embodiment only uses the division of the above functional modules as an example when executing the model processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model processing device provided in the above embodiment belongs to the same concept as the model processing method embodiment, and its implementation process is detailed in the method embodiment, which will not be repeated here.

[0201] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0202] In an embodiment of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing; and using a priori and rule-based method to correct parts of the cutout error map, the quality of object cutout can be further improved under the premise of ensuring real-time performance and without providing the original image background.

[0203] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 to Figure 7 The model processing method of the embodiment shown in the figure can be found in the specific execution process. Figure 1 to Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0204] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 to Figure 7 The model processing method of the embodiment shown in the figure can be found in the specific execution process. Figure 1 to Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0205] Please refer to Fig.12 , which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0206] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0207] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.

[0208] See also Fig.13As shown, the memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, different application scenarios in the same third-party application also have different requirements for system resources. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0209] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0210] Taking the Android operating system as an example, the programs and data stored in the memory 120 are as follows: Fig.14As shown, the memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360 and an application layer 380, wherein the Linux kernel layer 320, the system runtime library layer 340 and the application framework layer 360 belong to the operating system space, and the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, etc. The system runtime library layer 340 provides the main feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support, etc. The Android runtime library (Android runtime) is also provided in the system runtime library layer 340, which mainly provides some core libraries that allow developers to use the Java language to write Android applications. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the application layer 380. These applications can be native applications that come with the operating system, such as contact applications, text messaging applications, clock applications, camera applications, etc.; they can also be third-party applications developed by third-party developers, such as game applications, instant messaging applications, photo beautification applications, etc.

[0211] Taking the operating system as an IOS system as an example, the programs and data stored in the memory 120 are as follows: Fig.15As shown, the IOS system includes: a core operating system layer 420 (Core OS layer), a core service layer 440 (Core Services layer), a media layer 460 (Media layer), and a touchable layer 480 (Cocoa Touch Layer). The core operating system layer 420 includes the operating system kernel, drivers, and underlying program frameworks, which provide functions closer to the hardware for use by the program framework located in the core service layer 440. The core service layer 440 provides system services and / or program frameworks required by the application, such as the foundation framework, account framework, advertising framework, data storage framework, network connection framework, geographic location framework, motion framework, etc. The media layer 460 provides audio-visual interfaces for the application, such as graphics and image related interfaces, audio technology related interfaces, video technology related interfaces, and wireless playback (AirPlay) interfaces for audio and video transmission technologies. The touchable layer 480 provides various commonly used interface-related frameworks for application development, and the touchable layer 480 is responsible for the user's touch interaction operations on the electronic device. For example, local notification service, remote push service, advertising framework, game tool framework, message user interface (UI) framework, user interface UIKit framework, map framework, etc.

[0212] exist Fig.15 Among the frameworks shown, the frameworks related to most applications include but are not limited to: the basic framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The basic framework provides many basic object classes and data types, provides the most basic system services for all applications, and has nothing to do with UI. The classes provided by the UIKit framework are basic UI class libraries for creating touch-based user interfaces. iOS applications can provide UIs based on the UIKit framework, so it provides the basic architecture of applications for building user interfaces, drawing, processing and user interaction events, responding to gestures, etc.

[0213] Among them, the method and principle of implementing data communication between third-party applications and the operating system in the IOS system can be referred to the Android system, and this application will not go into details here.

[0214] Among them, the input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays, which are used to receive touch operations on or near the user using any suitable object such as a finger or a touch pen, and to display the user interface of each application. The touch screen display is usually set on the front panel of the electronic device. The touch screen display can be designed as a full screen, a curved screen or a special-shaped screen. The touch screen display can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.

[0215] In addition, those skilled in the art will appreciate that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the electronic device also includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module and other components, which will not be described in detail here.

[0216] In the embodiment of the present application, the execution subject of each step may be the electronic device described above. Optionally, the execution subject of each step is the operating system of the electronic device. The operating system may be an Android system, an IOS system, or other operating systems, which is not limited in the embodiment of the present application.

[0217] The electronic device of the embodiment of the present application may also be equipped with a display device, which may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the electronic device 101 to view displayed text, images, videos and other information. The electronic device may be a smart phone, a tablet computer, a gaming device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, an electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, electronic clothing and other devices.

[0218] exist Fig.12 In the electronic device shown, the processor 110 may be used to call an application stored in the memory 120 and specifically perform the following operations:

[0219] Determine a sample matting image and a matting error map corresponding to the sample training image through an initial object matting model;

[0220] The initial object cutout model is subjected to model correction processing based on the sample cutout image and the cutout error map to obtain an object cutout model.

[0221] In one embodiment, when the processor 110 determines the sample matting image and the matting error map corresponding to the sample training image by using the initial object matting model, the processor 110 specifically performs the following steps:

[0222] Based on the sample training image, an initial object cutout model is used to determine image hidden semantic features and a semantic segmentation map;

[0223] Determine a high-frequency image feature map using the initial object cutout model based on the sample training image, the image hidden semantic features, and the semantic segmentation map;

[0224] The initial object matting model is used to determine a sample matting image and a matting error map based on the semantic segmentation map and the high-frequency image feature map.

[0225] In one embodiment, when the processor 110 performs the model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain the object cutout model, the processor 110 specifically performs the following steps:

[0226] Obtaining a reference cutout loss for the initial object cutout model;

[0227] Performing error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image;

[0228] Determining a target matting loss for the initial object matting model based on the matting-corrected image and the sample matting image;

[0229] The initial object cutout model is subjected to model correction processing based on the target cutout loss and the reference cutout loss to obtain an object cutout model.

[0230] In one embodiment, before executing the step of obtaining the reference cutout loss for the initial object cutout model, the processor 110 further performs the following steps:

[0231] A reference matting loss for the initial object matting model is determined.

[0232] In one embodiment, when the processor 110 determines the reference cutout loss for the initial object cutout model, the processor 110 specifically performs the following steps:

[0233] Obtaining semantic segmentation loss, edge segmentation loss, and image fusion loss for the initial object cutout model;

[0234] Based on the semantic segmentation loss, the edge segmentation loss and the image fusion loss, a reference matting loss for an initial object matting model is obtained.

[0235] In one embodiment, when the processor 110 executes the step of obtaining the semantic segmentation loss, the edge segmentation loss, and the image fusion loss for the initial object cutout model, the processor 110 specifically performs the following steps:

[0236] Obtain a standard cutout image corresponding to the sample training image, and obtain a semantic segmentation loss by using a first loss calculation formula based on the semantic segmentation map and the standard cutout image;

[0237] Determine a ternary image corresponding to the sample training image and a target mask image corresponding to a target color region in the ternary image, and obtain an edge segmentation loss by using a second loss calculation formula based on the target mask image, the high-frequency image feature map, and the standard matting image;

[0238] Obtain a reference cutout error map for the sample cutout image, and obtain an image fusion loss by using a third loss calculation formula based on the sample cutout image, the standard cutout image, the reference cutout error map, and the cutout error map;

[0239] The first loss calculation formula satisfies the following formula: L c =1 / 2||s c -D(s gt )||2;

[0240] Wherein, the is the semantic segmentation loss, the Sc is the semantic segmentation map, the Sgt is the standard cutout image, and the D() is the downsampling function;

[0241] The second loss calculation formula satisfies the following formula: L d =m d ||s d -s gt ||1;

[0242] Wherein, the Ld is the edge segmentation loss, the md is the target mask image, the Sd is the high-frequency image feature map, and the Sgt is the standard cutout image;

[0243] The third loss calculation formula satisfies the following formula: L a =||s p -s gt ||1+||s pe -s gte ||1

[0244] Among them, the is the image fusion loss, the Sp is the sample cutout image, the Sgt is the standard cutout image, the Spe is the cutout error map, and the Sgte is the reference cutout error map.

[0245] In one embodiment, when the processor 110 executes the step of obtaining a reference clipping error map for a sample clipping image, the processor 110 specifically performs the following steps:

[0246] Obtaining a standard cutout image corresponding to the sample training image;

[0247] Based on the standard matting image and the sample matting image, a reference matting error map is determined by using the initial object matting model.

[0248] In one embodiment, the processor 110 performs error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image, including:

[0249] Determine at least one reference pixel based on the pixel value of each pixel in the matting error map, wherein the pixel value is used to represent the matting confidence corresponding to the pixel;

[0250] Pixel blocks corresponding to the reference pixels are obtained from the cutout error map, and error fine-tuning processing is performed on the sample cutout image based on the pixel blocks to obtain a cutout corrected image.

[0251] In one embodiment, when the processor 110 determines the target cutout loss for the initial object cutout model based on the cutout correction image and the sample cutout image, the processor 110 specifically performs the following steps:

[0252] Performing an operator operation on the sample cutout image to obtain a first operation result; performing an operator operation on the standard cutout image to obtain a second operation result;

[0253] Determine a target cutout loss for the initial object cutout model using a fourth loss calculation formula based on the cutout correction image, the standard cutout image, the first operation result, and the second operation result;

[0254] The fourth loss calculation formula satisfies the following formula:

[0255] L c =1 / 2||s c -D(s gt )||2

[0256] Among them, the Lft is the target cutout loss, the Sft is the cutout correction image, the Sgt is the standard cutout image, the sobel() is the sobel operator operation function, the Sp is the sample cutout image, and the λedge is the weight factor.

[0257] In one embodiment, when the processor 110 performs the model correction processing on the initial object cutout model based on the target cutout loss and the reference cutout loss to obtain the object cutout model, the processor 110 specifically performs the following steps:

[0258] Determining a target cutout loss based on the target cutout loss and the reference cutout loss;

[0259] Based on the target cutout loss, a back propagation training method is adopted to perform model correction processing on the initial object cutout model to obtain an object cutout model.

[0260] In one embodiment, the reference clipping loss includes semantic segmentation loss, edge segmentation loss and image fusion loss. When the processor 110 determines the target clipping loss based on the target clipping loss and the reference clipping loss, the processor 110 specifically performs the following steps:

[0261] A weighted fusion process is performed based on the semantic segmentation loss, the edge segmentation loss, the image fusion loss and the reference cutout loss to obtain a target cutout loss.

[0262] In one embodiment, when executing the model processing method, the processor 110 further performs the following steps:

[0263] Acquire a target image, input the target image into the object cutout model, and output a target cutout image corresponding to the target image.

[0264] In one embodiment, before executing the step of determining the sample matting image and the matting error map corresponding to the sample training image by using the initial object matting model, the processor 110 further performs the following steps:

[0265] Acquire a training image for an initial object cutout model and a standard cutout image corresponding to the training image;

[0266] The light source data of the training image is enhanced based on the standard cutout image to obtain at least one sample training image for the training image.

[0267] In an embodiment of the present application, the electronic device can determine the sample cutout image and the cutout error map corresponding to the sample training image through the initial object cutout model, and then perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map, so as to obtain a trained object cutout model. In the object cutout model training process, by determining the cutout error map predicted for the sample cutout image, according to the error of each pixel misclassification predicted by the model in the cutout error map, the error correction effect can be improved, and an object cutout model with good robustness and high accuracy can be obtained, thereby improving the accuracy of image processing; and using a priori and rule-based method to correct parts of the cutout error map, the quality of object cutout can be further improved under the premise of ensuring real-time performance and without providing the original image background.

[0268] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0269] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A model processing method, characterized in that: The method comprises: Determine a sample matting image and a matting error map corresponding to the sample training image through an initial object matting model, wherein a pixel of any pixel point in the matting error map represents a matting confidence level corresponding to the pixel point; Performing model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model; The step of determining a sample matting image and a matting error map corresponding to a sample training image by using an initial object matting model includes: Based on the sample training image, an initial object cutout model is used to perform semantic segmentation to determine hidden semantic features of the image and a semantic segmentation map; The hidden semantic features of the image are obtained by downsampling the sample training image layer by layer to obtain the downsampled image result, then extracting the semantic features of the downsampled image result, and then fitting the extracted semantic features. The pixel of each pixel in the semantic segmentation map represents the confidence that the predicted pixel belongs to the foreground of the object. Based on the sample training image, the image hidden semantic features and the semantic segmentation map, the initial object cutout model is used to perform edge semantic segmentation processing on the object edge portion of the sample training image to determine a high-frequency image feature map, wherein the high-frequency image feature map is obtained after the initial object cutout model performs edge semantic segmentation processing on the object edge portion of the sample training image; The initial object matting model is used to determine a sample matting image and a matting error map based on the semantic segmentation map and the high-frequency image feature map.

2. The method according to claim 1, characterized in that The performing model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain the object cutout model includes: Obtaining a reference cutout loss for the initial object cutout model; Performing error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image; Determining a target matting loss for the initial object matting model based on the matting-corrected image and the sample matting image; The initial object cutout model is subjected to model correction processing based on the target cutout loss and the reference cutout loss to obtain an object cutout model.

3. The method according to claim 2, characterized in that Before obtaining the reference cutout loss for the initial object cutout model, the method further includes: A reference matting loss for the initial object matting model is determined.

4. The method according to claim 3, characterized in that The determining a reference cutout loss for the initial object cutout model comprises: Obtaining semantic segmentation loss, edge segmentation loss, and image fusion loss for the initial object cutout model; Based on the semantic segmentation loss, the edge segmentation loss and the image fusion loss, a reference matting loss for an initial object matting model is obtained.

5. The method according to claim 4, characterized in that The obtaining of semantic segmentation loss, edge segmentation loss and image fusion loss for the initial object cutout model includes: Obtain a standard cutout image corresponding to the sample training image, and obtain a semantic segmentation loss by using a first loss calculation formula based on the semantic segmentation map and the standard cutout image; Determine a ternary image corresponding to the sample training image and a target mask image corresponding to a target color region in the ternary image, and obtain an edge segmentation loss by using a second loss calculation formula based on the target mask image, the high-frequency image feature map, and the standard matting image; Obtain a reference cutout error map for the sample cutout image, and obtain an image fusion loss by using a third loss calculation formula based on the sample cutout image, the standard cutout image, the reference cutout error map, and the cutout error map; The first loss calculation formula satisfies the following formula: ; Among them, the is the semantic segmentation loss, the S c is the semantic segmentation map, the S gt is a standard cutout image, and D() is a downsampling function; The second loss calculation formula satisfies the following formula: ; Among them, the L d is the edge segmentation loss, the m d is the target mask image, the S d is the high-frequency image feature map, the S gt is a standard cutout image; The third loss calculation formula satisfies the following formula: Among them, the is the image fusion loss, the S p is the sample cutout image, the S gt is a standard cutout image, the S pe is the clipping error map, the S gte This is the reference clipping error map.

6. The method according to claim 5, characterized in that The obtaining of a reference cutout error map for the sample cutout image comprises: Obtaining a standard cutout image corresponding to the sample training image; Based on the standard matting image and the sample matting image, a reference matting error map is determined by using the initial object matting model.

7. The method according to claim 2, characterized in that The performing error correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain a cutout corrected image includes: Determine at least one reference pixel based on the pixel value of each pixel in the matting error map, wherein the pixel value is used to represent the matting confidence corresponding to the pixel; Pixel blocks corresponding to the reference pixels are obtained from the cutout error map, and error fine-tuning processing is performed on the sample cutout image based on the pixel blocks to obtain a cutout corrected image.

8. The method according to claim 2, characterized in that: The determining, based on the matting correction image and the sample matting image, a target matting loss for the initial object matting model comprises: Performing an operator operation on the sample cutout image to obtain a first operation result; performing an operator operation on the standard cutout image to obtain a second operation result; Determine a target cutout loss for the initial object cutout model using a fourth loss calculation formula based on the cutout correction image, the standard cutout image, the first operation result, and the second operation result; The fourth loss calculation formula satisfies the following formula: Among them, the L ft is the target cutout loss, the S ft To correct the image by cutting out, the S gt is a standard cutout image, sobel() is a sobel operator function, and S p is the sample cutout image, the λ edge is the weight factor.

9. The method according to claim 2, characterized in that: The performing model correction processing on the initial object cutout model based on the target cutout loss and the reference cutout loss to obtain the object cutout model includes: Determining a target cutout loss based on the target cutout loss and the reference cutout loss; Based on the target cutout loss, a back propagation training method is adopted to perform model correction processing on the initial object cutout model to obtain an object cutout model.

10. The method according to claim 9, characterized in that The reference image loss includes semantic segmentation loss, edge segmentation loss and image fusion loss. The determining of the target cutout loss based on the target cutout loss and the reference cutout loss includes: A weighted fusion process is performed based on the semantic segmentation loss, the edge segmentation loss, the image fusion loss and the reference cutout loss to obtain a target cutout loss.

11. The method according to claim 1, characterized in that: The method further comprises: Acquire a target image, input the target image into the object cutout model, and output a target cutout image corresponding to the target image.

12. The method according to claim 1, characterized in that Before determining the sample matting image and the matting error map corresponding to the sample training image by the initial object matting model, the method further includes: Acquire a training image for an initial object cutout model and a standard cutout image corresponding to the training image; The light source data enhancement process is performed on the training image based on the standard cutout image to obtain at least one sample training image for the training image.

13. A model processing device, characterized in that: The device comprises: A model training module, used to determine a sample matting image and a matting error map corresponding to a sample training image through an initial object matting model; A model correction module, configured to perform model correction processing on the initial object cutout model based on the sample cutout image and the cutout error map to obtain an object cutout model; The step of determining a sample matting image and a matting error map corresponding to a sample training image by using an initial object matting model includes: Based on the sample training image, an initial object cutout model is used to perform semantic segmentation to determine hidden semantic features of the image and a semantic segmentation map; The hidden semantic features of the image are obtained by downsampling the sample training image layer by layer to obtain the downsampled image result, then extracting the semantic features of the downsampled image result, and then fitting the extracted semantic features. The pixel of each pixel in the semantic segmentation map represents the confidence that the predicted pixel belongs to the foreground of the object. Based on the sample training image, the image hidden semantic features and the semantic segmentation map, the initial object cutout model is used to perform edge semantic segmentation processing on the object edge portion of the sample training image to determine a high-frequency image feature map, wherein the high-frequency image feature map is obtained after the initial object cutout model performs edge semantic segmentation processing on the object edge portion of the sample training image; The initial object matting model is used to determine a sample matting image and a matting error map based on the semantic segmentation map and the high-frequency image feature map.

14. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps as claimed in any one of claims 1 to 12.

15. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 12.