Image processing method and apparatus

Abstract

The application discloses an image processing method and device in the field of computer vision in the field of artificial intelligence. The image processing method comprises the following steps: acquiring a to-be-processed image; inputting the to-be-processed image into a segmentation network and a matting network in parallel, acquiring a first segmentation result of a foreground region and a background region output by the segmentation network, and acquiring a matting result output by the matting network; and fusing the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region. According to the technical scheme, the to-be-processed image is processed in parallel between the segmentation network and the matting network, so that the accuracy of the fine segmentation result of the to-be-processed image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and more particularly, to an image processing method and device in the field of computer vision. BACKGROUND

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military fields, and is a science about how to use cameras and computers to obtain the data and information of the photographed object. In a figurative way, it is to install eyes (cameras) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that the computer can perceive the environment.

[0003] Fine segmentation tasks have a wide range of applications in the field of computer vision. For example, image background blurring, background replacement, e-commerce advertisement production, live broadcast, film (animation) production, etc., wherein fine segmentation can refer to pixel-level fine segmentation of the target object on the obtained image or video. For example, the fine segmentation task of a portrait can refer to obtaining the fine segmentation result of the hair region.

[0004] Currently, when obtaining the fine segmentation result of an image, the image is usually input into a segmentation network to obtain a coarse segmentation result, and then the image and the coarse segmentation result of the image are input into a matting network to obtain the fine segmentation result of the image, that is, the matting network needs to rely on the coarse segmentation result to obtain the fine segmentation result of the image. However, when the coarse segmentation result of the image is inaccurate, the accuracy of the fine segmentation result of the image will be reduced.

[0005] Therefore, how to improve the accuracy of the fine segmentation result of the image to be processed has become a problem to be solved. SUMMARY

[0006] The present application provides an image processing method and device, which can improve the accuracy of the fine segmentation result of the image to be processed.

[0007] In a first aspect, an image processing method is provided, comprising: obtaining an image to be processed; inputting the image to be processed into a segmentation network and a matting network in parallel, obtaining a first segmentation result of a foreground region and a background region output by the segmentation network, and obtaining a matting result output by the matting network; fusing the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region.

[0008] It should be noted that in the present application, the first segmentation result can refer to a coarse segmentation result, and the second segmentation result can refer to a fine segmentation result, wherein the accuracy of the second segmentation result is greater than that of the first segmentation result.

[0009] In the present application, the to-be-processed image can be input into the segmentation network and the matting network in parallel, that is, the segmentation network and the matting network can process the same to-be-processed image in parallel, the matting network can process the to-be-processed image directly without depending on the coarse segmentation result, and the to-be-processed image is processed in parallel between the segmentation network and the matting network, so that the accuracy of the fine segmentation result of the to-be-processed image can be improved.

[0010] It should be understood that the to-be-processed image is input into the segmentation network and the matting network in parallel in the embodiments of the present application, which means that the to-be-processed image is input into the segmentation network and the matting network for processing respectively, and the order of inputting the to-be-processed image into the segmentation network and the matting network is not limited, wherein the segmentation network and the matting network can be two networks for processing the to-be-processed image respectively, that is, the matting network can process the to-be-processed image without depending on the segmentation result of the segmentation network.

[0011] Optionally, inputting the to-be-processed image into the segmentation network and the matting network in parallel can mean that the to-be-processed image is input into the segmentation network and the matting network for processing at the same time.

[0012] Optionally, inputting the to-be-processed image into the segmentation network and the matting network in parallel can mean that the to-be-processed image is input into the segmentation network and the matting network for processing in sequence.

[0013] In the embodiments of the present application, the image processing method can include: obtaining a to-be-processed image, the to-be-processed image containing a foreground region where a target object is located and a background region; inputting the to-be-processed image into a segmentation network and a matting network in parallel to obtain a coarse segmentation result of the foreground region and the background region output by the segmentation network, and to obtain a matting result of fine features of the target object output by the matting network; and fusing the coarse segmentation result and the matting result to obtain a fine segmentation result of the foreground region and the background region.

[0014] The target object contained in the to-be-processed image can be an object with a specific irregular edge region, for example, the target object can be a portrait, an animal, a stuffed toy, or a collar of clothes.

[0015] The image processing method can be executed on a server, in the cloud, or on a terminal device.

[0016] It should be noted that the fine features of the target object can be the detailed features of the target object. For example, when the target object is a portrait, the fine features can be the hair of the portrait; when the target object is an animal, the fine features can be the fur of the animal.

[0017] Optionally, the target object contained in the to-be-processed image can be a portrait, and the fine feature can be a hair of the portrait.

[0018] Optionally, the target object contained in the to-be-processed image can be an animal, and the fine feature can be a hair of the animal.

[0019] Optionally, the target object contained in the to-be-processed image can be a stuffed toy, and the fine feature can be a hair of the stuffed toy.

[0020] In a possible implementation, the method further includes: obtaining, according to the to-be-processed image, an attention map for an edge region of the foreground region; and fusing the first segmentation result and the matting result, including: fusing the first segmentation result and the matting result according to the attention map.

[0021] In the present application, the coarse segmentation result and the matting result are fused through the attention map, which can make the fusion pay more attention to the fine feature of the target object, that is, pay more attention to the detailed information in the to-be-processed image, thereby improving the accuracy of the fusion result, that is, improving the accuracy of the fine segmentation result of the to-be-processed image.

[0022] In a possible implementation, the fusing the first segmentation result and the matting result according to the attention map includes: performing convolution processing on the first segmentation result and the matting result according to the attention map.

[0023] In a possible implementation, the fusing the first segmentation result and the matting result according to the attention map includes: multiplying the values of corresponding pixel points in the first segmentation result and the matting result according to the attention map.

[0024] In a possible implementation, the obtaining, according to the to-be-processed image, an attention map for an edge region of the foreground region includes: obtaining feature information processed by the segmentation network on the to-be-processed image; and obtaining the attention map according to the to-be-processed image and the feature information.

[0025] The feature information can be a feature map of the to-be-processed image obtained by the segmentation network, or the feature information can also be a parameter of a convolution layer of the segmentation network.

[0026] In a possible implementation, the obtaining the attention map according to the to-be-processed image and the feature information includes: obtaining an attention map extraction network according to the feature information; and inputting the to-be-processed image into the attention map extraction network to obtain the attention map.

[0027] In the present application, the feature information of the segmentation network can be acquired, and the feature information of the trained segmentation network can be shared to reduce the calculation amount of the attention map extraction network for acquiring the attention map.

[0028] In a possible implementation, the matting network comprises an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than that of the encoding module.

[0029] In the present application, the matting network is used to learn more detailed information in the image, and the importance of the position information of the details in the image needs to be emphasized. Therefore, the asymmetric encoding-decoding structure can be used in the design of the matting network, that is, the number of neural network layers of the decoding module is greater than that of the encoding module, so that the accuracy of the matting network in processing fine features in the image to be processed can be improved.

[0030] In a possible implementation, the matting network is trained based on a first loss function and a second loss function, wherein the first loss function is used to distinguish the foreground region and the background region, and the second loss function is used to acquire the matting result of the fine features of the target object.

[0031] In the present application, the matting network is trained by using the first loss function and the second loss function, so that when the matting network is only input with the image to be processed without the coarse segmentation result, the semantic information of the matting network can be avoided to be lost, the accuracy of the matting network can be reduced, and the matting network can generally learn the difference between the foreground region and the background region where the target object is located.

[0032] Optionally, the first loss function can be a total loss function, which can be used to represent the loss value between the predicted matting result of the sample image obtained by the matting network and the labeled semantic segmentation feature of the sample image.

[0033] Optionally, the second loss function can be a local loss function, which can be used to represent the loss value between the edge region feature in the predicted matting result and the labeled edge region feature of the sample image, and the edge region can refer to the edge of the foreground region where the target object is located in the sample image.

[0034] In a second aspect, an image processing apparatus is provided, comprising: an acquisition unit configured to acquire an image to be processed; a processing unit configured to input the image to be processed into a segmentation network and a matting network in parallel, acquire a first segmentation result of a foreground region and a background region output by the segmentation network, and acquire a matting result output by the matting network; and fuse the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region.

[0035] It should be noted that the first segmentation result in the present application can refer to a coarse segmentation result, and the second segmentation result can refer to a fine segmentation result, wherein the accuracy of the second segmentation result is greater than that of the first segmentation result.

[0036] In the present application, the to-be-processed image can be input into the segmentation network and the matting network in parallel, that is, the segmentation network and the matting network can process the same to-be-processed image in parallel, the matting network can process the to-be-processed image directly without depending on the coarse segmentation result, and the to-be-processed image is processed in parallel between the segmentation network and the matting network, thereby improving the accuracy of the fine segmentation result of the to-be-processed image.

[0037] The image processing apparatus described above can be a server, and can also be a terminal device.

[0038] It should be understood that the to-be-processed image is input into the segmentation network and the matting network in parallel in the embodiments of the present application, which means that the to-be-processed image is input into the segmentation network and the matting network for processing respectively, and the order of inputting the to-be-processed image into the segmentation network and the matting network is not limited, wherein the segmentation network and the matting network can be two networks for processing the to-be-processed image respectively, that is, the matting network can process the to-be-processed image without depending on the segmentation result of the segmentation network.

[0039] Optionally, inputting the to-be-processed image into the segmentation network and the matting network in parallel can mean inputting the to-be-processed image into the segmentation network and the matting network simultaneously for processing.

[0040] Optionally, inputting the to-be-processed image into the segmentation network and the matting network in parallel can mean inputting the to-be-processed image into the segmentation network and the matting network in sequence for parallel processing.

[0041] In the embodiments of the present application, the image processing apparatus can include an acquisition unit and a processing unit, the acquisition unit can be used to acquire a to-be-processed image, the to-be-processed image contains a foreground region where a target object is located and a background region; the processing unit is used to input the to-be-processed image into a segmentation network and a matting network in parallel, obtain a coarse segmentation result of the foreground region and the background region output by the segmentation network, and obtain a matting result of fine features of the target object output by the matting network; and fuse the coarse segmentation result and the matting result to obtain a fine segmentation result of the foreground region and the background region.

[0042] The target object contained in the to-be-processed image described above can refer to an object with a specific irregular edge region. For example, the target object can be a portrait, an animal, a stuffed toy, or a clothing collar, etc.

[0043] It should be understood that the fine features of the target object can refer to the detailed features of the target object.

[0044] Optionally, the target object contained in the to-be-processed image can be a human portrait, and the fine feature can be a hair of the human portrait.

[0045] Optionally, the target object contained in the to-be-processed image can be an animal, and the fine feature can be a hair of the animal.

[0046] Optionally, the target object contained in the to-be-processed image can be a stuffed toy, and the fine feature can be a hair of the stuffed toy.

[0047] In a possible implementation, the processing unit is further configured to: obtain, according to the to-be-processed image, an attention map for an edge region of the foreground region; and fuse the first segmentation result and the matting result according to the attention map.

[0048] In the present application, the coarse segmentation result and the matting result are fused through the attention map, so that the fusion pays more attention to the fine feature of the target object, that is, pays more attention to the detailed information in the to-be-processed image, thereby improving the accuracy of the fusion result, that is, improving the accuracy of the fine segmentation result of the to-be-processed image.

[0049] In a possible implementation, the processing unit is specifically configured to: perform convolution processing on the first segmentation result and the matting result according to the attention map.

[0050] In a possible implementation, the processing unit is specifically configured to: multiply the values of corresponding pixel points in the first segmentation result and the matting result according to the attention map.

[0051] In a possible implementation, the processing unit is specifically configured to: obtain feature information obtained by the segmentation network from the to-be-processed image; and obtain the attention map according to the to-be-processed image and the feature information.

[0052] The feature information can be a feature map of the to-be-processed image obtained by the segmentation network, or the feature information can also be a parameter of a convolution layer of the segmentation network.

[0053] In a possible implementation, the processing unit is specifically configured to: obtain an attention map extraction network according to the feature information; and input the to-be-processed image into the attention map extraction network to obtain the attention map.

[0054] In the present application, the feature information of the segmentation network can be obtained, and the feature information of the trained segmentation network can be shared to reduce the calculation amount of the attention map extraction network used to obtain the attention map.

[0055] In a possible implementation, the matting network comprises an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than the number of neural network layers of the encoding module.

[0056] In the present application, the matting network is used to learn more detailed information in the image, and the importance of the position information of the details in the image needs to be emphasized. Therefore, in the design of the matting network, an asymmetric encoding-decoding structure can be used, that is, the number of neural network layers of the decoding module is greater than the number of neural network layers of the encoding module, so that the accuracy of the matting network in processing fine features in the image to be processed can be improved.

[0057] In a possible implementation, the matting network is trained based on a first loss function and a second loss function, wherein the first loss function is used to distinguish the foreground region and the background region, and the second loss function is used to obtain a matting result of fine features of the target object, and the foreground region contains the target object.

[0058] In the present application, by training the matting network through the first loss function and the second loss function, the matting network can avoid losing semantic information when only the image to be processed is input without inputting the coarse segmentation result, which reduces the accuracy of the matting network, and enables the matting network to roughly learn the difference between the foreground region and the background region where the target object is located.

[0059] Optionally, the first loss function can be a total loss function, which can be used to represent the loss value between the predicted matting result of the sample image obtained by the matting network and the labeled semantic segmentation features of the sample image.

[0060] Optionally, the second loss function can be a local loss function, which can be used to represent the loss value between the edge region features in the predicted matting result and the labeled edge region features of the sample image, and the edge region can refer to the edge of the foreground region where the target object is located in the sample image.

[0061] In a third aspect, an image processing apparatus is provided, which comprises a memory configured to store a program, and a processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the image processing method in the first aspect and any one of the implementation manners of the first aspect.

[0062] In a fourth aspect, a computer storage medium is provided, which stores program code, and the program code comprises instructions for executing the steps in the image processing method in the first aspect and any one of the implementation manners of the first aspect.

[0063] The storage medium can be a non-volatile storage medium.

[0064] In a fifth aspect, a chip is provided, which includes a processor and a data interface, the processor reads instructions stored on a memory through the data interface, and executes the image processing method in the first aspect and any one of the implementation manners of the first aspect.

[0065] Optionally, as an implementation manner, the chip can further include a memory, the memory stores instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to execute the image processing method in the first aspect and any one of the implementation manners of the first aspect.

[0066] The chip can be a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0068] Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0069] Figure 3 is a structural schematic diagram of a system architecture provided by an embodiment of the present application;

[0070] Figure 4 is a structural schematic diagram of a convolutional neural network provided by an embodiment of the present application;

[0071] Figure 5 is a structural schematic diagram of a chip hardware provided by an embodiment of the present application;

[0072] Figure 6 is a structural schematic diagram of a system architecture provided by an embodiment of the present application;

[0073] Figure 7 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0074] Figure 8 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0075] Figure 9 is a schematic diagram of a segmentation network processing an image provided by an embodiment of the present application;

[0076] Figure 10 is a schematic diagram of a matting network processing an image provided by an embodiment of the present application;

[0077] Figure 11 is a schematic diagram of an attention map extraction network processing an image provided by an embodiment of the present application;

[0078] Figure 12 is a schematic diagram of fusing the segmentation result and the matting result provided by an embodiment of the present application;

[0079] Figure 13 is a schematic diagram of a network architecture of the image processing method provided by an embodiment of the present application;

[0080] Figure 14 is an effect schematic diagram of the image processing method provided by an embodiment of the present application;

[0081] Figure 15 is a schematic block diagram of the image processing apparatus of an embodiment of the present application;

[0082] Figure 16 is a schematic block diagram of the image processing apparatus of an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0084] It should be understood that the image in the embodiments of the present application can be a static image (or static picture) or a dynamic image (or dynamic picture), for example, the image in the present application can be a video or a dynamic picture, or the image in the present application can also be a static picture or a photo. For ease of description, the static image or the dynamic image is uniformly referred to as an image in the following embodiments of the present application.

[0085] The image processing method of the embodiments of the present application can be applied in the scenarios of background blurring of an image, background replacement of an image, advertisement making of e-commerce, live broadcast, film (animation) making, etc. The following will introduce in detail the two scenarios of background blurring and background replacement of an image.

[0086] Background blurring of an image

[0087] As Figure 1As shown, the image processing method of this application embodiment can be applied to background blurring scenarios in images. For example, firstly, an original image can be acquired, such as an image acquired through a camera or an image stored locally on an electronic device; then, the original image is processed to obtain a fine segmentation result of the foreground and background regions of the original image. Specifically, the original image can be input in parallel into a segmentation network and a matting network to obtain the first segmentation result of the foreground and background regions output by the segmentation network, and the matting result output by the matting network; the first segmentation result and the matting result are then fused to obtain the second segmentation result of the original image; finally, based on the second segmentation result of the original image, the foreground region containing the target object can be made clearer, and the background blurred, achieving the effect of a large aperture on a DSLR camera.

[0088] In this application, the first segmentation result can refer to a coarse segmentation result, and the second segmentation result can refer to a fine segmentation result, wherein the precision of the second segmentation result is greater than that of the first segmentation result. The original image may contain a foreground region and a background region where the target object is located.

[0089] For example, when a user turns on the shooting function on a smart device, the image can be segmented in real time during the shooting process, so that the foreground area of ​​the subject is clear and the background area is blurred, achieving the image effect of a large aperture DSLR.

[0090] Image background replacement

[0091] like Figure 2 As shown, the image processing method of this application embodiment can be applied to background blurring scenarios in images. For example, firstly, an original image can be acquired, such as an image acquired through a camera or an image stored locally on an electronic device; then, the original image is processed to obtain a fine segmentation result of the foreground and background regions of the original image. Specifically, the original image can be input in parallel into a segmentation network and a matting network to obtain the first segmentation result of the foreground and background regions output by the segmentation network, and the matting result output by the matting network; the first segmentation result and the matting result are then fused to obtain a second segmentation result of the foreground and background regions; finally, based on the second segmentation result of the original image, the foreground region containing the target object can be retained, and the background region in the original image can be replaced, thereby achieving background replacement of the image.

[0092] In this application, the first segmentation result can refer to a coarse segmentation result, and the second segmentation result can refer to a fine segmentation result, wherein the precision of the second segmentation result is greater than that of the first segmentation result. The original image may contain a foreground region and a background region where the target object is located.

[0093] For example, a user starts a video call function in a smart terminal, and during the shooting process, image segmentation can be performed in real time, only the target object region is reserved, and the background region of the video call is replaced.

[0094] The smart terminal can be mobile or fixed, for example, the smart terminal can be a mobile phone, a tablet personal computer (TPC), a media player, a smart television, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smart watch, a wearable device (WD), or an autonomous vehicle, etc. with image processing function, and the embodiments of the present application are not limited thereto.

[0095] It should be understood that the background blurring of the image and the background replacement of the image introduced above are only two specific scenes to which the image processing method of the embodiments of the present application is applied, and the image processing method of the embodiments of the present application is not limited to the above two scenes when applied. The image processing method of the embodiments of the present application can be applied to any scene requiring image segmentation.

[0096] It should be understood that the above is an example of the application scenario, and does not limit the application scenario of the present application.

[0097] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts of the neural networks that may be involved in the embodiments of the present application will be introduced first.

[0098] (1) Neural network

[0099] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with x s and intercept 1 as input, and the output of the operation unit can be:

[0100]

[0101] wherein s = 1, 2, … n, n is a natural number greater than 1, W s x swhere b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neuron to an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neurons, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neurons.

[0102] (2) Deep neural network

[0103] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0104] Although the DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: wherein, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Due to the large number of layers in the DNN, the number of coefficients W and offset vectors is also relatively large. These parameters in the DNN are defined as follows: taking the coefficient W as an example: assuming that in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

[0105] In summary, the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as

[0106] It is noted that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better capture the complexity of real-world situations. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity" to learn more complex tasks. Training a deep neural network is the process of learning the weight matrices, and the ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (the weight matrices formed by the vectors W of many layers).

[0107] (3) Convolutional Neural Network

[0108] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The CNN contains a feature extractor composed of convolutional layers and subsampling layers, which can be regarded as a filter. The convolutional layer refers to the layer of neurons in the CNN that performs convolutional processing on the input signal. In the convolutional layer of the CNN, a neuron can be connected only to some of the adjacent layer neurons. In a convolutional layer, there are usually several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights are the convolutional kernels.

[0109] The shared weights can be understood as a way of extracting image information that is independent of the location. The convolutional kernels can be initialized in the form of a matrix of random size, and the convolutional kernels can obtain reasonable weights through learning in the training process of the CNN. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the CNN and to reduce the risk of overfitting.

[0110] (4) Loss Function

[0111] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vectors of each layer of the neural network can be updated by comparing the predicted value of the current network with the target value that is truly intended to be predicted, and according to the difference between the two. For example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the target value that is truly intended to be predicted or a value very close to it.

[0112] Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss.

[0113] (5) Back propagation algorithm

[0114] The neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal will produce an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0115] Figure 3 A system architecture 100 provided by an embodiment of the present application is shown.

[0116] In Figure 3 In the embodiment of the present application, the data acquisition device 160 is used to collect training data. For the image processing method of the present application, the matting network or the segmentation network can be trained by training images, that is, the training data collected by the data acquisition device 160 can be training images, and the training images can include annotated fine portrait segmentation data (A) and ordinary portrait segmentation data (S).

[0117] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0118] The training device 120 obtains the target model / rule 101 based on the training data, which is described as follows. The training device 120 processes the input original image, compares the output image with the original image, and until the difference between the output image of the training device 120 and the original image is less than a certain threshold, the training of the target model / rule 101 is completed.

[0119] For example, in the image processing method provided in this application, the target model can refer to a segmentation network and / or a matting network. The segmentation network can be trained using ordinary portrait segmentation data (S), and the obtained semantic segmentation results are compared with the real labels to calculate the difference. The network parameters are then updated using the obtained difference, and the above steps are repeated until a pre-set number of training iterations is reached or the segmentation network difference is within a preset range.

[0120] Furthermore, after the segmentation network is trained, its parameters can be fixed, and one or more of the matting network, attention module, and fusion module can be trained using labeled, high-resolution human image segmentation data (A). The refined segmentation results are compared with the real labels, and the difference is calculated. The network parameters are then updated using the difference, and the above steps are repeated until a pre-set number of training iterations is reached or the segmentation network difference is within a preset range.

[0121] The aforementioned target model / rule 101 can be used to implement the image processing method of this application embodiment. Specifically, the target model / rule 101 in this application embodiment can be a segmentation network and / or a matting network.

[0122] It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0123] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 3 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, or the cloud, etc.

[0124] exist Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include the image to be processed input by the client device.

[0125] The preprocessing modules 113 and 114 are configured to perform preprocessing on input data (e.g., images to be processed) received by the I / O interface 112. In some embodiments, the preprocessing modules 113 and 114 can be omitted (only one of them can be used) and the input data can be directly processed by the computing module 111.

[0126] During the preprocessing of the input data by the execution device 110 or the processing performed by the computing module 111 of the execution device 110, the execution device 110 can call data and codes in the data storage system 150 for the corresponding processing, and the data and instructions obtained by the corresponding processing can be stored in the data storage system 150.

[0127] Finally, the I / O interface 112 returns the processing result (e.g., the fine segmentation result obtained in some embodiments) to the client device 140, so as to provide the user with the result.

[0128] It is worth noting that the training device 120 can generate a corresponding target model / rule 101 based on different training data for different targets or tasks, and the corresponding target model / rule 101 can be used to achieve the above targets or complete the above tasks, so as to provide the user with the required result.

[0129] In the case shown in Figure 3 , the user can manually give the input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send the input data to the I / O interface 112. If the authorization of the user is required for the client device 140 to automatically send the input data, the user can set the corresponding permission in the client device 140. The user can view the result output by the execution device 110 in the client device 140, and the specific presentation form can be a specific way such as display, sound, and action. The client device 140 can also be used as a data collection end to collect the input data input into the I / O interface 112 and the output result output from the I / O interface 112 as new sample data and store them in the database 130. Of course, the input data input into the I / O interface 112 and the output result output from the I / O interface 112 can be directly stored in the database 130 as new sample data without the collection through the client device 140.

[0130] It is worth noting that Figure 3 The system architecture shown in the figure is only an example provided by some embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in some embodiments, the training device 120 can be integrated into the execution device 110. Figure 3In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0131] like Figure 3 As shown, the target model / rule 101 is obtained by training the training device 120. In this embodiment of the application, the target model / rule 101 may be the segmentation network or matting network in this application. Specifically, the segmentation network or matting network provided in this embodiment of the application may be a convolutional neural network (CNN) or a deep convolutional neural network (DCNN), etc.

[0132] The following is combined with Figure 4 This section focuses on a detailed introduction to the structure of Convolutional Neural Networks (CNNs). CNNs are a very common type of neural network. As mentioned in the basic concept introduction above, a CNN is a deep neural network with a convolutional structure, and it is a deep learning architecture. Deep learning architectures refer to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNNs are feedforward artificial neural networks, where each neuron can respond to the input image.

[0133] In the image processing method of this application embodiment, the neural network structure used by the segmentation network and the matting network can be as follows: Figure 4 As shown. In Figure 4 In this example, the convolutional neural network 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 acquires the image to be processed and then passes the acquired image to the convolutional / pooling layer 220 and the subsequent neural network layer 230 for processing to obtain the image processing result. The following section... Figure 4 This section provides a detailed introduction to the internal layer structure of CNN 200.

[0134] Convolutional / pooling layers 220:

[0135] like Figure 4 The convolutional / pooling layer 220 shown may include layers such as 221-226 in the examples. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer.

[0136] In another implementation, 221 and 222 are convolution layers, 223 is a pooling layer, 224 and 225 are convolution layers, and 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0137] The internal working principle of a convolution layer will be described below taking the convolution layer 221 as an example.

[0138] The convolution layer 221 can include a plurality of convolution operators, also known as kernels, which function as filters for extracting specific information from an input image matrix in image processing. A convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation on an image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, etc., depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix will produce a single-depth-dimension convolution output, but most cases do not use a single weight matrix, but rather apply a plurality of weight matrices of the same size (rows x columns), i.e., a plurality of homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the above-mentioned “plurality”.

[0139] Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract a specific color of the image, and yet another weight matrix is used to blur unwanted noise in the image, etc. The plurality of weight matrices are of the same size (rows x columns), and the sizes of the convolution feature maps extracted by the plurality of weight matrices of the same size are also the same. The plurality of convolution feature maps of the same size are combined to form the output of the convolution operation.

[0140] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the trained weight values can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0141] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0142] Pooling layer:

[0143] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 4 Layers 221-226, as exemplified in convolutional / pooling layers 220, can be one convolutional layer followed by one pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling operators and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0144] Neural network layer 230:

[0145] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize neural network layers 230 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 230 can include multiple hidden layers (such as...). Figure 4 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image detection, and image super-resolution reconstruction.

[0146] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, the output layer 240 has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error, once the forward propagation of the entire convolutional neural network 200 (such as Figure 4 propagation from 210 to 240 is completed, the back propagation (such as Figure 4 propagation from 240 to 210) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200, and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0147] Optionally, in some implementations, due to the convolution operation in the above-mentioned convolution layer, the pooling operation in the pooling layer and the operation in the neural network layer can be regarded as linear calculation, therefore, in order to improve the description ability of the CNN model, an activation function can be introduced into the above-mentioned CNN network, in order to increase the non-linear operation in the CNN, therefore, the operation based on the activation function can also be called "non-linear operation".

[0148] For example, if the above-mentioned linear operation is performed in the layer 221 and the layer 222, an activation function can be inserted between the layer 221 and the layer 222, that is, the output of the layer 221 is taken as the input of the activation function, and the output of the activation function is taken as the input of the layer 222. It should be understood that the above-mentioned activation function can also be arranged between the hidden layer 231 and the hidden layer 232, and the arrangement mode of the activation function is not limited in the embodiments of the present application.

[0149] It should be noted that, Figure 4 The convolutional neural network shown is only an example of the convolutional neural network used by the matting network and the segmentation network of the embodiments of the present application, and in specific applications, the convolutional neural network used by the image processing method of the embodiments of the present application can also exist in the form of other network models.

[0150] Figure 5 A hardware structure of a chip provided by the embodiments of the present application, the chip includes a neural network processing unit 300 (neural-network process units, NPU). The chip can be arranged in the execution device 110 as shown, to complete the calculation work of the calculation module 111. The chip can also be arranged in the training device 120 as shown, to complete the training work of the training device 120 and output the target model / rule 101. The algorithm of each layer in the convolutional neural network as shown Figure 3 Figure 3 Figure 4 Figure 5 ​​​implemented in the illustrated chip.

[0151] The neural network processor (NPU) 300 is mounted as a co-processor to a main central processing unit (CPU) and is assigned tasks by the main CPU. The core of the NPU 300 is an arithmetic circuit 303, and a controller 304 controls the arithmetic circuit 303 to fetch data from a memory (weight memory or input memory) and perform arithmetic operations.

[0152] In some implementations, the arithmetic circuit 303 includes a plurality of processing engines (PEs) inside. In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0153] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 fetches the corresponding data of the matrix B from the weight memory 302 and caches it on each PE of the arithmetic circuit 303. The arithmetic circuit 303 performs matrix operations on the data of the matrix A fetched from the input memory 301 and the matrix B to obtain partial results or final results of the matrix, which are saved in an accumulator 308.

[0154] The vector computation unit 307 can further process the output of the arithmetic circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 307 can be used for network computation of non-convolution / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0155] In some implementations, the vector computation unit 307 stores the processed output vector to the unified memory 306. For example, the vector computation unit 307 can apply a non-linear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 303, such as for use in a subsequent layer in a neural network.

[0156] The unified memory 306 is used to store input data and output data.

[0157] The input data in the external memory is stored into the input memory 301 and / or the unified memory 306, the weight data in the external memory is stored into the weight memory 302, and the data in the unified memory 306 is stored into the external memory by the direct memory access controller (DMAC) 305.

[0158] The bus interface unit 310 (BIU) is used to realize the interaction among the main CPU, the DMAC and the instruction fetch memory 309 through a bus.

[0159] The instruction fetch memory 309 connected with the controller 304 is used to store the instructions used by the controller 304.

[0160] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to realize the control of the working process of the operation accelerator.

[0161] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memories.

[0162] Wherein, Figure 4 The operations of each layer in the convolutional neural network shown in the above can be executed by the operation circuit 303 or the vector calculation unit 307.

[0163] The execution device 110 in the above Figure 3 The execution device 110 in the above Figure 4 The chip shown in the above Figure 5 The chip shown in the above

[0164] The image processing method of the embodiments of the present application will be described in detail below. Figures 6 to 14 The image processing method of the embodiments of the present application will be described in detail below.

[0165] As Figure 6As shown, the embodiments of the present application provide a system architecture 400. The system architecture includes a local device 420, a local device 430, and an execution device 410 and a data storage system 450, wherein the local device 420 and the local device 430 are connected with the execution device 410 through a communication network.

[0166] The execution device 410 can be implemented by one or more servers.

[0167] Optionally, the execution device 410 can be used in cooperation with other computing devices, such as data storage, routers, load balancers, and the like. The execution device 410 can be arranged on one physical site or distributed on multiple physical sites. The execution device 410 can use data in the data storage system 450 or call program code in the data storage system 450 to implement the image processing method of the embodiments of the present application.

[0168] It should be noted that the above-mentioned execution device 410 can also be referred to as a cloud device, and at this time the execution device 410 can be deployed in the cloud.

[0169] Specifically, the execution device 410 can perform the following processes: obtaining a to-be-processed image; inputting the to-be-processed image into a segmentation network and a matting network in parallel, obtaining a first segmentation result of a foreground region and a background region output by the segmentation network, and obtaining a matting result output by the matting network; and fusing the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region.

[0170] In a possible implementation, the above-mentioned image processing method of the execution device 410 can be an offline method executed in the cloud.

[0171] In the embodiments of the present application, the to-be-processed image can include a foreground region where a target object is located and a background region; the to-be-processed image is input into the segmentation network and the matting network in parallel, so as to obtain a coarse segmentation result of the foreground region and the background region output by the segmentation network, and obtain a matting result of fine features of the target object output by the matting network.

[0172] Users can operate their respective user devices (for example, the local device 420 and the local device 430) to interact with the execution device 410. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, and the like.

[0173] The local device of each user can interact with the execution device 410 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, or the like, or any combination thereof.

[0174] In an implementation manner, the local device 420 and the local device 430 can obtain the related parameters of the segmentation network and the matting network from the execution device 410, deploy the segmentation network and the matting network on the local device 420 and the local device 430, and perform image processing and the like by using the segmentation network and the matting network.

[0175] In another implementation manner, the segmentation network and the matting network can be directly deployed on the execution device 410, and the execution device 410 can obtain the to-be-processed image from the local device 420 and the local device 430, and perform image processing on the to-be-processed image according to the segmentation network and the matting network.

[0176] The image processing method of the embodiment of the present application will be described in detail below. Figure 7 The image processing method of the embodiment of the present application will be described in detail below. Figure 7 The image processing method shown in the figure can be executed by an image processing apparatus, which can be an execution device or a local device. Figure 6 The image processing method shown in the figure can be executed by an image processing apparatus, which can be an execution device or a local device. Figure 7 The method 500 shown in the figure includes steps 510 to 530, which will be described in detail below.

[0177] Step 510, obtaining a to-be-processed image.

[0178] In the embodiment of the present application, the to-be-processed image can be an image captured by a camera of an electronic device, or the to-be-processed image can also be an image obtained from the inside of the electronic device (for example, an image stored in the photo album of the electronic device, or a picture obtained from the cloud by the electronic device).

[0179] Step 520, inputting the to-be-processed image into the segmentation network and the matting network in parallel, obtaining a first segmentation result of a foreground region and a background region output by the segmentation network, and obtaining a matting result output by the matting network.

[0180] It should be noted that, in the embodiment of the present application, the first segmentation result can be a coarse segmentation result, and the second segmentation result can be a fine segmentation result, wherein the accuracy of the second segmentation result is higher than that of the first segmentation result.

[0181] In the embodiments of the present application, the to-be-processed image can include a foreground region where the target object is located and a background region; the to-be-processed image is input into the segmentation network and the matting network in parallel, so that the coarse segmentation result of the foreground region and the background region output by the segmentation network can be obtained, and the matting result of the fine feature of the target object output by the matting network can be obtained.

[0182] It should be understood that in the embodiments of the present application, the target object can be an image with a specific irregular edge region. For example, the target object can be a portrait, an animal, a stuffed toy, or a clothing collar, etc.

[0183] Optionally, the to-be-processed image can be an RGB image.

[0184] It should be understood that the fine feature of the target object can refer to the detailed feature of the target object.

[0185] For example, the target object included in the to-be-processed image described above can be a portrait, and the fine feature can be a hair of the portrait.

[0186] For example, the target object included in the to-be-processed image described above can be an animal, and the fine feature can be the fur of the animal.

[0187] For example, the target object included in the to-be-processed image described above can be a stuffed toy, and the fine feature can be the fur of the stuffed toy.

[0188] It should be noted that in the embodiments of the present application, the to-be-processed image is input into the segmentation network and the matting network in parallel, which means that the to-be-processed image is input into the segmentation network and the matting network respectively and processed in parallel, and the order in which the to-be-processed image is input into the segmentation network and the matting network is not limited, wherein the to-be-processed image can be input into the segmentation network and the matting network for processing at the same time, that is, in the embodiments of the present application, the matting network can process the to-be-processed image directly without depending on the segmentation result of the segmentation network.

[0189] For example, the to-be-processed image is input into the segmentation network and the matting network in parallel can mean that the to-be-processed image is input into the segmentation network and the matting network at the same time.

[0190] For example, the to-be-processed image is input into the segmentation network and the matting network in parallel can mean that the to-be-processed image is first input into the segmentation network, and then the to-be-processed image is input into the matting network.

[0191] For example, the to-be-processed image is input into the segmentation network and the matting network in parallel can mean that the to-be-processed image is first input into the matting network, and then the to-be-processed image is input into the segmentation network.

[0192] In the embodiments of this application, the segmentation network can roughly distinguish between the foreground and background regions in the image to be processed, but it cannot learn more detailed information about the image to be processed.

[0193] For example, in an embodiment of this application, the segmentation network may be a segmentation network whose output is a three-classification mask, wherein the three-classification mask may refer to dividing the image to be processed into a foreground region where the target object is located, a background region, and an uncertain region.

[0194] For example, in an embodiment of this application, the segmentation network may be a segmentation network whose output is a binary classification mask, wherein the binary classification mask refers to dividing the image to be processed into a foreground region where the target object is located and a background region.

[0195] It should be noted that the segmentation network in this application embodiment is used to obtain preliminary segmentation results of the foreground and background regions of the image to be processed. As for the specific structure of the segmentation network, this application does not impose any limitations.

[0196] like Figure 9 As shown, the segmentation network can adopt a conventional encoder-decoder structure. By processing the image to be processed through the segmentation network, a coarse segmentation result of the image to be processed can be obtained, which can also be called the segmentation feature of the image to be processed.

[0197] It should be understood that in the embodiments of this application, the segmentation network can be used to process the image to be processed to obtain coarse segmentation results of the foreground region and the background region. The coarse segmentation result can refer to the segmentation result that has a general foreground region and background region but has low accuracy for the fine features of the target object.

[0198] For example, when the target object is a human portrait, fine features can refer to the hair area. Coarse segmentation results can roughly distinguish the foreground and background areas where the portrait is located, but the accuracy of coarse segmentation results for fine features of the portrait, such as the hair area, is low.

[0199] In the embodiments of this application, the matting network can refer to a fine segmentation network, that is, a network with high accuracy that can obtain more detailed information in the image to be processed.

[0200] For example, if the target object in the image to be processed is an animal, the matting network can perform fine-grained matting of the animal's fur and other features.

[0201] Step 530: Merge the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region.

[0202] In the embodiments of this application, the second segmentation result can refer to the fine segmentation result or the fine segmentation mask (Alpha). For example, when the target object in the image to be processed is a human portrait, the fine segmentation result can refer to the pixel-level fine segmentation result of the human portrait, with a value range of 0 to 1.

[0203] In one possible implementation, in order to make the coarse segmentation result and the matting result pay more attention to the fine features of the target object, that is, to pay more attention to the details in the image to be processed, when fusing the coarse segmentation result and the matting result, an attention map can be used to fuse the coarse segmentation result and the matting result. Here, the attention map can refer to the attention map of the edge region of the foreground region in the image to be processed.

[0204] For example, in an embodiment of this application, an attention map of the edge region of the foreground region can be obtained based on the image to be processed; and the coarse segmentation result and the matting result can be fused based on the attention map.

[0205] In one possible implementation, the above fusion can be performed by convolving the first segmentation result and the matting result based on the attention map.

[0206] For example, a fine segmentation result of the foreground and background regions in the image to be processed can be obtained by convolving the attention map, coarse segmentation result and matting result through a fusion network.

[0207] For example, Figure 12 The diagram illustrates the fusion of coarse segmentation and matting results based on an attention map. The fusion module, which can be a neural network, fuses the coarse segmentation results (segmentation features) and matting results (mating features) through the attention map to obtain the final refined segmentation mask (Alpha).

[0208] In another possible implementation, the above fusion may refer to multiplying the pixel values ​​of the corresponding regions in the first segmentation result and the matting result according to the attention map.

[0209] Optionally, an attention map for the edge region of the foreground region of the image to be processed can be obtained by training a neural network with the edge region of the foreground region of a manually annotated sample image to obtain an attention map extraction network.

[0210] like Figure 11 The diagram shown is a schematic of the attention map obtained from the image to be processed, where the attention module can be a neural network used to extract the attention map.

[0211] Further, in order to reduce the amount of calculation of the attention map extraction network for obtaining the attention map, i.e., to design an attention map extraction network with a simple network structure, the segmentation network trained can be used to obtain the feature information of the image to be processed.

[0212] Optionally, obtaining the attention map for the edge region of the foreground region according to the image to be processed can refer to obtaining the feature information of the image to be processed by the segmentation network; and obtaining the attention map according to the image to be processed and the feature information.

[0213] The feature information can refer to the feature map of the image to be processed obtained by the segmentation network, or the feature information can also refer to the parameters of the convolutional layer of the segmentation network. By sharing the features of the trained segmentation network, the amount of calculation of the attention map extraction network can be reduced.

[0214] Optionally, the attention map extraction network can be obtained according to the obtained feature information; and the image to be processed is input into the attention map network to obtain the attention map.

[0215] In the traditional framework, the matting network is based on the rough segmentation result to perform matting on the image to be processed, and the rough segmentation result can provide the matting network with approximate semantic information. When the matting network is only input with the image to be processed without the rough segmentation result, the matting network loses the semantic information, which reduces the accuracy of matting. Therefore, in the embodiments of the present application, the parameters of the matting network can be trained by the first loss function and the second loss function, so that the matting network can roughly learn the difference between the foreground region where the target object is located and the background region.

[0216] Optionally, the matting network can be trained based on the first loss function and the second loss function, wherein the first loss function can be used to distinguish the foreground region and the background region, and the second loss function can be used to obtain the matting result of the fine features of the target object of the image to be processed.

[0217] In addition, a symmetrical encoding-decoding structure can be generally used in image processing. In order to achieve balance in image processing, i.e., balance between convolution processing and dimension reduction operation and convolution processing and up-sampling operation. The encoding process can be roughly the process of convolution processing and dimension reduction processing. The encoding is to enable the network to learn the information in the image features. In general, the shallow convolution in the encoding can learn the detailed information in the image, and the deep convolution in the encoding can learn the overall information in the image. The decoding process can be regarded as the process of convolution processing and up-sampling processing, which is used to restore the size and pixels of the image. The deeper the decoding network is designed, the more benefits it can have, i.e., the image can be restored pixel by pixel, and the position information of each pixel in the image can be ensured.

[0218] In the embodiments of the present application, the matting network is to learn more detailed information in the image, and meanwhile, the importance of the position information of the details in the image can be emphasized. Therefore, the symmetrical encoding-decoding structure can not be used when the matting network is designed. In order to make the matting network learn more detailed features in the image, a relatively shallow neural network layer can not be used when the encoding module is designed; in order to pay more attention to the position information of the detailed features in the image, a relatively deep neural network layer can be used when the decoding module is designed.

[0219] It should be understood that when the number of neural network layers in the decoding module is small, the image size can be directly recovered by upsampling after some pixel points are recovered, and at this time, the positions of some pixel points can be inaccurate.

[0220] Exemplarily, as shown in Figure 10 In the embodiments of the present application, the matting network can include an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than that of the encoding module.

[0221] In the present application, the to-be-processed image is input into the segmentation network and the matting network in parallel, which can include but is not limited to the following two cases.

[0222] Case one:

[0223] In one embodiment, the above-mentioned segmentation network and the above-mentioned matting network can use the segmentation network and the matting network in the prior art.

[0224] Since the segmentation semantic information of the image is not provided to the matting network when the existing matting network is used, the matting result of the fine features of the target object obtained by directly performing matting processing on the to-be-processed RGB image can have errors, and at this time, the coarse segmentation result and the matting result can be input into a neural network for fusion processing, and the parameters of the neural network for fusion processing can be obtained by training the first loss function and the second loss function, wherein the first loss function can be used to distinguish the foreground region and the background region of the to-be-processed image, and the second loss function can be used to obtain the matting result of the fine features of the target object.

[0225] Exemplarily, the first loss function can be a global loss function, which can be used to represent the loss value between the predicted matting result of the sample image obtained by the matting network and the labeled semantic segmentation features of the sample image; the second loss function can be a local loss function, which can be used to represent the loss value between the edge region features in the predicted matting result and the labeled edge region features of the sample image, and the edge region refers to the edge of the foreground region of the target object in the sample image.

[0226] For example, the to-be-processed image is an image containing a foreground region where a portrait is located and a background region, the overall loss function can be used to represent an overall loss value of the foreground region and the background region of the obtained predicted portrait matting result, and the local loss function can be used to represent a loss value of a hair region of the portrait.

[0227] Further, in order to make the coarse segmentation result and the matting result pay more attention to the fine features of the target object, that is, pay more attention to the detail information in the to-be-processed image, when the coarse segmentation result and the matting result are fused, the fusion can be performed through an attention map, where the attention map can refer to an attention map of an edge region of the foreground region in the to-be-processed image.

[0228] For example, the attention map of the edge region of the foreground region can be obtained according to the to-be-processed image, and the coarse segmentation result and the matting result are fused according to the attention map.

[0229] As shown in Figure 11 FIG. 1 is a schematic diagram of an attention map obtained according to a to-be-processed image, where the attention module can be a neural network used to extract the attention map.

[0230] In a possible implementation, when the above attention map is obtained, a neural network can be trained through an edge region of a foreground region of a sample image marked by a person to obtain an attention map extraction network used to obtain the attention map.

[0231] In another possible implementation, in order to reduce the calculation amount of the attention map extraction network used to obtain the attention map, that is, to design a simple network structure of the attention map extraction network, the feature information obtained by processing the to-be-processed image by using the trained segmentation network can be shared.

[0232] The feature information can refer to a feature map of the to-be-processed image obtained by the segmentation network, or the feature information can also refer to parameters of a convolution layer of the segmentation network. The feature information of the trained segmentation network can be shared to reduce the calculation amount of the attention map extraction network.

[0233] Optionally, the attention map extraction network can be obtained according to the obtained feature information, and the to-be-processed image is input into the attention map network to obtain the attention map.

[0234] Case two:

[0235] In an embodiment, the above segmentation network can adopt a segmentation network in the prior art, and the matting network can adopt the matting network proposed in the embodiments of the present application.

[0236] For example, the parameters of the matting network can be trained based on a first loss function and a second loss function, where the first loss function can be used to distinguish the foreground region and the background region, and the second loss function can be used to obtain a matting result of fine features of the target object.

[0237] It should be noted that in the traditional framework, the matting network performs matting on the to-be-processed image based on a coarse segmentation result, and the coarse segmentation result can provide rough semantic information for the matting network. When the matting network is only input with the to-be-processed image without the coarse segmentation result, the matting network loses the semantic information, and thus the first loss function can make up for the lost semantic information of the matting network.

[0238] For example, when training the matting network, a sample image and an artificial annotated fine segmentation result corresponding to the sample image can be input, and the pixel points with a value of 1 or close to 1 in the artificial annotated fine segmentation result can represent the foreground region where the target object is located, and the pixel points with a value of 0 or close to 0 can represent the background region; and the matting network is trained by using the overall loss function, so that the matting network can roughly distinguish the foreground region and the background region when performing matting processing based on the RGB image.

[0239] Optionally, the matting network can include an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than the number of neural network layers of the encoding module.

[0240] The commonly used encoding-decoding structure is symmetrical, which is to realize the balance in image processing, that is, the balance between convolution processing and dimension reduction operation and convolution processing and up-sampling operation. The encoding process can generally refer to the process of convolution processing and dimension reduction processing, and the encoding is to enable the network to learn the information in the image features. In general, the shallow convolution in the encoding can learn the detailed information in the image, and the deep convolution in the encoding can learn the overall information in the image; the decoding process can be regarded as the process of convolution processing and up-sampling processing, which is used to restore the size and pixels of the image. The deeper the decoding network is designed, the more benefits it has, which can gradually recover the image pixel by pixel to ensure the position information of each pixel in the image.

[0241] For example, when the decoding network is designed to be relatively shallow, that is, the number of layers is small, the up-sampling is directly used to restore the image size after recovering part of the pixels, and at this time, the positions of part of the pixels are inaccurate.

[0242] In the embodiments of the present application, the matting network is to learn more detailed information in the image, and thus the convolution in the encoding network can not be designed to be deep. For example, as shown in FIG. 1, the encoding network can be designed to be shallow, that is, the number of layers is small. Figure 10It is shown that the symmetry of the encoding-decoding structure can be changed in the embodiments of the present application. After increasing the overall loss function of the image when training the matting network, it is still desired that the matting network also pay attention to the detailed information, i.e., the low-level features, so the matting network does not need to be too deep in the encoding stage, but should strengthen the importance of the position information of the details such as hair, so the matting network adopts a deeper decoding stage than the encoding stage.

[0243] Exemplarily, the parameters of the above-mentioned matting network in the embodiments of the present application can be obtained by training the first loss function and the second loss function, and the accuracy of the matting result of the fine features of the target object obtained by the above-mentioned matting network is higher than that obtained by using the matting network in the prior art, so when the coarse segmentation result and the matting result are fused, the fine segmentation result of the foreground region and the background region of the to-be-processed image can also be obtained by multiplying the numerical values of the corresponding pixel points of the coarse segmentation result and the matting result.

[0244] Further, in order to make the coarse segmentation result and the matting result pay more attention to the fine features of the target object when they are fused, i.e., pay more attention to the detailed information in the to-be-processed image, the coarse segmentation result and the matting result can be fused by using an attention map when they are fused, wherein the attention map can refer to the attention map of the edge region of the foreground region in the to-be-processed image.

[0245] Exemplarily, the attention map of the edge region of the foreground region can be obtained according to the to-be-processed image, and the coarse segmentation result and the matting result are fused according to the attention map.

[0246] In a possible implementation, the above-mentioned fusion can be a convolutional processing of the first segmentation result and the matting result according to the attention map.

[0247] For example, the attention map, the coarse segmentation result and the matting result can be convoluted by the fusion network to obtain the fine segmentation result of the foreground region and the background region in the to-be-processed image.

[0248] For example, Figure 12 It is shown that the coarse segmentation result and the matting result are fused according to the attention map. The fusion module can be a neural network, and the coarse segmentation result (i.e., the segmentation feature) and the matting result (i.e., the matting feature) can be fused by the fusion module through the attention map to obtain the final fine segmentation mask (Alpha).

[0249] For example, the parameters of the neural network used for fusion processing can be trained by an overall loss function and a local loss function of an image, wherein the first loss function can be the overall loss function, and the overall loss function can be used to represent a loss value between a predicted matting result of a sample image obtained by a matting network and a labeled semantic segmentation feature of the sample image; the second loss function can be the local loss function, and the local loss function can be used to represent a loss value between an edge region feature in the predicted matting result and a labeled edge region feature of the sample image, wherein the edge region refers to an edge of a foreground region of a target object in the sample image.

[0250] In another possible implementation, the fusion can refer to multiplying the values of the pixels in the corresponding regions in the first segmentation result and the matting result according to the attention map.

[0251] Optionally, the neural network used for obtaining the attention map can be trained by an edge region of a foreground region of a sample image labeled by a human being to obtain an attention map extraction network used for obtaining the attention map.

[0252] In another possible implementation, in order to reduce the calculation amount of the attention map extraction network used for obtaining the attention map, i.e., to design a simple network structure of the attention map extraction network, the trained segmentation network can be used to obtain feature information of a to-be-processed image, and the attention map can be obtained according to the to-be-processed image and the feature information.

[0253] The feature information can refer to a feature map of the to-be-processed image obtained by the segmentation network, or the feature information can also refer to the parameters of a convolution layer of the segmentation network. The feature information of the trained segmentation network can be shared to reduce the calculation amount of the attention map extraction network.

[0254] For example, the attention map can be obtained according to the to-be-processed image and the feature information, which can include obtaining an attention map extraction network according to the feature information, inputting the to-be-processed image into the attention map extraction network, and obtaining the attention map.

[0255] In the embodiments of the present application, the network structure used for image processing can be as shown in Figure 13As shown, the network can be composed of a segmentation network, a matting network, an attention module (i.e., can be an attention map extraction network), and a fusion module. Among them, the segmentation network can adopt a conventional encoding-decoding structure, which is used to obtain segmentation features; the attention module (i.e., the attention map extraction network) can utilize the segmentation feature information output by the segmentation network and the RGB image information to be processed to obtain the attention map of the edge region such as hair; the matting network can include a matting encoding-decoding module and a matting optimization module to obtain the matting feature of the hair; and the fusion module can be used to fuse the segmentation feature and the matting feature through the attention map to obtain the final refined segmentation mask (Alpha).

[0256] Through the image processing method of the embodiment of the present application, a parallel refined segmentation network architecture is proposed for the refined segmentation task in the field of computer vision. The matting network can not depend on the segmentation result, and can directly obtain an accurate matting result of the refined feature (for example, the hair region of a portrait) according to the image to be processed. While maintaining the accuracy, the entire refined segmentation network training can only partially depend on the manually annotated refined segmentation mask, for example, the matting network can be trained by the manually annotated refined segmentation mask in the parallel network architecture, and the segmentation network can be trained by the ordinary segmentation data, so as to greatly reduce the data amount of the manually annotated refined segmentation mask; in addition, after optimizing the matting network, the parallel refined segmentation network architecture can greatly reduce the network running time and improve the robustness of the refined portrait segmentation result.

[0257] It should be noted that, Figure 7 The examples are only to help those skilled in the art to understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the examples given Figure 7 The examples are only to help those skilled in the art to understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the examples given

[0258] Figure 8 is a schematic flowchart of the image processing method provided by the embodiment of the present application. Among them, the method 600 includes steps 601 to 605, which will be described in detail below.

[0259] Step 601, obtaining an image to be processed.

[0260] Exemplarily, the image to be processed can contain a foreground region and a background region of a target object. For example, the image to be processed can be an RGB image.

[0261] Exemplarily, the image to be processed can be an image captured by the camera of the electronic device, or the image to be processed can also be an image obtained from the electronic device (for example, an image stored in the photo album of the electronic device, or a picture obtained from the cloud by the electronic device).

[0262] It should be noted that the image to be processed can be an image that needs image segmentation processing, and the application does not limit the source of the image to be processed.

[0263] It should be understood that in the embodiments of the application, the target object can be an image with irregular edge regions. For example, the target object can be a portrait, an animal, a stuffed toy, or a collar of clothes.

[0264] Step 602, inputting the image to be processed into a segmentation network to obtain a first segmentation result of the foreground region and the background region.

[0265] The first segmentation result can be a coarse segmentation result of the foreground region and the background region.

[0266] Optionally, the segmentation network can refer to a segmentation network that obtains a three-class mask, that is, a segmentation network that divides the image to be processed into a foreground region, a background region, and an uncertain region.

[0267] Optionally, the segmentation network can also be a segmentation network that performs preliminary segmentation on the image to be processed to obtain a preliminary segmentation result of the foreground region and the background region.

[0268] For example, as shown in FIG. 1, the segmentation network can adopt a conventional encoding-decoding structure, and the segmentation network can obtain a coarse segmentation result of the image to be processed by processing the image to be processed, which can also be referred to as a segmentation feature of the image to be processed. Figure 9

[0269] Step 603, inputting the image to be processed into a matting network to obtain a matting result.

[0270] The image to be processed can obtain a matting result of a fine feature of the target object by processing the image to be processed through the matting network.

[0271] It should be noted that steps 602 and 603 are used to illustrate that the image to be processed is first input into the segmentation network and then input into the matting network. The image to be processed can also be input into the segmentation network and the matting network at the same time, or the image to be processed can also be first input into the matting network and then input into the segmentation network.

[0272] ​In the embodiments of the present application, the segmentation network and the matting network can process the to-be-processed image in parallel, that is, the to-be-processed image can be input into the segmentation network and the matting network in parallel, and the order of input into the segmentation network and the matting network is not limited.

[0273] Exemplarily, the target object contained in the to-be-processed image can be a human portrait, and the fine feature can be a hair of the human portrait.

[0274] Exemplarily, the target object contained in the to-be-processed image can be an animal, and the fine feature can be a hair of the animal.

[0275] Exemplarily, the target object contained in the to-be-processed image can be a stuffed toy, and the fine feature can be a hair of the stuffed toy.

[0276] In the traditional framework, the matting network is based on the coarse segmentation result to perform matting on the to-be-processed image, and the coarse segmentation result can provide the matting network with general semantic information. When the matting network is only input with the to-be-processed image without the coarse segmentation result, the matting network loses the semantic information, which reduces the accuracy of matting. Therefore, in the embodiments of the present application, the parameters of the matting network can be trained by the first loss function and the second loss function, so that the matting network can generally learn the difference between the foreground region and the background region of the target object.

[0277] Optionally, the matting network can be trained based on the first loss function and the second loss function, wherein the first loss function can be used to distinguish the foreground region and the background region, and the second loss function can be used to obtain the matting result of the fine feature of the target object of the to-be-processed image.

[0278] For example, the first loss function can be a global loss function, which can be used to represent the loss value between the predicted matting result of the sample image obtained by the matting network and the labeled semantic segmentation feature of the sample image; the second loss function can be a local loss function, which can be used to represent the loss value between the edge region feature in the predicted matting result and the labeled edge region feature of the sample image, and the edge region refers to the edge of the foreground region of the target object in the sample image.

[0279] In the embodiments of the present application, the matting network is to learn more detailed information in the image, and at the same time, the importance of the position information of the details in the image can be emphasized. Therefore, in the design of the matting network, a symmetrical encoding-decoding structure can not be used. In order to make the matting network learn more detailed features in the image, a relatively deep neural network layer can not be used in the design of the encoding module; in order to pay more attention to the position information of the detailed features in the image, a relatively deep neural network layer can be used in the design of the decoding module.

[0280] It should be understood that when the number of neural network layers in the decoding module is small, the image size can be recovered directly by upsampling after some pixels are recovered, and in this case, the positions of some pixels can be inaccurate.

[0281] Optionally, the matting network can include an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than the number of neural network layers of the encoding module.

[0282] In step 604, the first segmentation result and the matting result are fused.

[0283] Further, in order to make the coarse segmentation result and the matting result pay more attention to the fine features of the target object when they are fused, that is, pay more attention to the detailed information in the to-be-processed image, optionally, an attention map for the edge region of the foreground region of the to-be-processed image can be obtained according to the to-be-processed image, and the coarse segmentation result and the matting result are fused according to the attention map.

[0284] In a possible implementation, the fusion can mean that the first segmentation result and the matting result are convoluted according to the attention map.

[0285] For example, the fine segmentation result of the foreground region and the background region in the to-be-processed image is obtained by convoluting the attention map, the coarse segmentation result and the matting result through the fusion network. For example, the attention map for the edge region of the foreground region of the to-be-processed image can be obtained according to the to-be-processed image, and the attention map extraction network for obtaining the attention map can be obtained by training the neural network through the edge region of the foreground region of the sample image manually labeled.

[0286] In a possible implementation, the fusion can mean that the values of the corresponding pixels in the first segmentation result and the matting result are multiplied according to the attention map.

[0287] Optionally, the attention map for the edge region of the foreground region of the to-be-processed image can be obtained according to the to-be-processed image, and the attention map extraction network for obtaining the attention map can be obtained by training the neural network through the edge region of the foreground region of the sample image manually labeled.

[0288] Further, in order to reduce the calculation amount of the attention map extraction network for obtaining the attention map, that is, to design a simple network structure of the attention map extraction network, the feature information obtained by processing the to-be-processed image through the trained segmentation network can be obtained, wherein the feature information can mean the feature map of the to-be-processed image obtained by the segmentation network, or the feature information can also mean the parameters of the convolution layer of the segmentation network, and the calculation amount of the attention map extraction network can be reduced by sharing the features of the trained segmentation network.

[0289] For example, the segmentation network is used to obtain feature information of the to-be-processed image; and the attention map is obtained according to the to-be-processed image and the feature information.

[0290] For example, Figure 12 As shown in the figure, the coarse segmentation result and the matting result are fused according to the attention map. The fusion module can be a neural network, and the coarse segmentation result (i.e., segmentation feature) and the matting result (i.e., matting feature) can be fused through the attention map by the fusion module.

[0291] In step 605, a second segmentation result is obtained, i.e., a fine segmentation mask of the to-be-processed image is obtained.

[0292] The second segmentation result can be a fine segmentation result of the to-be-processed image, such as a fine segmentation mask (Alpha). For example, when the target object contained in the to-be-processed image is a portrait, the fine segmentation result can be a pixel-level fine segmentation result of the portrait, and the value range is 0 to 1.

[0293] By using the image processing method, a parallel fine segmentation network architecture is proposed for the fine segmentation task in the field of computer vision. The matting network can not depend on the segmentation result, and can directly obtain an accurate matting result of a fine feature (for example, a hair region of a portrait) according to the to-be-processed image. While maintaining the accuracy, the entire fine segmentation network training can only partially depend on the manually annotated fine segmentation mask. For example, the matting network can be trained by using the manually annotated fine segmentation mask, and the segmentation network can be trained by using normal segmentation data, so that the amount of data of the manually annotated fine segmentation mask can be greatly reduced. In addition, after the matting network is optimized, the parallel fine segmentation network architecture can greatly reduce the network running time and improve the robustness of the fine portrait segmentation result.

[0294] It should be noted that Figure 8 The examples of the above-mentioned embodiments of the present application are only used to help those skilled in the art to understand the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios exemplified. Those skilled in the art can make various equivalent modifications or changes to the examples of the above-mentioned embodiments of the present application, and such modifications or changes also fall within the scope of the embodiments of the present application. Figure 8 The examples of the above-mentioned embodiments of the present application are only used to help those skilled in the art to understand the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios exemplified. Those skilled in the art can make various equivalent modifications or changes to the examples of the above-mentioned embodiments of the present application, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0295] It should also be understood that the above-mentioned extensions, limitations, explanations and descriptions of the image processing method of the present application are also applicable to the image processing method shown in Figure 7 The above-mentioned extensions, limitations, explanations and descriptions of the image processing method of the present application are also applicable to the image processing method shown in Figure 8 The above-mentioned extensions, limitations, explanations and descriptions of the image processing method of the present application are also applicable to the image processing method shown in Figure 8 The above-mentioned extensions, limitations, explanations and descriptions of the image processing method of the present application are also applicable to the image processing method shown in

[0296] Figure 14 is an effect diagram of an image processing method provided by an embodiment of the present application. As shown in Figure 14 the image processing effect of the image processing method in the present application is compared with the image processing effect of the manual annotation fine segmentation mask (ground-truth Alpha) image by using the automatic portrait matting algorithm (semantic human matting, SHM) and the deep image matting algorithm (deep image matting, DIM) in the prior art on the to-be-processed image, the processing effect of the image processing method of the present application on the integrity and hollow of hair prediction even exceeds the manual annotation fine segmentation mask.

[0297] Table 1

[0298] Mode MSE GradErr Time (GPU / ms) SHM 0.0124 0.1397 201 DIM 0.0091 0.0912 150 The present application 0.0090 0.0942 65

[0299] As shown in Table 1, the evaluation results of the parallel fine segmentation network architecture proposed by the present application according to the human matting dataset are shown, wherein the evaluation results include mean-square error (MSE), gradient error (GradErr) and running time test data. From the test results in Table 1, it can be seen that the effect of the image processing method proposed by the present application is better than that of the traditional framework algorithm (for example, the automatic portrait matting algorithm), and the evaluation index of the image effect by using the image processing method in the present embodiment is close to the effect of directly using the manual annotation fine segmentation mask.

[0300] For example, assuming that the errors of n measurement values are ε1, ε2...ε n , the mean-square error of the measurement values can be obtained by the following equation:

[0301]

[0302] , wherein σ represents the mean-square error, which is the square root of the average value of the sum of squares of the measurement errors; the above-mentioned gradient error can be the mean-square error of the image gradient.

[0303] Compared with the existing portrait fine segmentation network framework, the matting network in the network framework proposed in the embodiments of the present application can not depend on the coarse segmentation network, realizing parallel running of the segmentation network and the matting, thereby improving the efficiency of image processing. For example, it can be seen from the running time test results in Table 1 that the running time of the image processing method of the embodiments of the present application is reduced to 32.34% of the time used by the traditional framework. In addition, since the network training of the traditional serial algorithm needs to completely depend on the data labeling of the fine segmentation mask, the parallel fine segmentation network framework proposed in the embodiments of the present application can train the matting network and the segmentation network step by step, and the experimental measurement shows that the training data can be reduced from 35311 (for example, for the SHM algorithm) fine segmentation mask data labeling to 1089, and the rest only needs to use ordinary segmentation labeling data.

[0304] In the parallel fine segmentation network framework proposed in the embodiments of the present application, the matting network can directly process the image to be processed without depending on the coarse segmentation result. In the training process of the fine segmentation network, the fine segmentation data labeled by artificial labeling can be partially depended on, that is, the fine segmentation data labeled by artificial labeling can be used to train the matting network during training, and the segmentation network is trained by using ordinary portrait segmentation data, thereby the amount of data labeled by artificial labeling can be greatly reduced; secondly, the parallel fine segmentation network framework can greatly reduce the network running time and improve the robustness of the fine segmentation result.

[0305] It should be understood that the above examples are intended to help those skilled in the art to understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific values or specific scenarios exemplified. Those skilled in the art can obviously make various equivalent modifications or changes according to the above examples given, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0306] The embodiments of the present application are described in detail above Figures 1 to 14 , the device embodiments of the present application will be described in detail below Figure 15 and Figure 16 . It should be understood that the image processing device in the embodiments of the present application can perform various image processing methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0307] Figure 15 is a schematic block diagram of the image processing device 700 provided by the embodiments of the present application. It should be understood that the image processing device 700 can perform each step in the method of Figure 7 or Figure 8 , and to avoid repetition, it will not be described in detail here. The image processing device 700 comprises an acquisition unit 710 and a processing unit 720.

[0308] The obtaining unit 710 is configured to obtain a to-be-processed image; the processing unit 720 is configured to input the to-be-processed image into a segmentation network and a matting network in parallel, obtain a first segmentation result of a foreground region and a background region output by the segmentation network, and obtain a matting result output by the matting network; and the processing unit 720 is further configured to fuse the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region.

[0309] It should be noted that the first segmentation result can be a coarse segmentation result, and the second segmentation result can be a fine segmentation result, where the accuracy of the second segmentation result is higher than that of the first segmentation result.

[0310] Optionally, as an embodiment, the to-be-processed image can include a foreground region where a target object is located and a background region, and the processing unit 720 can be configured to input the to-be-processed image into the segmentation network and the matting network in parallel, obtain a coarse segmentation result of the foreground region and the background region output by the segmentation network, and obtain a matting result of fine features of the target object output by the matting network; and the processing unit 720 is further configured to fuse the coarse segmentation result and the matting result to obtain a fine segmentation result of the foreground region and the background region.

[0311] Optionally, as an embodiment, the processing unit 720 is further configured to obtain an attention map of an edge region of the foreground region according to the to-be-processed image; and the processing unit 720 is further configured to fuse the first segmentation result and the matting result according to the attention map.

[0312] Optionally, as an embodiment, the processing unit 720 is specifically configured to perform convolution processing on the first segmentation result and the matting result according to the attention map.

[0313] Optionally, as an embodiment, the processing unit 720 is specifically configured to obtain feature information obtained by the segmentation network processing the to-be-processed image; and the processing unit 720 is further configured to obtain the attention map according to the to-be-processed image and the feature information.

[0314] Optionally, as an embodiment, the processing unit 720 is specifically configured to obtain an attention map extraction network according to the feature information; and the processing unit 720 is further configured to input the to-be-processed image into the attention map extraction network to obtain the attention map.

[0315] Optionally, as an embodiment, the matting network includes an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than the number of neural network layers of the encoding module.

[0316] Optionally, as an embodiment, the matting network is trained based on a first loss function and a second loss function, wherein the first loss function is used to distinguish the foreground region and the background region, and the second loss function is used to obtain a matting result of a fine feature of the target object, and the foreground region contains the target object.

[0317] Optionally, as an embodiment, the target object is a portrait, and the fine feature is a hair of the portrait.

[0318] It should be understood that the image processing apparatus 700 herein is embodied in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and no specific limitation is made thereto.

[0319] For example, the "unit" can be a software program, a hardware circuit, or a combination of both, which realizes the above functions. The hardware circuit can include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, and other suitable components for supporting the described functions.

[0320] Therefore, the units of each example described in the embodiments of the present application can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0321] Figure 16 FIG. 1 is a schematic diagram of a hardware structure of an image processing apparatus according to an embodiment of the present application. Figure 16 The image processing apparatus 800 shown includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, the processor 802, and the communication interface 803 are communicatively connected to each other through the bus 804.

[0322] The memory 801 can be a read-only memory (ROM), a static storage device, and a random access memory (RAM). The memory 801 can store programs, and when the programs stored in the memory 801 are executed by the processor 802, the processor 802 and the communication interface 803 are used to execute each step of the image processing method according to the embodiments of the present application, for example, the programs can be executed to Figure 7 and Figure 8The image processing method shown in the embodiment of the present application comprises the following steps.

[0323] The processor 802 can be a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, for executing relevant programs to implement the functions of the image processing device shown in the embodiment of the present application. Figure 15 The processor 802 can be a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, for executing relevant programs to implement the functions of the image processing device shown in the embodiment of the present application.

[0324] The processor 802 can be a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, for executing relevant programs to implement the functions of the image processing device shown in the embodiment of the present application.

[0325] The processor 802 can be a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, for executing relevant programs to implement the functions of the image processing device shown in the embodiment of the present application.

[0326] For example, the processor 802 can correspond to the processing unit 720 in the image processing device shown in the embodiment of the present application. Figure 15 For example, the processor 802 can correspond to the processing unit 720 in the image processing device shown in the embodiment of the present application.

[0327] The communication interface 803 uses a transceiver device such as, but not limited to, a transceiver, to implement the communication between the device 800 and other devices or communication networks.

[0328] For example, the communication interface 803 can correspond to the acquisition unit 710 in the image processing device shown in the embodiment of the present application. Figure 15 For example, the communication interface 803 can correspond to the acquisition unit 710 in the image processing device shown in the embodiment of the present application.

[0329] The bus 804 can include a path for transmitting information between the components of the device 800 (e.g., the memory 801, the processor 802, the communication interface 803).

[0330] It should be noted that although the image processing apparatus 800 is only shown as including a memory, a processor, and a communication interface, in actual implementation, those skilled in the art should understand that the image processing apparatus 800 can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the image processing apparatus 800 can also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the image processing apparatus 800 can also only include devices necessary for implementing the embodiments of the present application, and does not necessarily include all the devices shown in the image processing apparatus 800. Figure 16

[0331] It should be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0332] It should also be understood that the memory in the embodiments of the present application can include read-only memory and random access memory, and provide instructions and data for the processor. A portion of the processor can also include non-volatile random access memory. For example, the processor can also store device type information.

[0333] It should be understood that the term "and / or" in the present application is only used to describe an associated relationship between associated objects, and can represent three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0334] It should be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0335] ​Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0336] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0337] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0338] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0339] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit.

[0340] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0341] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a to-be-processed image, the to-be-processed image comprising a foreground region in which a target object is located and a background region; inputting the to-be-processed image into a segmentation network to obtain a first segmentation result of the foreground region and the background region output by the segmentation network; inputting the to-be-processed image into a matting network to obtain a matting result of fine features of the target object output by the matting network; fusing the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region; the accuracy of the second segmentation result is greater than that of the first segmentation result; obtaining feature information obtained by processing the to-be-processed image by using the segmentation network; obtaining an attention map according to the to-be-processed image and the feature information; wherein the feature information is a feature map of the to-be-processed image obtained based on the segmentation network, or the feature information is a parameter of a convolution layer of the segmentation network; performing convolution processing on the first segmentation result and the matting result according to the attention map.

2. The image processing method of claim 1, wherein, The image processing method further comprises: obtaining an attention map of an edge region of the foreground region according to the to-be-processed image; the fusing of the first segmentation result and the matting result comprises: fusing the first segmentation result and the matting result according to the attention map.

3. The image processing method of claim 1, wherein, The obtaining of the attention map according to the to-be-processed image and the feature information comprises: obtaining an attention map extraction network according to the feature information; inputting the to-be-processed image into the attention map extraction network to obtain the attention map.

4. The image processing method of claim 1 or 2, wherein The matting network comprises an encoding module and a decoding module, and the number of neural network layers of the decoding module is greater than that of the encoding module.

5. The image processing method according to claim 1 or 2, wherein The matting network is trained based on a first loss function and a second loss function, wherein the first loss function is used to distinguish the foreground region and the background region, the second loss function is used to obtain the matting result of the fine features of the target object, and the foreground region contains the target object.

6. The image processing method of claim 5, wherein, The target object is a portrait, and the fine features are hair of the portrait.

7. An image processing apparatus characterized by comprising: The method comprises: an obtaining unit configured to obtain a to-be-processed image, the to-be-processed image comprising a foreground region in which a target object is located and a background region; a processing unit configured to input the to-be-processed image into a segmentation network to obtain a first segmentation result of the foreground region and the background region output by the segmentation network; inputting the to-be-processed image into a matting network to obtain a matting result of fine features of the target object output by the matting network; fusing the first segmentation result and the matting result to obtain a second segmentation result of the foreground region and the background region; the accuracy of the second segmentation result is greater than that of the first segmentation result; obtaining feature information obtained by processing the to-be-processed image by using the segmentation network; obtaining an attention map according to the to-be-processed image and the feature information; The feature information is a feature map of the to-be-processed image obtained based on the segmentation network, or the feature information is a parameter of a convolution layer of the segmentation network. The first segmentation result and the matting result are convoluted according to the attention map.

8. The image processing apparatus of claim 7, wherein The processing unit is further configured to: obtain an attention map for an edge region of the foreground region according to the to-be-processed image; fuse the first segmentation result and the matting result according to the attention map.

9. The image processing apparatus of claim 7, wherein The processing unit is specifically configured to: obtain an attention map extraction network according to the feature information; input the to-be-processed image into the attention map extraction network to obtain the attention map.

10. The image processing apparatus of claim 7 or 8, wherein The matting network comprises an encoding module and a decoding module, and a number of neural network layers of the decoding module is greater than that of the encoding module.

11. The image processing apparatus of claim 7 or 8, wherein The matting network is trained based on a first loss function and a second loss function, the first loss function is used to distinguish the foreground region and the background region, the second loss function is used to obtain a matting result of fine features of the target object, and the foreground region contains the target object.

12. The image processing apparatus of claim 11, wherein The target object is a portrait, and the fine features are hair of the portrait.

13. An image processing apparatus characterized by comprising: Comprise: a memory for storing a program; a processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is configured to execute the image processing method in any one of claims 1 to 6.

14. A computer storage medium, characterized in that The computer storage medium stores program code, and the program code comprises instructions for executing steps in the image processing method in any one of claims 1 to 6. The computer storage medium stores program code, and the program code comprises instructions for executing steps in the image processing method in any one of claims 1 to 6.

Images