Image processing method, device and electronic equipment

Abstract

The present disclosure relates to an image processing method, device and electronic device. The method comprises: inputting an image to be processed into a person segmentation module for calculation, and obtaining a person image contained in the image to be processed output by the person segmentation module; inputting the image to be processed into a hair segmentation module for calculation, and obtaining a hair image with a refined effect contained in the image to be processed output by the hair segmentation module; and fusing the person image with the hair image with a refined effect to obtain a person image with a refined hair effect. By using the method, a person image with clear hair boundaries and sharp hair edges can be segmented from the image to be processed, and the effect of fine hair segmentation at the hair level can be achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] The human segmentation technology is a specific type of semantic segmentation technology. When implementing it, each pixel in the image needs to be classified into two categories to determine the attribute category of each pixel, which generally includes human attributes and background attributes. Then, based on the attribute category, the pixels belonging to the human attributes are extracted to construct an image containing only human pixels. In this way, the purpose of segmenting the human image from the image is achieved.

[0003] In the existing human segmentation technology, a human segmentation module can usually be used for human segmentation. However, due to the limitations of the learning ability of the existing human segmentation module and the fineness of sample annotation, although human images can be obtained by using the human segmentation module, the boundary of the hair area in such human images is not clear enough, the edge of the hair is not sharp enough, and there is a strong feathering effect, and it is impossible to segment a human image with fine hair effect at the hair level. Summary of the invention

[0004] The present disclosure provides an image processing method, device and electronic device to at least solve the problem in the related art that a person image with a fine hair effect cannot be segmented from an original image. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Inputting the image to be processed into a person segmentation module for calculation, and obtaining a person image contained in the image to be processed output by the person segmentation module;

[0007] Inputting the image to be processed into a hair segmentation module for calculation, and obtaining a hair image with a refined effect contained in the image to be processed output by the hair segmentation module;

[0008] The character image is merged with the hair image with the refined effect to obtain a character image including the refined hair effect.

[0009] Optionally, the method further includes:

[0010] The following steps are performed to perform multi-task learning training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set used for training are labeled with the person area and the hair area:

[0011] Inputting a sample image in the image sample set into the person segmentation module, taking the region where the person is marked in the sample image as a supervisory signal, and calculating a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image;

[0012] Inputting the person image output by the person segmentation module into the hair segmentation module, taking the area where the hair is marked in the sample image as a supervisory signal, and calculating the second loss value of the second loss function of the hair segmentation module when calculating the fine hair effect segmented from the person image;

[0013] The first loss value and the second loss value are weightedly calculated to obtain a loss value of the loss function of the multi-task learning training, and the module parameters of the person segmentation module and the hair segmentation module are adjusted respectively based on the loss value.

[0014] Optionally, when calculating the fine hair effect segmented from the person image, the second loss value of the second loss function of the hair segmentation module includes:

[0015] Calculate the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image;

[0016] The hair edge area and the character image output by the character segmentation module are calculated to obtain a second loss value of the second loss function of the hair segmentation module.

[0017] Optionally, calculating the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge region corresponding to the sample image to obtain the hair edge region of the sample image includes:

[0018] Taking the image center points of the fine hair effect and the edge area of ​​the character as references, calculating the intersection of the fine hair effect and the edge area of ​​the character;

[0019] The intersection area is determined as the hair edge area of ​​the sample image.

[0020] Optionally, the character edge area is calculated in the following way:

[0021] The input sample image is dilated and eroded respectively to obtain an enlarged dilated sample image and a reduced eroded sample image;

[0022] Negate pixel values ​​of the dilated sample image or the eroded sample image;

[0023] Taking the image center points of the dilated sample image and the eroded sample image as references, calculating the intersection of the dilated sample image and the eroded sample image;

[0024] The intersection area is determined as the character edge area.

[0025] Optionally, the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module includes:

[0026] The hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

[0027] Optionally, the first loss value and the second loss value are weightedly calculated by the following formula to obtain the loss value of the loss function of the multi-task learning training:

[0028] Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin;

[0029] Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0031] A first calculation unit is configured to input the image to be processed into a person segmentation module for calculation, and obtain a person image contained in the image to be processed output by the person segmentation module;

[0032] A second calculation unit is configured to input the image to be processed into a hair segmentation module for calculation, and obtain a hair image with a refined effect contained in the image to be processed output by the hair segmentation module;

[0033] The image fusion unit is configured to fuse the character image with the hair image with the refined effect to obtain a character image including a refined hair effect.

[0034] Optionally, the device further includes:

[0035] The module training unit is configured to execute the following first training unit, second training unit and third training unit to perform multi-task learning training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set for training are labeled with the person area and the hair area:

[0036] The first training unit is configured to input the sample image in the image sample set into the person segmentation module, and use the region where the marked person is located in the sample image as a supervisory signal to calculate a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image;

[0037] The second training unit is configured to input the person image output by the person segmentation module into the hair segmentation module, and use the region where the hair is marked in the sample image as a supervisory signal to calculate a second loss value of a second loss function of the hair segmentation module when a fine hair effect is segmented from the person image;

[0038] The third training unit is configured to perform weighted calculation on the first loss value and the second loss value to obtain a loss value of the loss function of the multi-task learning training, and adjust the module parameters of the person segmentation module and the hair segmentation module respectively based on the loss value.

[0039] Optionally, the second training unit is further configured to execute the first computing subunit and the second computing subunit;

[0040] The first calculation subunit is configured to calculate the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image;

[0041] The second calculation subunit is configured to calculate the hair edge area and the character image output by the character segmentation module to obtain a second loss value of the second loss function of the hair segmentation module.

[0042] Optionally, the first calculation subunit is further configured to calculate the intersection of the fine hair effect and the character edge area based on the image center point of the fine hair effect and the character edge area, and determine the area of ​​the intersection as the hair edge area of ​​the sample image.

[0043] Optionally, the first computing subunit is further configured to perform dilation and erosion processing on the input sample image respectively to obtain an enlarged dilated sample image and a reduced eroded sample image; invert the pixel values ​​of the dilated sample image or the eroded sample image; calculate the intersection of the dilated sample image and the eroded sample image based on the image center points of the dilated sample image and the eroded sample image, and determine the area of ​​the intersection as the edge area of ​​the character.

[0044] Optionally, the second training unit is further configured so that the hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

[0045] Optionally, the first loss value and the second loss value are weightedly calculated by the following formula to obtain the loss value of the loss function of the multi-task learning training:

[0046] Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin;

[0047] Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

[0048] According to a third aspect of an embodiment of the present disclosure, an image processing electronic device includes:

[0049] processor;

[0050] a memory for storing instructions executable by the processor;

[0051] The processor is configured to execute the instructions to implement any of the aforementioned image processing methods.

[0052] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an image processing electronic device, the image processing electronic device is enabled to perform any of the aforementioned image processing methods.

[0053] According to a seventh aspect of an embodiment of the present disclosure, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the image processing method described in any one of the above is implemented.

[0054] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0055] Under the premise of increasing less calculation amount, the hair segmentation effect during character segmentation is optimized. On the basis of the character segmentation module, a hair segmentation module with a hair segmentation branch is added. The hair segmentation module segments a hair image containing fine hair effects from the image to be processed, and then the hair image containing fine hair effects is merged with the character image segmented from the image to be processed by the original character segmentation module, so that the character image containing fine hair effects can be obtained, thereby achieving the effect of fine hair segmentation.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0058] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0059] Figure 2 is a schematic diagram of various intermediate images in a training process of a machine learning module according to an exemplary embodiment.

[0060] Figure 3 The figure is a schematic diagram showing the generation of a character edge region according to an exemplary embodiment.

[0061] Figure 4 The figure is a schematic diagram showing the effect of image processing according to an exemplary embodiment.

[0062] Figure 5 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0063] Figure 6 The diagram is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] Figure 1 The present invention is a flowchart of an image processing method according to an exemplary embodiment. The method can be applied to a client or a server, and includes the following steps:

[0067] Step 110: input the image to be processed into a person segmentation module for calculation, and obtain a person image contained in the image to be processed output by the person segmentation module.

[0068] Step 120: input the image to be processed into a hair segmentation module for calculation, and obtain a hair image with a refined effect contained in the image to be processed output by the hair segmentation module.

[0069] Step 130: Merge the character image with the hair image with the refined effect to obtain a character image including the refined hair effect.

[0070] The client may refer to a client device on hardware, such as a desktop computer, a laptop computer, a tablet computer, a smart phone, a handheld computer, a personal digital assistant ("PDA"), or any other wired or wireless processor-driven device.

[0071] The client may refer to an application client on the software, such as an image processing APP or a third-party APP with built-in image processing function. The image processing APP may include a selfie APP or a beauty APP.

[0072] The client may refer to a client that combines software and hardware, such as a smart phone with an image processing APP installed.

[0073] The server side may be a server, a server cluster or a cloud platform constructed by a server cluster corresponding to the client side, for example, an operation server corresponding to an image processing APP.

[0074] When the embodiments of this specification are applied to the client, it means that the client can perform image processing locally. When the embodiments of this specification are applied to the server, it can be considered that after the client sends the image to be processed to the server, the server performs image processing and returns the processing result to the client.

[0075] The following takes the client's local image processing as an example for explanation.

[0076] The client first needs to determine the image to be processed that contains a person. The image to be processed can be an image containing a person taken by a user through a camera device (such as a camera), or an image containing a person downloaded or received from an album or a third-party application.

[0077] The client is provided with a machine learning module (in this specification, the module may also be referred to as a model) for segmenting a person image containing a fine hair effect from an image to be processed; wherein the machine learning module includes a person segmentation module for segmenting a person image from the image to be processed, and a hair segmentation module for segmenting a fine hair effect from the image to be processed.

[0078] In machine learning technology, if you want the machine learning module to accurately segment human images with fine hair effects from the image to be processed, you first need to train the machine learning module. Through continuous training, you can improve the accuracy of the cluster learning module to meet the actual business needs.

[0079] The following introduces the training process of the machine learning module, which can also be understood as the training process of the person segmentation module and the hair segmentation module.

[0080] Perform the following steps to perform multi-task learning (MTL) training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set for training are labeled with the person area and the hair area, and the loss function includes a first loss function corresponding to the person area and a second loss function corresponding to the hair area;

[0081] Step A1: inputting a sample image in the image sample set into the person segmentation module, taking the region where the person is marked in the sample image as a supervisory signal, and calculating a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image;

[0082] Step A2: inputting the person image output by the person segmentation module into the hair segmentation module, taking the hair region marked in the sample image as a supervisory signal, and calculating the second loss value of the second loss function of the hair segmentation module when the fine hair effect segmented from the person image is calculated;

[0083] Step A3: performing weighted calculation on the first loss value and the second loss value to obtain the loss value of the loss function of the multi-task learning training, and adjusting the module parameters of the person segmentation module and the hair segmentation module respectively based on the loss value.

[0084] Among them, the character segmentation module and the hair segmentation module can adopt existing machine learning models, such as deep convolutional neural network DCNN, recurrent neural network RNN, etc.

[0085] Generally, during module training, the sample images in the image sample set used for training are usually labeled with the area where the person is and the area where the hair is, and the labeled area where the person is and the area where the hair is can be called sample labels. The sample labels can be used as supervisory signals of the loss function to calculate the loss value of the loss function. For example, the area where the person is labeled in the sample image is used as the supervisory signal of the first loss function to calculate the first loss value of the first loss function of the person segmentation module when the person image is segmented from the sample image; the area where the hair is labeled in the sample image is used as the supervisory signal of the second loss function to calculate the second loss value of the second loss function of the hair segmentation module when the fine hair effect segmented from the person image is calculated.

[0086] Steps A1-A3 are iteratively executed to continuously adjust the parameters of the module until the accuracy of the person image and hair image segmented by the module and the marked area in the sample image meets the business requirements, then the module training is terminated. At this time, the loss value in the last adjusted parameter result is the optimal loss value of the loss function of the multi-task learning training.

[0087] By applying the above embodiment, a hair segmentation module is added to the existing person segmentation module, and the optimal loss value of the loss function in each module is obtained through multi-task learning training; when the optimal loss value is reached, it means that the person image and hair image obtained by segmenting the image to be processed by the person segmentation module and the hair segmentation module are the most accurate, which can also improve the accuracy of the person image containing a fine hair effect obtained by the final fusion.

[0088] In one embodiment, the character segmentation module may be composed of an encoder Encode and a decoder Decoder. The encoder is mainly used to extract image features in the input image (generally including shallow surface features and deep semantic features); then the deep convolutional neural network DCNN in the decoder is used to predict the extracted image features to predict the attribute features of each pixel in the input image. Since the character segmentation network for character segmentation is a commonly used technology in the industry, it will not be described in detail in this embodiment.

[0089] The following combination Figure 2 Schematic diagram of each image during the training process of the machine learning module shown ( Figure 2 Desensitization of faces in the video).

[0090] For the sample image in the leftmost input image sample set, execute step A1, input the sample image into the person segmentation module, and calculate the first loss value of the first loss function of the person segmentation module when the person image is segmented from the sample image based on the area where the marked person in the sample image is located.

[0091] Wherein, when calculating the person image segmented from the sample image, the first loss value of the first loss function of the person segmentation module may include:

[0092] The person image segmented from the sample image by the person segmentation module and the sample image with the person area marked ( Figure 2 ), calculate the first loss value of the first loss function of the character segmentation module.

[0093] After introducing the character segmentation module, the hair segmentation module for hair segmentation is further introduced. As the name implies, the purpose of hair segmentation is to separate the hair of a character from the character. The hair segmentation module can adopt the same architecture as the aforementioned character segmentation module.

[0094] Similarly, for the sample image in the leftmost input image sample set, execute step A2, input the person image output by the person segmentation module into the hair segmentation module, and calculate the second loss value of the second loss function of the hair segmentation module when the fine hair effect segmented from the person image is calculated.

[0095] In an exemplary embodiment, the step A2 may further include:

[0096] Step A21, calculating the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image.

[0097] Step A22, calculating the hair edge area and the character image output by the character segmentation module to obtain a second loss value of the second loss function of the hair segmentation module.

[0098] By applying this example, in the multi-task learning training process, by calculating the second loss value of the second loss function of the hair segmentation module and combining it with the first loss value of the first loss function of the aforementioned person segmentation module, the loss value of the loss function of the multi-task learning can be determined, thereby providing a basis for adjusting the module parameters of the hair segmentation module and the person segmentation module. Through continuous iterative training, the optimal module parameters of the hair segmentation module and the person segmentation module can be finally adjusted.

[0099] like Figure 2 As shown, the hair edge area can be obtained by calculating the fine hair effect and the character edge area.

[0100] In an exemplary embodiment, the fine hair effect segmented by the hair segmentation module from the person image output by the person segmentation module includes:

[0101] The hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

[0102] Among them, the above-mentioned guided filtering can adopt filtering algorithms commonly used in the industry such as bilateral filtering algorithm, guided filtering algorithm, etc.

[0103] By applying this example, edge-preserving filtering is performed on the hair prediction results through guided filtering, which can not only reduce the filtering area and improve the processing speed, but also reduce the noise introduced by filtering other parts of the character and improve the accuracy.

[0104] In an exemplary embodiment, the character edge area can be calculated in the following way:

[0105] Step B1, performing dilation and erosion processing on the input sample image respectively to obtain an enlarged dilated sample image and a reduced eroded sample image;

[0106] The labeled sample image is expanded to obtain an enlarged expanded sample image, and the sample image is corroded to obtain a reduced corroded sample image, wherein the size parameters of the expansion and corrosion can be random values ​​within a certain range (such as less than or equal to 5); the random value is randomly obtained in each iterative training.

[0107] Step B2, negating the pixel values ​​of the dilated sample image or the eroded sample image;

[0108] The following references Figure 3 The schematic diagram shown is used to further introduce the principle of edge area generation. Figure 3 The pixel value of the black area in the image is 1, and the pixel value of the white area is 0. After the eroded image is inverted, its edge is white (the boundary is marked by a dotted line in the figure, which is actually invisible because the pixel value is 0). Taking the center point of the dilated image and the eroded image as the reference, the two images are superimposed to obtain Figure 3 in the edge area.

[0109] Step B3: taking the image center points of the dilated sample image and the eroded sample image as references, calculating the intersection of the dilated sample image and the eroded sample image.

[0110] Step B4: determining the intersection area as the character edge area.

[0111] and Figure 3 The same principle is used. Using the dilated sample image and the eroded sample image, we can get Figure 2 The edge area of ​​the character shown in .

[0112] By applying the above example, by performing dilation and erosion processing on the input sample image respectively, the edge area of ​​the person corresponding to the sample image can be quickly determined, and then used to calculate the second loss value of the second loss function of the hair segmentation module.

[0113] After the edge region of the person is determined, it can be further calculated with the fine hair effect to obtain the hair edge region of the sample image.

[0114] In an exemplary embodiment, the step A21 may further include:

[0115] Taking the image center points of the fine hair effect and the edge area of ​​the character as references, calculating the intersection of the fine hair effect and the edge area of ​​the character;

[0116] The intersection area is determined as the hair edge area of ​​the sample image.

[0117] Among them, if the pixel value corresponding to the edge area of ​​the character is 1, the pixel value of the area where the fine hair effect and the edge area of ​​the character are intersected is calculated as the pixel value corresponding to the fine hair effect; if the pixel value corresponding to the edge area of ​​the character is 0, the pixel value of the area where the fine hair effect and the edge area of ​​the character are intersected is set to 0.

[0118] Applying the above example, by solving the intersection of the fine hair effect and the edge area of ​​the character, the hair edge area of ​​the sample image can be determined, which is then used to calculate the second loss value of the second loss function of the hair segmentation module.

[0119] In an exemplary embodiment, for step A3, the first loss value and the second loss value may be weightedly calculated by the following formula to obtain the loss value of the loss function of the multi-task learning training:

[0120] Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin;

[0121] Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

[0122] It should be noted that the weights of the first loss value and the second loss value are dynamically adjusted with the number of iterations; wherein the weight of the second loss value increases with the number of iterations, and the weight of the first loss value decreases with the number of iterations.

[0123] Applying the above example, by performing weighted calculation on the first loss value and the second loss value, the loss value of the loss function of the multi-task learning training can be obtained, and then the module parameters of the person segmentation module and the hair segmentation module can be adjusted respectively based on the loss value; so that the person segmentation module and the hair segmentation module are more perfect and the calculation results are more accurate.

[0124] After the human segmentation module and hair segmentation module are trained through the above examples, they can be applied to actual business.

[0125] Specifically, after determining the image to be processed, the image to be processed can be input into the person segmentation module and the hair segmentation module for calculation, and the person image contained in the image to be processed output by the person segmentation module and the hair image with a refined effect contained in the image to be processed output by the hair segmentation module are obtained. Finally, the person image and the hair image with a refined effect are fused to obtain a person image with a refined hair effect.

[0126] Please refer to Figure 4 The effect diagram of image processing shown in the figure ( Figure 4The images from left to right are respectively the image to be processed, the image of the person segmented from the image to be processed by the existing person segmentation module (i.e., the person prediction result, which has the problem that the boundary of the hair area is not clear enough, the edge of the hair is not sharp enough, there is a strong feathering effect, and the fine matting capability of the hair level cannot be achieved), the image of the person segmented from the image to be processed using the embodiment of this specification (the image of the person with the fine hair effect), the fine hair effect, and the hair edge area.

[0127] By comparison, it can be clearly found that the hair area of ​​the character image obtained by applying the embodiments of this specification has clear boundaries and sharp hair edges, achieving fine matting performance at the hair level.

[0128] In conclusion, by applying the embodiments of the present specification, the hair segmentation effect during character segmentation is optimized under the premise of increasing the amount of calculation. On the basis of the character segmentation module, a hair segmentation module with a hair segmentation branch is added. The hair segmentation module segments a hair image containing a fine hair effect from the image to be processed, and then the hair image containing the fine hair effect is merged with the character image segmented from the image to be processed by the original character segmentation module, so that the character image containing the fine hair effect can be obtained, thereby achieving the effect of fine hair segmentation.

[0129] With the aforementioned Figure 1 Corresponding to the embodiment of the image processing method shown, this specification also provides Figure 5 The embodiment of the image processing device shown in the figure can be implemented by software, hardware or a combination of software and hardware.

[0130] Figure 5 It is the same as the above Figure 1 A corresponding image processing device block diagram. Figure 5 The device includes a first computing unit 310, a second computing unit 320 and an image fusion unit 330.

[0131] The first calculation unit 310 is configured to input the image to be processed into the person segmentation module for calculation, and obtain the person image contained in the image to be processed output by the person segmentation module;

[0132] The second calculation unit 320 is configured to input the image to be processed into the hair segmentation module for calculation, and obtain a hair image with a refined effect contained in the image to be processed output by the hair segmentation module;

[0133] The image fusion unit 330 is configured to fuse the character image with the hair image with the refined effect to obtain a character image including a refined hair effect.

[0134] In an exemplary embodiment, the apparatus further comprises:

[0135] The module training unit is configured to execute the following first training unit, second training unit and third training unit to perform multi-task learning training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set for training are labeled with the person area and the hair area:

[0136] The first training unit is configured to input the sample image in the image sample set into the person segmentation module, and use the region where the marked person is located in the sample image as a supervisory signal to calculate a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image;

[0137] The second training unit is configured to input the person image output by the person segmentation module into the hair segmentation module, and use the region where the hair is marked in the sample image as a supervisory signal to calculate a second loss value of a second loss function of the hair segmentation module when a fine hair effect is segmented from the person image;

[0138] The third training unit is configured to perform weighted calculation on the first loss value and the second loss value to obtain a loss value of the loss function of the multi-task learning training, and adjust the module parameters of the person segmentation module and the hair segmentation module respectively based on the loss value.

[0139] In an exemplary embodiment, the second training unit is further configured to execute the first computing subunit and the second computing subunit;

[0140] The first calculation subunit is configured to calculate the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image;

[0141] The second calculation subunit is configured to calculate the hair edge area and the character image output by the character segmentation module to obtain a second loss value of the second loss function of the hair segmentation module.

[0142] In an exemplary embodiment, the first calculation subunit is further configured to calculate the intersection of the fine hair effect and the character edge area based on the image center point of the fine hair effect and the character edge area, and determine the area of ​​the intersection as the hair edge area of ​​the sample image.

[0143] In an exemplary embodiment, the first computing subunit is further configured to perform dilation and erosion processing on the input sample image, respectively, to obtain an enlarged dilated sample image and a reduced eroded sample image; to negate the pixel values ​​of the dilated sample image or the eroded sample image; to calculate the intersection of the dilated sample image and the eroded sample image based on the image center points of the dilated sample image and the eroded sample image, and to determine the area of ​​the intersection as the edge area of ​​the character.

[0144] In an exemplary embodiment, the second training unit is further configured so that the hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

[0145] In an exemplary embodiment, the first loss value and the second loss value are weightedly calculated by the following formula to obtain the loss value of the loss function of the multi-task learning training:

[0146] Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin;

[0147] Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

[0148] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0149] In an exemplary embodiment, an image processing electronic device is also provided, comprising a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to complete the image processing method described in any of the above embodiments.

[0150] In an exemplary embodiment, a computer-readable storage medium including instructions is further provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method described in any of the above embodiments.

[0151] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, and when the computer program or instruction is executed by a processor, the image processing method described in any of the above embodiments is completed.

[0152] Figure 6 It is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 The electronic device 400 in the embodiment may include one or more of the following components: a processing component 402, a memory 404, a power component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 418. The above electronic devices may adopt a similar hardware architecture.

[0153] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned three-dimensional virtual image generation method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0154] The memory 404 is configured to store various types of data to support operations on the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0155] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0156] The multimedia component 408 includes a screen that provides an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed or optical lens system with a focal length and optical zoom capability.

[0157] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 418. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0158] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboard, click wheel, options, etc. These options may include but are not limited to: home options, volume options, startup options, and lock options.

[0159] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of the components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of contact between the user and the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0160] The communication component 418 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 4G), or a combination thereof. In an exemplary embodiment, the communication component 418 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 418 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0161] In one embodiment of the present disclosure, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the image processing method described in any of the above embodiments.

[0162] In an embodiment of the present disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of an electronic device 400 to complete the image processing method described in any of the above embodiments. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0163] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0164] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: The following steps are performed to perform multi-task learning training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set used for training are labeled with the areas where the person is located and the areas where the hair is located: Inputting a sample image in the image sample set into the person segmentation module, taking the region where the marked person is located in the sample image as a supervisory signal, and calculating a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image; Inputting the person image output by the person segmentation module into the hair segmentation module, taking the area where the hair is marked in the sample image as a supervisory signal, and calculating the second loss value of the second loss function of the hair segmentation module when calculating the fine hair effect segmented from the person image; Performing weighted calculation on the first loss value and the second loss value to obtain a loss value of the loss function of the multi-task learning training, and adjusting the module parameters of the person segmentation module and the hair segmentation module respectively based on the loss value; Inputting the image to be processed into the trained person segmentation module for calculation, and obtaining a person image contained in the image to be processed output by the person segmentation module; The image to be processed is input into the trained hair segmentation module for calculation, so as to obtain a hair image with a refined effect contained in the image to be processed output by the hair segmentation module; wherein the hair image with a refined effect is more refined than the hair in the person image output by the person segmentation module; The character image is merged with the hair image with the refined effect to obtain a character image including the refined hair effect.

2. The method according to claim 1, characterized in that When calculating the fine hair effect segmented from the person image, the second loss value of the second loss function of the hair segmentation module includes: Calculate the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image; The hair edge area and the character image output by the character segmentation module are calculated to obtain a second loss value of the second loss function of the hair segmentation module.

3. The method according to claim 2, characterized in that The step of calculating the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image includes: Taking the image center points of the fine hair effect and the edge area of ​​the character as references, calculating the intersection of the fine hair effect and the edge area of ​​the character; The intersection area is determined as the hair edge area of ​​the sample image.

4. The method according to claim 2, characterized in that: The character edge area is calculated in the following way: The input sample image is dilated and eroded respectively to obtain an enlarged dilated sample image and a reduced eroded sample image; Negate pixel values ​​of the dilated sample image or the eroded sample image; Taking the image center points of the dilated sample image and the eroded sample image as references, calculating the intersection of the dilated sample image and the eroded sample image; The intersection area is determined as the character edge area.

5. The method according to claim 2, characterized in that: The fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module includes: The hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

6. The method according to claim 1, characterized in that The first loss value and the second loss value are weighted by the following formula to obtain the loss value of the loss function of the multi-task learning training: Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin; Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

7. An image processing device, characterized in that: The device comprises: The module training unit is configured to execute the following first training unit, second training unit and third training unit to perform multi-task learning training on the person segmentation module and the hair segmentation module until the loss value of the loss function of the multi-task learning training reaches an optimal value, wherein the sample images in the image sample set for training are labeled with the person area and the hair area: The first training unit is configured to input the sample image in the image sample set into the person segmentation module, and use the region where the marked person is located in the sample image as a supervisory signal to calculate a first loss value of a first loss function of the person segmentation module when segmenting the person image from the sample image; The second training unit is configured to input the person image output by the person segmentation module into the hair segmentation module, and use the region where the hair is marked in the sample image as a supervisory signal to calculate a second loss value of a second loss function of the hair segmentation module when a fine hair effect is segmented from the person image; The third training unit is configured to perform weighted calculation on the first loss value and the second loss value to obtain a loss value of the loss function of the multi-task learning training, and adjust the module parameters of the person segmentation module and the hair segmentation module respectively based on the loss value; A first calculation unit is configured to input the image to be processed into a person segmentation module for calculation, and obtain a person image contained in the image to be processed output by the person segmentation module; A second calculation unit is configured to input the image to be processed into a hair segmentation module for calculation, so as to obtain a hair image with a refined effect contained in the image to be processed output by the hair segmentation module; wherein the hair image with a refined effect is more refined than the hair in the person image output by the person segmentation module; The image fusion unit is configured to fuse the character image with the hair image with the refined effect to obtain a character image including a refined hair effect.

8. The device according to claim 7, characterized in that The second training unit is further configured to execute the first computing subunit and the second computing subunit; The first calculation subunit is configured to calculate the fine hair effect segmented by the hair segmentation module from the character image output by the character segmentation module and the character edge area corresponding to the sample image to obtain the hair edge area of ​​the sample image; The second calculation subunit is configured to calculate the hair edge area and the character image output by the character segmentation module to obtain a second loss value of the second loss function of the hair segmentation module.

9. The device according to claim 8, characterized in that The first calculation subunit is further configured to calculate the intersection of the fine hair effect and the character edge area based on the image center point of the fine hair effect and the character edge area, and determine the area of ​​the intersection as the hair edge area of ​​the sample image.

10. The device according to claim 8, characterized in that The first computing subunit is further configured to perform dilation and erosion processing on the input sample image respectively to obtain an enlarged dilated sample image and a reduced eroded sample image; invert the pixel values ​​of the dilated sample image or the eroded sample image; calculate the intersection of the dilated sample image and the eroded sample image based on the image center points of the dilated sample image and the eroded sample image, and determine the area of ​​the intersection as the edge area of ​​the character.

11. The device according to claim 8, characterized in that The second training unit is further configured so that the hair segmentation module segments the hair prediction result from the character image output by the character segmentation module, and performs guided filtering on the hair prediction result to obtain a fine hair effect.

12. The device according to claim 7, characterized in that The first loss value and the second loss value are weighted by the following formula to obtain the loss value of the loss function of the multi-task learning training: Loss=loss_human+(curr_epoch) / (total_epoch)*loss_hair_margin; Among them, the Loss is the loss value of the loss function of multi-task learning training, the loss_human is the first loss value, the curr_epoch is the current iteration number, the total_epoch is the preset total iteration number, and the loss_hair_margin is the second loss value.

13. An electronic device for image processing, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images