Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, the image processing method comprising: obtaining a face region image and a face stylized image corresponding to the face region in the image to be processed, the face stylized image being obtained by stylizing the face region image; performing a first migration process on the face stylized image for the face region according to the face region image to obtain a first stylized image, the face region color of the first stylized image matching the face region color of the face region image; fusing the first stylized image with other regions except the face region in the image to be processed to obtain a target stylized image. According to the image processing method, device, electronic device and storage medium of the present disclosure, the overall harmony of the stylized image can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and more specifically, to an image processing method, device, electronic device and storage medium. Background Art

[0002] Face stylization technology can convert a face portrait into a portrait of a specific style (e.g., sketch portrait style, cartoon style, oil painting style, etc.). Since it can achieve stylization while preserving the facial content, face stylization technology has been increasingly widely used. Stylized face fusion technology can achieve a natural fusion of the stylized face with the image background, which can improve the overall harmony of the picture and show a better stylization effect.

[0003] Current stylization techniques usually process the foreground and background areas of the face separately, and then integrate them through a fusion algorithm to obtain a stylized image. However, in the stylized image, there may be an unharmonious transition near the face area, for example, unnatural connections between the hair, neck, hands and other areas, which reduces the harmony of the image. In addition, there may be inharmonious phenomena between the stylized face and the original background. For example, there may be a large inconsistency between the lighting and color distribution of the stylized face and the original background, resulting in a sense of falseness and abruptness. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium to at least solve the above-mentioned problems in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: obtaining a face region image and a face stylized image corresponding to a face region in an image to be processed, wherein the face stylized image is obtained by stylizing the face region image; performing a first migration process on the face stylized image for the face region according to the face region image to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image; and fusing the first stylized image with other regions except the face region in the image to be processed to obtain a target stylized image.

[0006] Optionally, performing a first migration processing on the face stylized image for the face region according to the face region image to obtain a first stylized image includes: performing a second migration processing on the face stylized image for the hair region according to the face region image to obtain a second stylized image, wherein the hair color of the second stylized image matches the hair color of the face region image; performing a first migration processing on the second stylized image for the face region according to the face region image to obtain the first stylized image.

[0007] Optionally, performing a second migration processing on the face stylized image with respect to a hair region according to the face region image to obtain a second stylized image includes: performing a third migration processing on the face region image with respect to a hair region according to the face stylized image to obtain a third stylized image, wherein a hair texture of the third stylized image matches a hair texture of the face stylized image; performing a fourth migration processing on the hair region on a fourth stylized image obtained by performing a preset processing on the third stylized image according to the face region image to obtain a fifth stylized image, wherein a hair color of the fifth stylized image matches a hair color of the face region image; and intercepting the fifth stylized image through a portrait mask of the face region image, and fusing the intercepted image with the face region image to obtain the second stylized image.

[0008] Optionally, performing a third migration process on the face area image for the hair area according to the face stylized image to obtain a third stylized image includes: performing segmentation process on the face area image and the face stylized image for the hair area respectively to obtain an original image hair mask and a stylized hair mask; performing a third migration process on the face area image for the hair area according to an intersection of the original image hair mask and the stylized hair mask and the face stylized image to obtain the third stylized image.

[0009] Optionally, performing a fourth migration processing on a hair region on a fourth stylized image obtained by performing a preset processing on the third stylized image according to the face region image to obtain a fifth stylized image includes: intercepting the third stylized image by using the stylized hair mask, and fusing the intercepted image with the face stylized image to obtain the fourth stylized image; and performing a fourth migration processing on the hair region on the fourth stylized image according to the original hair mask and the face region image to obtain the fifth stylized image.

[0010] Optionally, performing a first migration process on the second stylized image for the face area according to the face area image to obtain the first stylized image includes: performing color fitting on the second stylized image according to the face area image and a portrait mask of the face area image to obtain a style image that matches the face area color of the face area image; and using the obtained style image that matches the face area color of the face area image as the first stylized image.

[0011] Optionally, performing a first migration process on the second stylized image for the face area according to the face area image to obtain the first stylized image includes: performing color fitting on the second stylized image according to the face area image and a portrait mask of the face area image to obtain a residual map between the face area image and the second stylized image; and obtaining the first stylized image according to the obtained residual map and the second stylized image.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: an image acquisition unit, configured to acquire a face region image and a face stylized image corresponding to the face region in an image to be processed, wherein the face stylized image is obtained by stylizing the face region image; a migration processing unit, configured to perform a first migration processing on the face stylized image for the face region according to the face region image, to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image; and an image fusion unit, configured to fuse the first stylized image with other regions except the face region in the image to be processed, to obtain a target stylized image.

[0013] Optionally, the migration processing unit may be configured to perform a second migration processing on the hair area of ​​the face stylized image based on the face area image to obtain a second stylized image, wherein the hair color of the second stylized image matches the hair color of the face area image; and perform a first migration processing on the face area of ​​the second stylized image based on the face area image to obtain the first stylized image.

[0014] Optionally, the migration processing unit may be configured to perform a third migration processing on the face area image for a hair area according to the face stylized image, so as to obtain a third stylized image, and the hair texture of the third stylized image matches the hair texture of the face stylized image; perform a fourth migration processing on the hair area according to the face area image on a fourth stylized image obtained by performing a preset processing on the third stylized image, so as to obtain a fifth stylized image, and the hair color of the fifth stylized image matches the hair color of the face area image; and intercept the fifth stylized image through the portrait mask of the face area image, and fuse the intercepted image with the face area image to obtain the second stylized image.

[0015] Optionally, the migration processing unit may be configured to perform segmentation processing on the face area image and the face stylized image for the hair area respectively, to obtain an original hair mask and a stylized hair mask; and perform a third migration processing on the face area image for the hair area according to the intersection of the original hair mask and the stylized hair mask and the face stylized image, to obtain the third stylized image.

[0016] Optionally, the migration processing unit may be configured to intercept the third stylized image using the stylized hair mask, and fuse the intercepted image with the face stylized image to obtain the fourth stylized image; and perform a fourth migration processing on the hair area of ​​the fourth stylized image according to the original hair mask and the face area image to obtain the fifth stylized image.

[0017] Optionally, the migration processing unit may be configured to perform color fitting on the second stylized image based on the face area image and the portrait mask of the face area image to obtain a style image that matches the face area color of the face area image; and use the obtained style image that matches the face area color of the face area image as the first stylized image.

[0018] Optionally, the migration processing unit may be configured to perform color fitting on the second stylized image according to the face area image and the portrait mask of the face area image to obtain a residual map between the face area image and the second stylized image; and obtain the first stylized image according to the obtained residual map and the second stylized image.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the image processing method according to the present disclosure.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute the image processing method according to the present disclosure.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, and instructions in the computer program product can be executed by a processor of a computer device to complete the image processing method according to the present disclosure.

[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0023] According to the image processing method, device, electronic device and storage medium disclosed in the present invention, based on the face area image of the original image, the color migration processing for the face area is performed on the stylized image corresponding thereto, so that the color of the face area in the processed stylized image and the face area color in the face area image can be kept relatively consistent, thereby reducing the unnaturalness of the transition between the foreground face frame and the background, improving the overall harmony of the stylized image, and retaining the attributes in the face area image (for example, accessories on the hair and other objects) to a greater extent, thereby improving the user experience.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 FIG. 1 is a diagram showing a stylized face picture obtained by using the current face stylization technology according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 2 is a diagram showing an exemplary terminal 200 to which exemplary embodiments according to the present disclosure can be applied.

[0028] Figure 3 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 is a flowchart illustrating a process of acquiring a first stylized image according to an exemplary embodiment of the present disclosure.

[0030] Figure 5 3 is a diagram showing the effect of a stylized image obtained by using the image processing method shown in the present disclosure according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 is a diagram showing the effect of a stylized image obtained by using the image processing method shown in the present disclosure according to another exemplary embodiment of the present disclosure.

[0032] Figure 7 An overall schematic diagram showing a process of acquiring a target stylized image according to an exemplary embodiment of the present disclosure is shown.

[0033] Figure 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0034] Fig. 9 is a block diagram illustrating an electronic device 900 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0037] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0038] Current face stylization technologies usually divide images into foreground and background, and process the foreground and background separately. Specifically, the foreground refers to the face frame cropped from the image with the face key points as anchor points, and the background refers to the area outside the face frame in the image. First, the foreground model is inferred for the portrait area within the face frame to produce a stylized effect for the face within the face frame. Then, the background model is inferred for the background or a filter is applied to the background to produce a stylized effect corresponding to the background. Finally, the stylized foreground and background are integrated through a fusion algorithm to obtain a stylized image.

[0039] The current face stylization technology has the following shortcomings: First, there is a relatively obvious dividing line in the transition area of ​​the face frame, which makes the foreground and background unnatural, reduces the harmony of the stylized image, and affects the user experience; second, there is a large inconsistency in color distribution, lighting, etc. between the foreground and background images after stylization, which easily makes people feel false and abrupt. For example, Figure 1 FIG. 2 is a diagram showing a face stylized picture obtained by using the current face stylization technology according to an exemplary embodiment of the present disclosure. Figure 1 , there is an obvious face frame boundary at the position pointed by the arrow, and the effect is not natural enough.

[0040] In order to avoid obvious dividing lines in the stylized image while maintaining the overall harmony of the stylized image, the present disclosure proposes an image processing method, device, electronic device and storage medium. Specifically, based on the face area image of the original image, the stylized image corresponding to it is subjected to color migration processing for the face area, so that the color of the face area in the processed stylized image is relatively consistent with the color of the face area in the face area image, thereby reducing the unnaturalness of the transition between the foreground face frame and the background, improving the overall harmony of the stylized image, and retaining the attributes in the face area image (for example, accessories on the hair, etc.) to a greater extent, thereby improving the user experience of the stylized image. Figures 2 to 9 An image processing method, apparatus, electronic device, and storage medium according to exemplary embodiments of the present disclosure are described in detail.

[0041] Figure 2 2 is a diagram showing an exemplary terminal 200 to which exemplary embodiments according to the present disclosure can be applied.

[0042] Reference Figure 2, the terminal 200 can perform stylized processing on the image to be processed (photos or video frames obtained by shooting, etc.) according to the image processing method shown in the present disclosure, and obtain a stylized image with a more natural stylized effect and higher overall consistency (for example, color distribution, lighting, etc.). In some embodiments, the terminal 200 can be a computer, a laptop, a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, a medical device, etc. In the specific implementation process, the terminal 200 can take or record photos or videos containing portraits in real time, and perform stylized processing on the photos and videos taken or recorded in real time according to the image processing method shown in the present disclosure, or the terminal 200 can also perform stylized processing on the pictures or videos input by the user according to the image processing method shown in the present disclosure, without limitation. In other embodiments, the terminal 200 can also be a chip with computing capabilities, which can be integrated into a computer, a laptop, a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, a medical device, etc., or independently execute the image processing method shown in the present disclosure. Of course, the terminal 200 in the embodiment of the present disclosure may also be a server, a server cluster, a distributed subsystem, a cloud processing platform, a server including a blockchain node, and a combination of these devices, and is not limited to a device implemented on the user side.

[0043] Figure 3 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

[0044] Reference Figure 3 In step 301, a face region image and a face stylized image corresponding to the face region in the image to be processed may be obtained, wherein the face stylized image is obtained by stylizing the face region image. Here, the image to be processed may be a picture (e.g., a photograph obtained by shooting, etc.) or a video frame.

[0045] Specifically, the image to be processed may include multiple face regions, and the face region images corresponding to the multiple face regions can be first detected and extracted from the image to be processed. human , through the face key point detection technology (for example, through the face key point detection network Net detect_human_lmk) detects the key point coordinates corresponding to each face region in the image to be processed (i.e., the relative coordinates in the image to be processed). Here, the face key point detection technology includes but is not limited to the traditional algorithm based on ASM (Active Shape Model) and AAM (Active Appearance Model), the method based on CPR (Cascaded Pose Regression) or the method based on deep learning, etc., and the present disclosure does not limit this. Then, the left eye contour point set, the right eye contour point set and the lip contour point set in the detected face key point coordinates can be used to calculate I human The left eye center point P of each face region in left_eye_center , right eye center point P right_eye_center and the lip center point P mouth_center , taking these three points as calibration points, each face region is rotated, cropped and filled according to a certain ratio (for example, according to the cropping standard of the FFHQ (Flickr-Faces-High-Quality) dataset or the cropping ratio set according to the actual situation, etc.), and the aligned face region image I is obtained. aligned_human_i , where i represents I human Here, each I aligned_human_i The corresponding parameters when rotating, cropping and filling are recorded as Params aligned_human_i In addition, for the convenience of calculation, the face area image I aligned_human_i The coordinate system of the key points of the face is transformed from the original coordinates of I human The upper left corner is the origin, I human The coordinate system with the height as the x-axis and the width as the y-axis is converted to I aligned_human_i The upper left corner is the origin, I aligned_human_i The height is the x-axis and the width is the y-axis.

[0046] After extracting the face region image, it can be stylized to obtain the corresponding face stylized image. Specifically, the face region image can be input into a pre-trained stylized network (e.g., Net stylization ) to obtain the corresponding face stylized image (for example, it can be recorded as I aligned_stylization_i Here, the stylized network can be any generative model that supports inputting real-world images and outputting stylized results, such as the StyleGAN_v2 model, etc., and the present disclosure does not limit this.

[0047] In step 302, a first migration process for the face region may be performed on the face stylized image based on the face region image to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image. Here, the face region refers to all regions related to the portrait in the face region image, including hair, face, neck and other human body regions that may be included, such as shoulders, hands, etc. The "matching" appearing here and below means "similar", for example, here it means that the face region color of the first stylized image is similar to the face region color of the face region image.

[0048] According to an exemplary embodiment of the present disclosure, Figure 4 The illustrated process results in a first stylized image.

[0049] Figure 4 is a flowchart illustrating a process of acquiring a first stylized image according to an exemplary embodiment of the present disclosure.

[0050] Reference Figure 4 In step 401, a second migration process for the hair region of the face stylized image may be performed based on the face region image to obtain a second stylized image, where the hair color of the second stylized image matches the hair color of the face region image.

[0051] According to an exemplary embodiment of the present disclosure, firstly, according to the stylized face image, the third migration process for the hair region can be performed on the face region image to obtain a third stylized image, where the hair texture of the third stylized image matches the hair texture of the stylized face image. Specifically, the face region image and the stylized face image can be segmented for the hair region to obtain an original image hair mask and a stylized hair mask, and according to the intersection of the original image hair mask and the stylized hair mask and the stylized face image, the third migration process for the hair region can be performed on the face region image to obtain the third stylized image. Here, the segmentation process for the hair region can be implemented, for example, by a hair segmentation network, and the hair segmentation network is any network that can achieve hair segmentation with the criterion of obtaining accurate segmentation prediction results, such as a U-Net network, a RefineNet network, etc. The third migration process for the hair region of the face region image can be implemented, for example, by any texture migration algorithm that supports mask guidance, including but not limited to Poisson fusion, AdaIN (Adaptive Instance Normalization, normalization method), etc., and the present disclosure does not limit this. In a specific implementation process, for example, the original hair mask corresponding to the face area image can be recorded as I hairseg_human_i , the stylized hair mask corresponding to the stylized face image is recorded as I hairseg_stylization_i , and the intersection of the two is recorded as Ihairseg_intersection_i , using Poisson fusion algorithm, to stylize the face image I aligned_stylization_i For the foreground, through the hair mask intersection I hairseg_intersection_i to I aligned_stylization_i The stylized hair area and the background face area image I are intercepted. aligned_human_i Fusion, get the third stylized image I aligned_blending_i That is, the third stylized image is an image with a stylized effect in the hair area and the rest of the area is an image of the face area. aligned_blending_i Hair texture and hair color with human face stylized image I aligned_stylization_i However, if we want to improve the overall consistency of the final target stylized image, we need a third stylized image I aligned_blending_i Hair color and face area image I hairseg_human_i Therefore, we can target I aligned_blending_i The hair color is migrated, that is, the fourth migration process (i.e., hair color migration process) for the hair region can be performed on the fourth stylized image obtained by performing a preset process on the third stylized image according to the face region image to obtain a fifth stylized image, where the hair color of the fifth stylized image matches the hair color of the face region image. In some embodiments, the third stylized image can be intercepted using a stylized hair mask, and the intercepted image can be fused with the face stylized image to obtain a fourth stylized image, and the fourth migration process for the hair region can be performed on the fourth stylized image according to the original hair mask and the face region image to obtain a fifth stylized image. For example, the third stylized image I aligned_blending_i As foreground, use the stylized hair mask I hairseg_stylization_i For L aligned_blending_i The captured image is then compared with the face stylized image I aligned_stylization_i Fusion, get the fused image I aligned_paste_i (ie, the fourth stylized image), where the fusion process, for example, but not limited to, can be expressed by formula (1):

[0052] I aligned_paste_i =I hairseg_stylization_i *I aligned_blending_i +(1-I hairseg_stylization_i )*I aligned_stylization_i (1)

[0053] Then, the face area image I can be transformed into aligned_human_i The hair color is transferred to the fused image I aligned_paste_i , get the result after hair color migration I aligned_hair_colortransfer_i (ie, the fifth stylized image).

[0054] According to an exemplary embodiment of the present disclosure, the fifth stylized image includes some non-face areas in addition to the face area. In order to avoid the inconsistency between the non-face area and the background in the image to be processed during the subsequent foreground and background fusion process, after obtaining the fifth stylized image, the fifth stylized image can be intercepted by the portrait mask of the face area image, and the intercepted image can be fused with the face area image to obtain a second stylized image. Therefore, in the second stylized image, only the area defined by the portrait mask is the result of stylization. Here, the area defined by the portrait mask includes the hair area, the facial area, and other areas related to the human body in the face area image, such as the neck area, the shoulder area, etc. In some embodiments, the face area image I aligned_human_i Input portrait segmentation network (can be recorded as Net humanseg ) and calculate the portrait mask I aligned_humanseg_i The portrait segmentation network can adopt any model that can achieve hair segmentation with the criterion of obtaining accurate segmentation prediction results, such as U-Net network, RefineNet network, etc., and the present disclosure does not limit this. The process of fusing the captured image with the face area image, for example, but not limited to, can be expressed by formula (2):

[0055] I aligned_hair_blending_i =I aligned_humanseg_i* I aligned_hair_colortransfer_i +(1-I aligned_humanseg_i )*I aligned_human_i (2)

[0056] Among them, I aligned_hair_blending_i I represents the second stylized image; aligned_humanseg_i I represents the portrait mask; aligned_hair_colortransfer_i Represents the fifth style image.

[0057] The hair color of the second stylized image obtained by the method shown in the above step 401 can be kept similar to the hair color of the face area image, and the hair texture can be kept similar to the hair texture of the face stylized image. Therefore, after being fused with other areas in the image to be processed, the harmony of the overall picture can be improved, and the features in the face area image (for example, accessories on the hair, etc.) can be retained to a greater extent, thereby improving the user experience.

[0058] Return to reference Figure 4 In step 402, a first migration process for the face region may be performed on the second stylized image according to the face region image to obtain a first stylized image.

[0059] According to an exemplary embodiment of the present disclosure, color fitting can be performed on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a style image that matches the face region color of the face region image. Here, the portrait mask can be obtained by performing mask calculation on the face region image (for example, calculating the portrait mask through the aforementioned portrait segmentation network). The execution of color fitting can be implemented, for example, by using the Lasso (The Least Absolute Shrinkage and Selectionator Operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not limit this. For example, in the process of performing color fitting using the Lasso regression algorithm, the second stylized image can be used as the input of the Lasso regression algorithm, the portrait mask of the face region image can be used as the constraint condition, the face region image can be used as the iterative optimization target, and iterative optimization can be continuously performed until the iterative end condition is reached (for example, the preset number of iterations is reached or the iteration loss is less than the set threshold value, etc.), and the style image that best matches the face region color of the face region image can be obtained, and the obtained style image that best matches the face region color of the face region image can be used as the first stylized image.

[0060] According to an exemplary embodiment of the present disclosure, color fitting can also be performed on the second stylized image according to the face area image and the portrait mask of the face area image to obtain a residual map between the face area image and the second stylized image. Similarly, the portrait mask can be obtained by performing mask calculation on the face area image (for example, calculating the portrait mask through the aforementioned portrait segmentation network). The execution of color fitting can be achieved, for example, by using the Lasso (The Least AbsoluteShrinkage and Selectionator Operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not impose any restrictions on this. For example, in the process of performing color fitting using the Lasso regression algorithm, the second stylized image can be used as the input of the Lasso regression algorithm, the portrait mask of the face area image can be used as a constraint condition, and the residual map between the face area image and the second stylized image can be used as the iterative optimization target to perform iterative optimization until the iterative end condition is reached (for example, a preset number of iterations is reached or the iteration loss is less than a set threshold, etc.), and a residual map that can reflect the true residual between the face area image and the second stylized image can be obtained. The first stylized image can be obtained based on the final residual map and the second stylized image. For example, but not limited to, the first stylized image can be obtained by adding the final residual map to the second stylized image.

[0061] The first stylized image obtained by the method shown in step 402 above can make the color of the face area of ​​the first stylized image adaptively keep relatively consistent with the color of the face area of ​​the portrait area image because the color migration of the face area of ​​the second stylized image is predicted by color fitting. Therefore, after being fused with other areas in the image to be processed, the unnatural degree of the connection between the foreground and the background can be reduced, and the inconsistency between the foreground and the background in terms of color distribution, lighting, etc. can be reduced, thereby further improving the harmony of the overall picture. In addition, it can retain the features in the face area image (for example, accessories on the hair, etc.) to a greater extent, thereby improving the user experience.

[0062] Figure 5 3 is a diagram showing the effect of a stylized image obtained by using the image processing method shown in the present disclosure according to an exemplary embodiment of the present disclosure.

[0063] Reference Figure 5 ,in, Figure 5 (a) is a screenshot of the eyes of a portrait image that needs to be stylized. Figure 5 (b) is to use the image processing method shown in the present disclosure to Figure 5 (a) The stylized image obtained by stylization processing, Figure 5 (c) To adopt the current relevant technology Figure 5 (a) A stylized image obtained by stylization processing, where: Figure 5 (b) and Figure 5 (c) For the same target style, compare Figure 5 (b) and Figure 5 (c) The eye colors of the stylized image obtained by the image processing method shown in the present disclosure are more natural and vivid, and closer to the desired target stylization effect.

[0064] Figure 6 is a diagram showing the effect of a stylized image obtained by using the image processing method shown in the present disclosure according to another exemplary embodiment of the present disclosure.

[0065] Reference Figure 6 ,in, Figure 6 (a) is a portrait image that needs to be stylized. Figure 6 (b) is to use the image processing method shown in the present disclosure to Figure 6 (a) The stylized image obtained by stylization processing, Figure 6 (c) To adopt the current relevant technology Figure 6 (a) A stylized image obtained by stylization processing, where: Figure 6 The stylized styles of (b) and (c) are "watercolor illustration" and "Korean comics", respectively. In addition, to avoid leaking user information, Figure 6 The face area in the image is mosaicked. Figure 6 (b) and Figure 6 (c) shows that the stylized image obtained by the image processing method shown in the present disclosure is more naturally connected with the background image in the transition area of ​​the face frame, and the overall color distribution is more consistent with the background image.

[0066] According to an exemplary embodiment of the present disclosure, the first migration process for the face region can also be performed directly based on the face region image and the face stylized image to obtain a first stylized image. Specifically, the face stylized image can be directly color-fitted. During the color-fitting process, the portrait mask corresponding to the face region image is used as a constraint condition, and the face region image is used as an optimization target to perform iterative optimization. At the end of the iterative optimization, a stylized image (i.e., the first stylized image) with a color similar to that of the face region in the face region image can be obtained. Alternatively, the residual map of the face region image and the face stylized image can be used as an optimization target to perform iterative optimization, and the residual map obtained at the end of the iterative optimization is added to the face stylized image to obtain the first stylized image. Similarly, the color fitting process here includes but is not limited to being implemented through the Lasso (The Least Absolute Shrinkage and Selectionator operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not limit this. Since the face area color migration of the face stylized image is predicted by color fitting, the face area color of the first stylized image can be adaptively kept relatively consistent with the face area color of the face area image. Therefore, after being fused with other areas in the image to be processed, the unnatural degree of the connection between the foreground and the background can be reduced, thereby improving the harmony of the overall picture.

[0067] Return to reference Figure 3 In step 303, the first stylized image may be fused with other regions of the image to be processed except the face region to obtain a target stylized image.

[0068] According to an exemplary embodiment of the present disclosure, each face region image in the image to be processed corresponds to a set of scaling and rotation parameters Params aligned_human_i , for Params aligned_human_i Perform inverse transformation to get a set of reverse transformation parameters Reverse_Params aligned_human_i , for each face region image, the first stylized image is transformed using the corresponding reverse transformation parameters Reverse_Params aligned_human_i, each first stylized image can be pasted to a corresponding position in the image to be processed (i.e., fused with other regions in the image to be processed except the face region), thereby obtaining a target stylized image. Here, when the first stylized image is fused with other regions in the image to be processed except the face region, other regions in the image to be processed except the face region can be stylized (for example, by stylized model inference or by filter processing) to obtain a target stylized image with better overall consistency.

[0069] The networks involved in the image processing method shown above (for example, the hair segmentation network, the portrait segmentation network, etc.) can use the PyTorch deep learning framework for neural network training, and the entire image processing method can be integrated using the Python language. For a 512x512 resolution portrait image, the image processing method shown in the present disclosure can complete the calculation of all the above processes on a server computer equipped with an NVIDIA GTX 2070 graphics card, and output a stylized portrait image with better picture consistency and more harmonious and natural foreground and background.

[0070] Based on a combination of one or more of the foregoing embodiments, Figure 7 An overall schematic diagram showing a process of acquiring a target stylized image according to an exemplary embodiment of the present disclosure is shown.

[0071] Reference Figure 7 First, a face frame is extracted from the image to be processed to obtain at least one face region image, and then, for each face region image in the at least one face region image, a stylization process is first performed to obtain a corresponding face stylized image, and then, texture and color migration is performed on the hair region according to the face region image and the corresponding face stylized image, and then face color migration is performed on the image obtained by the texture and color migration process to obtain a stylized image whose face region color matches the face region image, and finally, the stylized image is merged with the background in the image to be processed to obtain a target stylized image. Since the texture and color migration process is performed on the hair region, the hair texture of the face stylized image can be maintained while the hair color of the original face region image is maintained, so that the face stylized image obtained after the process can be more naturally and harmoniously consistent with the background in the image to be processed. In addition, since the face color migration is performed on the face stylized image obtained after the process, the face region color of the face stylized image finally obtained can be adaptively kept relatively consistent with the image to be processed, and better harmony is maintained as a whole.

[0072] Figure 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0073] Reference Figure 8, an image processing apparatus 800 according to an exemplary embodiment of the present disclosure may include an image acquisition unit 801 , a migration processing unit 802 , and an image fusion unit 803 .

[0074] The image acquisition unit 801 can acquire a face region image and a face stylized image corresponding to the face region in the image to be processed, wherein the face stylized image is obtained by performing stylized processing on the face region image.

[0075] The migration processing unit 802 may perform a first migration processing on the face region of the face stylized image according to the face region image to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image.

[0076] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 may perform a second migration process on the hair region of the face stylized image based on the face region image to obtain a second stylized image, and may perform a first migration process on the face region of the second stylized image based on the face region image to obtain a first stylized image. Here, the hair color of the second stylized image matches the hair color of the face region image.

[0077] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 may first perform a third migration processing on the face region image for the hair region according to the face stylized image to obtain a third stylized image, wherein the hair texture of the third stylized image matches the hair texture of the face stylized image. Specifically, the migration processing unit 802 may perform segmentation processing on the face region image and the face stylized image for the hair region respectively to obtain an original image hair mask and a stylized hair mask, and perform a third migration processing on the face region image for the hair region according to the intersection of the original image hair mask and the stylized hair mask and the face stylized image to obtain the third stylized image. Here, the segmentation processing on the hair region may be implemented, for example, by a hair segmentation network, and the hair segmentation network is any network that can implement hair segmentation with the criterion of obtaining an accurate segmentation prediction result, such as a U-Net network, a RefineNet network, etc. The third migration processing on the hair region may be implemented, for example, by any texture migration algorithm that supports mask guidance, such as Poisson fusion, AdaIN (Adaptive Instance Normalization, normalization method), etc., and the present disclosure does not limit this.

[0078] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 may perform a fourth migration processing on the hair area of ​​the fourth stylized image obtained by performing a preset processing on the third stylized image according to the face area image to obtain a fifth stylized image, wherein the hair color of the fifth stylized image matches the hair color of the face area image. In some embodiments, the migration processing unit 802 may intercept the third stylized image using a stylized hair mask, and merge the intercepted image with the face stylized image to obtain a fourth stylized image, and perform a fourth migration processing on the hair area of ​​the fourth stylized image according to the original hair mask and the face area image to obtain a fifth stylized image.

[0079] After obtaining the fifth stylized image, the migration processing unit 802 may intercept the fifth stylized image through the portrait mask of the face region image, and fuse the intercepted image with the face region image to obtain a second stylized image, where the region defined by the portrait mask includes the hair region, the facial region, and other regions related to the human body in the face region image, such as the neck region, the shoulder region, etc. In some embodiments, the face region image I may be merged into the second stylized image. aligned_human_i Input portrait segmentation network (can be recorded as Net humanseg ) and calculate the portrait mask I aligned_humanseg_i The portrait segmentation network can adopt any model that can achieve hair segmentation with the criterion of obtaining accurate segmentation prediction results, such as U-Net network, RefineNet network, etc., and the present disclosure does not limit this.

[0080] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 can perform color fitting on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a style image that matches the face region color of the face region image. Here, the portrait mask can be obtained by performing mask calculation on the face region image (for example, calculating the portrait mask through the aforementioned portrait segmentation network), and the execution of color fitting can be implemented, for example, by using the Lasso (The Least Absolute Shrinkage and Selectionator Operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not limit this. For example, in the process of performing color fitting using the Lasso regression algorithm, the second stylized image can be used as the input of the Lasso regression algorithm, the portrait mask of the face region image can be used as the constraint condition, the face region image can be used as the iterative optimization target, and iterative optimization can be continuously performed until the iteration end condition is reached (for example, the preset number of iterations is reached or the iteration loss is less than the set threshold value, etc.), and the style image that best matches the face region color of the face region image can be obtained, and the obtained style image that best matches the face region color of the face region image can be used as the first stylized image.

[0081] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 may also perform color fitting on the second stylized image based on the face area image and the portrait mask of the face area image to obtain a residual map between the face area image and the second stylized image. Similarly, the portrait mask can be obtained by performing mask calculation on the face area image (for example, calculating the portrait mask through the aforementioned portrait segmentation network). The execution of color fitting can be achieved, for example, by using the Lasso (The Least Absolute Shrinkage and Selectionator Operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not impose any restrictions on this. For example, in the process of performing color fitting using the Lasso regression algorithm, the second stylized image can be used as the input of the Lasso regression algorithm, the portrait mask of the face area image can be used as a constraint condition, and the residual map between the face area image and the second stylized image can be used as the iterative optimization target to perform iterative optimization until the iterative end condition is reached (for example, a preset number of iterations is reached or the iteration loss is less than a set threshold, etc.), and a residual map that can reflect the true residual between the face area image and the second stylized image can be obtained. The first stylized image can be obtained based on the final residual map and the second stylized image. For example, but not limited to, the first stylized image can be obtained by adding the final residual map to the second stylized image.

[0082] According to an exemplary embodiment of the present disclosure, the migration processing unit 802 may also directly perform the first migration processing for the face region based on the face region image and the face stylized image to obtain the first stylized image. Specifically, the face stylized image may be directly color-fitted. During the color-fitting process, the portrait mask corresponding to the face region image is used as a constraint condition, and the face region image is used as an optimization target to perform iterative optimization. At the end of the iterative optimization, a stylized image (i.e., the first stylized image) with a color similar to that of the face region in the face region image may be obtained. Alternatively, the residual image of the face region image and the face stylized image may be used as an optimization target to perform iterative optimization, and the residual image obtained at the end of the iterative optimization is added to the face stylized image to obtain the first stylized image. Similarly, the color fitting process here includes but is not limited to being implemented through the Lasso (The Least Absolute Shrinkage and Selector operator) regression algorithm, the Ridge regression algorithm, etc., and the present disclosure does not limit this.

[0083] The image fusion unit 803 may fuse the first stylized image with other regions of the image to be processed except the face region to obtain a target stylized image.

[0084] According to an exemplary embodiment of the present disclosure, each face region image in the image to be processed corresponds to a set of scaling and rotation parameters Params aligned_human_i The image fusion unit 803 can process Params aligned_human_i Perform an inverse transformation to obtain a set of reverse transformation parameters Reverse_Params aligned_humax_i , for each face region image, the first stylized image is transformed using the corresponding reverse transformation parameters Reverse_Params aligned_human_i , the image fusion unit 803 may paste each first stylized image to a corresponding position in the image to be processed (i.e., fuse with other regions in the image to be processed except the face region), thereby obtaining a target stylized image. Here, when fusing the first stylized image with other regions in the image to be processed except the face region, other regions in the image to be processed except the face region may be stylized (e.g., by stylized model inference or by filter processing) to obtain a target stylized image with better overall consistency.

[0085] Fig. 9 is a block diagram of an electronic device 900 according to an exemplary embodiment of the present disclosure.

[0086] Reference Fig. 9The electronic device 900 includes at least one memory 901 and at least one processor 902, wherein the at least one memory 901 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 902, an image processing method according to an exemplary embodiment of the present disclosure is executed.

[0087] As an example, the electronic device 900 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 900 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 900 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0088] In the electronic device 900, the processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0089] The processor 902 may execute instructions or codes stored in the memory 901, wherein the memory 901 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0090] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 901 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 901 and the processor 902 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 902 can read files stored in the memory.

[0091] In addition, the electronic device 900 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 900 may be connected to each other via a bus and / or a network.

[0092] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the image processing method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0093] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. Instructions in the computer program product may be executed by a processor of a computer device to implement the image processing method according to the exemplary embodiment of the present disclosure.

[0094] According to the image processing method, device, electronic device and storage medium disclosed in the present invention, based on the face area image of the original image, the color migration processing for the face area is performed on the stylized image corresponding thereto, so that the color of the face area in the processed stylized image and the face area color in the face area image can be kept relatively consistent, thereby reducing the unnaturalness of the transition between the foreground face frame and the background, improving the overall harmony of the stylized image, and retaining the attributes in the face area image (for example, accessories on the hair and other objects) to a greater extent, thereby improving the user experience.

[0095] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0096] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a face region image and a face stylized image corresponding to the face region in the image to be processed, wherein the face stylized image is obtained by performing stylized processing on the face region image; performing a third migration process on the face region image according to the face stylized image to obtain a third stylized image, wherein the hair texture of the third stylized image matches the hair texture of the face stylized image; According to the face region image, performing a fourth migration process on a hair region of a fourth stylized image obtained by performing a preset process on the third stylized image to obtain a fifth stylized image, wherein the hair color of the fifth stylized image matches the hair color of the face region image; intercepting the fifth stylized image through the portrait mask of the face region image, and fusing the intercepted image with the face region image to obtain a second stylized image; performing a first migration process on the face region of the second stylized image according to the face region image to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image; The first stylized image is fused with other regions of the image to be processed except the face region to obtain a target stylized image.

2. The image processing method according to claim 1, characterized in that: The step of performing a third migration process on the face region image with respect to the hair region according to the face stylized image to obtain a third stylized image includes: Performing segmentation processing on the face region image and the face stylized image respectively with respect to the hair region to obtain an original image hair mask and a stylized hair mask; According to the intersection of the original hair mask and the stylized hair mask and the stylized face image, a third migration process is performed on the face region image with respect to the hair region to obtain the third stylized image.

3. The image processing method according to claim 2, characterized in that: The step of performing a fourth migration process on a hair region of a fourth stylized image obtained by performing a preset process on the third stylized image according to the face region image to obtain a fifth stylized image includes: intercepting the third stylized image by using the stylized hair mask, and fusing the intercepted image with the stylized face image to obtain the fourth stylized image; According to the original image hair mask and the face region image, the fourth stylized image is subjected to a fourth migration process for the hair region to obtain the fifth stylized image.

4. The image processing method according to claim 1, wherein: The step of performing a first migration process on the face region of the second stylized image according to the face region image to obtain the first stylized image includes: performing color fitting on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a style image that matches the face region color of the face region image; The obtained style image that matches the face region color of the face region image is used as the first stylized image.

5. The image processing method according to claim 1, wherein: The step of performing a first migration process on the face region of the second stylized image according to the face region image to obtain the first stylized image includes: performing color fitting on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a residual map between the face region image and the second stylized image; The first stylized image is obtained according to the obtained residual image and the second stylized image.

6. An image processing device, characterized in that: include: An image acquisition unit is configured to: acquire a face region image and a face stylized image corresponding to the face region in the image to be processed, wherein the face stylized image is obtained by performing stylized processing on the face region image; The migration processing unit is configured as follows: performing a third migration process on the face region image according to the stylized face image to obtain a third stylized image, wherein the hair texture of the third stylized image matches the hair texture of the stylized face image; According to the face region image, performing a fourth migration process on a hair region of a fourth stylized image obtained by performing a preset process on the third stylized image to obtain a fifth stylized image, wherein the hair color of the fifth stylized image matches the hair color of the face region image; intercepting the fifth stylized image through the portrait mask of the face region image, and fusing the intercepted image with the face region image to obtain a second stylized image; performing a first migration process on the face region of the second stylized image according to the face region image to obtain a first stylized image, wherein the face region color of the first stylized image matches the face region color of the face region image; The image fusion unit is configured to fuse the first stylized image with other regions of the image to be processed except the face region to obtain a target stylized image.

7. The image processing device according to claim 6, wherein: The migration processing unit is configured to: Performing segmentation processing on the face region image and the face stylized image respectively with respect to the hair region to obtain an original image hair mask and a stylized hair mask; According to the intersection of the original hair mask and the stylized hair mask and the stylized face image, a third migration process is performed on the face region image with respect to the hair region to obtain the third stylized image.

8. The image processing device according to claim 7, wherein: The migration processing unit is configured to: intercepting the third stylized image by using the stylized hair mask, and fusing the intercepted image with the stylized face image to obtain the fourth stylized image; According to the original image hair mask and the face region image, the fourth stylized image is subjected to a fourth migration process for the hair region to obtain the fifth stylized image.

9. The image processing device according to claim 6, wherein: The migration processing unit is configured to: performing color fitting on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a style image that matches the face region color of the face region image; The obtained style image that matches the face region color of the face region image is used as the first stylized image.

10. The image processing device according to claim 6, wherein: The migration processing unit is configured to: performing color fitting on the second stylized image according to the face region image and the portrait mask of the face region image to obtain a residual map between the face region image and the second stylized image; The first stylized image is obtained according to the obtained residual image and the second stylized image.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the image processing method according to any one of claims 1 to 5.

12. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the image processing method according to any one of claims 1 to 5.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images