Image processing method, apparatus, device, and medium

Abstract

The present disclosure relates to an image processing method, device, equipment and medium, wherein the method comprises: obtaining a first transformed image corresponding to an original image; wherein the first transformed image is an image after performing transformation processing on the hair color of a target object in the original image; generating a second transformed image based on the first transformed image; wherein the non-hair area of the target object in the second transformed image does not contain hair shadow; generating a final transformed image according to the first transformed image and the second transformed image; and the final transformed image is an image in which the hair shadow in the non-hair area of the first transformed image is removed. The present disclosure can effectively improve the hair color transformation effect.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Facial effects have been widely used in various applications such as image editing software, photo-taking software, and live video streaming platforms. Users can change the appearance of their faces according to their needs, and changing hair color is one such user request. However, the inventors discovered through research that although hair color changes can be achieved in related technologies, the resulting hair color change images generally exhibit the phenomenon of hair shadows appearing in non-hair areas such as the forehead. This is especially noticeable when the original image shows a relatively dark hair color such as black or brown, resulting in more pronounced hair shadows on the forehead and thus unsatisfactory hair color change effects. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, device and medium.

[0004] This disclosure provides an image processing method, the method comprising: obtaining a first transformation map corresponding to an original image; wherein the first transformation map is an image after transforming the hair color of a target object in the original image; generating a second transformation map based on the first transformation map; wherein the non-hair region of the target object in the second transformation map does not contain hair shadows; generating a final transformation map according to the first transformation map and the second transformation map; wherein the final transformation map is an image in which hair shadows located in the non-hair region of the first transformation map are removed.

[0005] Optionally, the step of generating a second transformation map similar to the first transformation map based on the first transformation map includes: cropping the facial region of the first transformation map to obtain a facial region map in the first transformation map; and inputting the facial region map into a preset portrait generation model to obtain the second transformation map.

[0006] Optionally, the step of generating a final transformation map based on the first transformation map and the second transformation map includes: obtaining a target region to be corrected in the first transformation map; the target region includes hair shadows located in non-hair regions; and fusing the second transformation map and the first transformation map based on the target region to obtain the final transformation map.

[0007] Optionally, the step of fusing the second transformation image and the first transformation image based on the target region to obtain a final transformation image includes: obtaining a mask image corresponding to the target region, and fusing the second transformation image and the first transformation image based on the mask image to obtain a final transformation image; wherein, the pixels corresponding to the target region in the final transformation image are the pixels corresponding to the target region in the second transformation image, and the pixels corresponding to non-target regions in the final transformation image are the pixels corresponding to non-target regions in the first transformation image.

[0008] Optionally, the step of obtaining the target region to be corrected in the first transformation image includes: obtaining a first forehead prediction region and a second forehead prediction region of the target object in the first transformation image; and taking the region between the first hairline corresponding to the first forehead prediction region and the second hairline corresponding to the second forehead prediction region as the target region to be corrected.

[0009] Optionally, the step of obtaining the first forehead prediction region and the second forehead prediction region of the target object in the first transformation image includes: obtaining facial key points of the target object in the first transformation image; performing interpolation processing based on the facial key points, and obtaining the first forehead prediction region and the second forehead prediction region of the target object according to the interpolation result.

[0010] Optionally, the facial key points include multiple eyebrow key points; the step of performing interpolation processing based on the facial key points and obtaining the first forehead prediction region and the second forehead prediction region of the target object according to the interpolation result includes: determining the first bottom edge line of the first forehead prediction region based on the lines connecting the multiple eyebrow key points; setting a preset number of interpolation points on the first bottom edge line and obtaining a perpendicular line perpendicular to the first bottom edge line corresponding to each interpolation point; obtaining the first hairline corresponding to the first forehead prediction region of the target object based on the intersection of each perpendicular line and the edge of the hair area of ​​the target object; determining the first forehead prediction region based on the first bottom edge line and the first hairline, and generating the second forehead prediction region based on the first forehead prediction region; wherein, the hairline corresponding to the second forehead prediction region is lower than the hairline corresponding to the first forehead prediction region.

[0011] Optionally, the step of generating a second forehead prediction region based on the first forehead prediction region includes: determining a second bottom edge line of the second forehead prediction region based on the first bottom edge line; wherein the second bottom edge line coincides with the first bottom edge line; determining a second hairline corresponding to the second forehead prediction region based on the distance between the first hairline and the first bottom edge line; wherein the second hairline is located between the first hairline and the first bottom edge line; and determining the second forehead prediction region based on the second bottom edge line and the second hairline.

[0012] Optionally, the step of obtaining the target region to be corrected in the first transformation image includes: inputting the first transformation image into a pre-trained hair shadow recognition model and obtaining the hair shadow recognition result output by the hair shadow recognition model; and determining the target region to be corrected in the first transformation image based on the hair shadow recognition result.

[0013] Optionally, the method further includes: training a preset neural network model based on the original image and the final transformation image corresponding to the original image, and using the neural network model at the end of training as the hair color transformation model; wherein, the hair color transformation model is used to perform hair color change processing on the target object in the target image and output the final transformation image of the target image, and the final transformation image of the target image does not contain hair shadows located in non-hair areas.

[0014] This disclosure also provides an image processing apparatus, comprising: a first transformation map acquisition module, configured to acquire a first transformation map corresponding to an original image; wherein the first transformation map is an image after transforming the hair color of a target object in the original image; a second transformation map generation module, configured to generate a second transformation map based on the first transformation map; wherein the non-hair region of the target object in the second transformation map does not contain hair shadows; and a final transformation map generation module, configured to generate a final transformation map based on the first transformation map and the second transformation map; wherein the final transformation map is an image in which hair shadows located in the non-hair region of the first transformation map are removed.

[0015] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement any of the image processing methods described above.

[0016] This disclosure also provides a computer-readable storage medium storing a computer program for performing any of the image processing methods described above.

[0017] The technical solution provided in this disclosure first obtains a first transformation image corresponding to the original image; wherein, the first transformation image is an image after transforming the hair color of the target object in the original image; then, a second transformation image can be generated based on the first transformation image (the non-hair area of ​​the target object does not contain hair shadows); finally, a final transformation image is generated based on the first and second transformation images; the final transformation image is an image in which hair shadows located in the non-hair area of ​​the first transformation image are removed. This method can generate a second transformation image without hair shadows based on the first transformation image, and finally generate a final transformation image with hair shadows removed from the first transformation image based on the first and second transformation images. By removing hair shadows, the hair color transformation effect can be effectively improved.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a forehead prediction region provided in an embodiment of this disclosure;

[0023] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0027] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0028] To improve upon the poor hair color transformation effect and the problem of hair shadows appearing in non-hair areas in related technologies, this disclosure provides an image processing method, apparatus, device, and medium, which are described in detail below:

[0029] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S106:

[0030] Step S102: Obtain the first transformed image corresponding to the original image; wherein, the first transformed image is the image after transforming the hair color of the target object in the original image.

[0031] The original image is an image containing at least one object. This embodiment of the disclosure does not limit the type of object; for example, the object can be a person or other animal containing hair. In practical applications, the original image may only contain the object's face, or it may contain the object's whole body or half-body. The target object can be any object contained in the original image, or it can be an object specified by the user; there are no limitations here. The aforementioned first transformed image is the image obtained by initially performing hair color transformation processing on the target object in the original image. Any technology capable of hair color transformation can be used to obtain the first transformed image corresponding to the original image; there are no limitations here. For example, the hair area of ​​the target object in the original image can be identified first, and then the hair area can be color transformed to obtain the first transformed image. Since shadows may appear in non-hair areas such as the forehead during the hair color transformation process, the non-hair areas such as the forehead in the first transformed image after hair color transformation will mostly show varying degrees of hair shadows, especially when the hair color of the target object in the original image is dark, the hair shadow phenomenon is generally more obvious.

[0032] Step S104: Generate a second transformation map based on the first transformation map; wherein, the non-hair region of the target object in the second transformation map does not contain hair shadows.

[0033] In some implementations, a second transformation map can be generated based on the first transformation map using a human image generation model. The human image generation model can be a machine learning model, further including a deep learning model, and even more specifically, a generative adversarial network (GAN) capable of generating realistic images. The human image generation model can generate a second transformation map that is as realistic as possible to the first transformation map, for example, the similarity between the second and first transformation maps is greater than a preset similarity threshold. Although the second transformation map is generated by the human image generation model and is very similar to the first transformation map overall, there are still some differences in the image features of the second and first transformation maps in terms of details, such as subtle differences in color and texture. However, since the second transformation map is an image generated by the human image generation model itself, its non-hair areas typically do not contain hair shadows.

[0034] This disclosure does not limit the specific network structure of the portrait generation model. Furthermore, in practical applications, the portrait generation model can be pre-trained. A well-trained model has the function of generating images similar to the input image. Specific training methods can be found in related technologies and will not be elaborated here. In this disclosure, during training, the similarity between the model's output image and the input image can be set to be higher than a preset similarity threshold. Based on this, the similarity between the second transformed image output by the well-trained portrait generation model and the first transformed image input to it will be higher than the preset similarity threshold.

[0035] Step S106: Generate a final transformation image based on the first transformation image and the second transformation image; the final transformation image is an image with hair shadows removed from the non-hair region in the first transformation image. The non-hair region mainly includes the forehead region of the target object.

[0036] In practical applications, the first and second transformed images can be fused to obtain the final transformed image. In some specific implementation examples, the pixels corresponding to the target region (including the hair shadow in the first transformed image) in the final transformed image are the pixels corresponding to the target region in the second transformed image, and the pixels corresponding to the non-target region in the final transformed image are the pixels corresponding to the non-target region in the first transformed image. This fusion method cleverly removes the hair shadow from the first transformed image while ensuring that other regions in the first transformed image remain unchanged.

[0037] In summary, the above-described method provided by this disclosure can generate a second transformation image without hair shadows based on the first transformation image using a machine learning model, and finally generate a final transformation image with hair shadows removed from the first transformation image based on the first and second transformation images. By removing hair shadows, the hair color transformation effect can be effectively improved.

[0038] To facilitate the quick and easy acquisition of a relatively high-quality first transformation image in the initial stage, in some specific implementation examples, the original image can be input into a hair color initial transformation model to obtain the first transformation image corresponding to the original image output by the hair color initial transformation model. For example, the hair color initial transformation model can be a machine learning model, such as a neural network model. In specific implementations, a recurrent generative adversarial model can also be used. Since the first transformation image obtained from the hair color transformation processing based on the hair color initial transformation model often shows hair shadows in non-hair areas such as the forehead, this embodiment of the present disclosure further performs post-processing on the first transformation image output by the hair color initial transformation model to remove the hair shadows in the first transformation image.

[0039] After obtaining the first transformation image, this embodiment of the present disclosure can use a portrait generation model to generate a second transformation image similar to the first transformation image. In order to make the portrait generation model generate a more realistic second transformation image, in some specific embodiments, the portrait generation model may include a generative adversarial model. This embodiment of the present disclosure does not limit the implementation method of the generative adversarial model. For example, the generative adversarial model can be a style transfer generative adversarial model. The portrait generation model in this embodiment of the present disclosure can control the visual features to be expressed. From coarse features (such as posture, facial shape) to fine features (such as hair color), they can be processed according to requirements. The purpose of this embodiment of the present disclosure is to use a portrait generation model to generate a second transformation image similar to the first transformation image and without hair shadows in non-hair areas such as the forehead. The specific implementation steps for generating the second transformation image based on the first transformation image can be performed as follows: Steps (1) to (2):

[0040] Step (1): Crop the facial region in the first transformation image to obtain the facial region image in the first transformation image.

[0041] In practical applications, the first transformed image may contain not only the facial region but also other regions such as the target object's body or other objects in the background. To enable the model to better process the face, the facial region can be cropped from the first transformed image to obtain a facial region map. It should be noted that hair is also included in the facial region. Specifically, the facial region is cropped from the first transformed image based on one or more factors, such as the size of the facial image in the facial image dataset (e.g., a face image dataset), the occupancy of the facial region in the image, or the position of the facial region in the image, to facilitate subsequent model processing.

[0042] Step (2) involves inputting the facial region map into a preset portrait generation model to obtain a second transformed map. In some specific implementation examples, the portrait generation model can perform inversion processing on the facial region map to obtain the second transformed map.

[0043] Taking a style-transfer generative adversarial model (GAP) as an example, since the style-transfer GAP can generate highly realistic faces (such as human faces) based on random distributions, by inverting the facial region map using the style-transfer GAP, we can find the facial image that is closest to the model's input image from the model's output facial image. This results in a transformed image that approximates the facial region map; in other words, a second transformed image that is similar to the facial region in the first transformed image. While the second transformed image and the facial region map are very similar overall, they cannot be completely identical in detail and exhibit some differences. Because the second transformed image is generated by the style-transfer GAP, it typically does not display shadows in non-hair areas. Through this method, a relatively realistic second transformed image that removes hair shadows can be obtained conveniently and quickly.

[0044] Furthermore, this disclosure provides a specific implementation method for generating the final transformation diagram based on the first transformation diagram and the second transformation diagram, mainly including the following steps one and two:

[0045] Step 1: Obtain the target region to be corrected in the first transformation image; the target region includes hair shadows located in non-hair areas.

[0046] Step 2: Based on the target region, merge the second transformation map and the first transformation map to obtain the final transformation map.

[0047] This embodiment of the disclosure fully considers that the second transformation image is generated based on the first transformation image, and that there are certain differences between the second and first transformation images. Therefore, the first and second transformation images can be fused to ensure that the final hair color transformation image presented to the user is as realistic as possible. For example, the fused final transformation image retains the target area in the second transformation image and retains the area in the first transformation image other than the target area (non-target area), thereby effectively removing hair shadows located in non-hair areas based on the hair color transformation.

[0048] In some specific implementation examples, a mask image corresponding to the target region can be obtained, and the second transformation image and the first transformation image can be fused based on the mask image to obtain the final transformation image; wherein, the pixels corresponding to the target region in the final transformation image are the pixels corresponding to the target region in the second transformation image, and the pixels corresponding to the non-target region in the final transformation image are the pixels corresponding to the non-target region in the first transformation image.

[0049] The mask image corresponding to the target region mentioned above can also be called a mask image or a mask map. In the mask image corresponding to the target region, the target region and non-target regions can be distinguished in a specific way. For example, the pixel values ​​of the target region in the mask image are all 1, and the pixel values ​​of the non-target region are all 0; of course, the pixel values ​​of the target region in the mask image can also be all 0, and the pixel values ​​of the non-target region can all be 1. The above are just examples. By setting different pixel values ​​such as 0 / 1 for the target region and the non-target region, the target region and the non-target region can be clearly distinguished. Based on the mask image, the second transformation image can be fused with the first transformation image to obtain the final transformation image combining the target region in the second transformation image and the non-target region in the first transformation image, thus conveniently and effectively removing the hair shadow in the first transformation image.

[0050] In order to reliably obtain the target area to be corrected, the embodiments of this disclosure provide the following two implementation methods:

[0051] In Method 1 for obtaining the target area to be corrected, the following steps A to B can be followed:

[0052] Step A: Obtain the first and second forehead prediction regions of the target object in the first transformation image. In some implementations, the first and second forehead prediction regions have the same bottom edge, but different upper edges. The upper edge is the boundary line between the forehead region and the hair region, also known as the hairline. In some specific implementation examples, steps A1 to A2 can be referred to below:

[0053] Step A1: Obtain the facial key points of the target object in the first transformed image. It is understood that facial detection and other technologies annotate multiple key points on the face, such as eyebrow key points, nose key points, and mouth key points. In some specific implementation examples disclosed herein, the required facial key points include at least multiple eyebrow key points.

[0054] Step A2 involves interpolating facial key points and obtaining the first and second forehead prediction regions of the target object based on the interpolation results. Multiple key points can be interpolated based on existing facial key points to obtain prediction results for the two forehead regions.

[0055] For example, step A2 can be implemented by referring to steps A2.1 to A2.4 below:

[0056] Step A2.1: Determine the first bottom edge line of the first forehead prediction area based on the lines connecting multiple eyebrow key points. In some simple examples, the first bottom edge line is a straight line determined based on the left and right eyebrow peak key points.

[0057] Step A2.2: Set a preset number of interpolation points on the first bottom edge line, and obtain the perpendicular line to the first bottom edge line corresponding to each interpolation point. For example, insert multiple points on the first bottom edge line at preset intervals, and for each point, draw the line corresponding to that point that is perpendicular to the first bottom edge line (such as a straight line determined based on the key points of the left and right eyebrow peaks).

[0058] Step A2.3: Based on the intersection points between each vertical line and the edge of the hair area of ​​the target object, obtain the first hairline corresponding to the first forehead prediction area of ​​the target object. For example, multiple intersection points can be connected sequentially to obtain the upper edge line of the first forehead prediction area (i.e., the first hairline).

[0059] Step A2.4: Determine a first forehead prediction region based on the first bottom edge line and the first hairline, and generate a second forehead prediction region based on the first forehead prediction region. Specifically, based on the determined first bottom edge line and first hairline, the first bottom edge line and the first hairline enclose and constitute the first forehead prediction region. On this basis, a second forehead prediction region can be further generated based on the first forehead prediction region. It should be noted that both the first and second forehead prediction regions are for forehead prediction of the target object's face. The reason for the existence of two forehead prediction regions is that there is a shadow in the current first transformation image. Therefore, this embodiment of the disclosure obtains a first forehead prediction region containing the shadow and a second forehead prediction region not containing the shadow, so that the target area to be corrected (including the shadow) can be determined more accurately and reliably based on the two forehead prediction regions.

[0060] For example, the step of generating a second forehead prediction region based on a first forehead prediction region can refer to steps 1 to 3 below:

[0061] Step 1: Determine the second bottom edge line of the second forehead prediction region based on the first bottom edge line; wherein the second bottom edge line coincides with the first bottom edge line.

[0062] Step 2: Based on the distance between the first hairline and the first bottom edge line, determine the second hairline corresponding to the second forehead prediction area; wherein, the second hairline is located between the first hairline and the first bottom edge line.

[0063] In some implementation examples, given the known distance between the first hairline and the first bottom edge line, the hairline of the first forehead prediction area can be lowered based on empirical values, i.e., the distance between the hairline and the bottom edge line of the first forehead prediction area can be shortened, thereby obtaining the second forehead prediction area. In other words, the second forehead prediction area can be considered a sub-region of the first forehead prediction area. For example, in some specific implementation examples, the second hairline corresponding to the second forehead prediction area can be determined based on the distance between the first hairline and the first bottom edge line, and a preset proportional range. The ratio of the distance difference between the second hairline and the first hairline to the distance between the first hairline and the first bottom edge line is within a specified proportional range. This proportional range can be determined empirically, such as based on the range of hair shadows appearing on the forehead area in typical hair color transformation effects. The area between the first and second hairlines determined based on this proportional range typically includes hair shadows.

[0064] Step 3: Determine the second forehead prediction region based on the second bottom edge line and the second hairline. Based on the determined second bottom edge line and second hairline, the second bottom edge line and the second hairline together form the first forehead prediction region.

[0065] Step B involves defining the area between the first and second hairlines as the target area for correction. For clarity, please refer to [link to relevant documentation]. Figure 2 The diagram illustrates a forehead prediction region. The bottom edge lines of both the first and second forehead prediction regions are L0. The hairline (i.e., the first hairline) of the first forehead prediction region is L1, and the hairline (i.e., the second hairline) of the second forehead prediction region is L2. Initially, L0 and L1 can be determined. Then, L2 can be determined based on L1. The ratio of the distance difference between L1 and L2 to the distance between L1 and L0 needs to meet a preset ratio range (which can be determined empirically) to ensure that the area between L1 and L2 includes hair shadows on the forehead. For example, when calculating the ratio, the maximum distance between L1 and L2 and the maximum distance between L1 and L0 can be calculated. If the ratio of the two maximum distances is within the specified range, then L2 meets the requirement, and the area between L1 and L2 will typically include hair shadows. For details, please refer to the above description; further elaboration is omitted here. L0 and L1 enclose the first forehead prediction region, and L0 and L2 enclose the second forehead prediction region. The area enclosed between the hairline of the first forehead prediction region (L1) and the hairline of the second forehead prediction region (L2) is the target region to be corrected. The above method first obtains the first forehead prediction region, then obtains the second forehead prediction region through experience, and finally obtains the target region that needs to be corrected. This method is simpler and easier to implement.

[0066] The above method one can easily obtain the target area to be corrected without complicated calculations.

[0067] In method two for obtaining the target area to be corrected, the following steps a to b can be followed:

[0068] Step a: Input the first transformation image into the pre-trained hair shadow recognition model and obtain the hair shadow recognition result output by the hair shadow recognition model.

[0069] In practical applications, sample images labeled with hair shadow areas can be obtained in advance, and a preset neural network model can be trained using the sample images until the neural network model can output hair shadow prediction results that meet the expectations. The neural network model obtained when training stops is the hair shadow recognition model, which has the ability to identify hair shadows located in non-hair areas in the image.

[0070] Step b: Determine the target region to be corrected in the first transformation image based on the hair shadow recognition results.

[0071] The hair shadow recognition results can indicate the hair shadow located in the non-hair area in the first transformation image, and the area where the recognized hair shadow is located can be used as the target area to be corrected.

[0072] Although the above-mentioned model recognition method requires prior model pre-training, the model can be directly applied after training. It can effectively improve the efficiency of determining the target region to be corrected in the later stage, and can also better ensure the reliability of the target region to be corrected obtained in the end.

[0073] In practical applications, either method one or method two can be flexibly selected according to the needs, and there are no restrictions here.

[0074] Based on the foregoing, taking a person as an example, this disclosure also provides the following embodiments: Figure 3 The flowchart of an image processing method shown mainly includes the following steps S302 to S310:

[0075] Step S302: Input the original image into the hair color initial transformation model to obtain the first transformation image corresponding to the original image output by the hair color initial transformation model; wherein, the first transformation image is the image after transforming the hair color of the target person in the original image.

[0076] Step S304: Crop the face region from the first transformed image to obtain the face region image in the first transformed image.

[0077] Step S306: Input the face region map into the generative adversarial model, and perform inversion processing on the face region map through the generative adversarial model to obtain a second transformation map similar to the face region map; wherein, the non-hair region of the target person in the second transformation map does not contain hair shadows.

[0078] Step S308: Obtain the target region to be corrected in the first transformation image; the target region includes hair shadows located in non-hair regions.

[0079] Step S310: Obtain the mask image corresponding to the target region, and fuse the second transformation image and the first transformation image based on the mask image to obtain the final transformation image; wherein, the pixels corresponding to the target region in the final transformation image are the pixels corresponding to the target region in the second transformation image, and the pixels corresponding to the non-target region in the final transformation image are the pixels corresponding to the non-target region in the first transformation image.

[0080] The specific implementation methods of the above steps provided in this disclosure can all refer to the foregoing related content. Figure 3 The image processing method described above can initially obtain a preliminary and reliable hair color transformation result (first transformation image) using an initial hair color transformation model. Later, it uses the inversion technique of generative adversarial model to generate a second transformation image. Based on the mask image of the target region to be corrected in the first transformation image, the target region in the second transformation image is combined with the non-target region in the first transformation image to obtain the final transformation image. This method can conveniently and effectively remove hair shadows in the first transformation image and optimize the hair color transformation result.

[0081] After obtaining the final transformed graph, this disclosure further provides two application examples of the final transformed graph:

[0082] Application Example 1: Replace the original image with the final transformed image and display the final transformed image on the terminal interface. In this example, in various applications such as video editing software, photo-taking software, and live video streaming platforms, the original image is initially captured. Then, according to user settings, the hair color in the original image is transformed to a user-specified color. After obtaining the final transformed image based on the aforementioned image processing method, it can be directly displayed on the terminal interface.

[0083] Application Example 2: A preset neural network model is trained based on the original image and the corresponding final transformation image of the original image, and the neural network model at the end of training is used as the hair color transformation model; wherein, the hair color transformation model is used to process the hair color change of the target object in the target image and output the final transformation image of the target image, and the final transformation image of the target image does not contain hair shadows located in non-hair areas.

[0084] That is, the generated final transformed graph can be compared with the original... Figure 1These images are used as sample images for training the model. The model is trained to directly change the hair color of the original image without including hair shadows. This hair color change model can be widely used in various applications such as video editing software, photo software, and live video platforms. This makes it more convenient and faster for applications that require hair color change to directly use this model to obtain hair color change effects without including hair shadows.

[0085] In summary, the image processing method provided in this disclosure can effectively remove hair shadows in bald areas and improve the hair color transformation effect.

[0086] Corresponding to the aforementioned image processing method, Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 4 As shown, it includes:

[0087] The first transformation image acquisition module 402 is used to acquire the first transformation image corresponding to the original image; wherein, the first transformation image is the image after transforming the hair color of the target object in the original image;

[0088] The second transformation map generation module 404 is used to generate a second transformation map based on the first transformation map; wherein, the non-hair region of the target object in the second transformation map does not contain hair shadows;

[0089] The final transformation image generation module 406 is used to generate a final transformation image based on the first transformation image and the second transformation image; the final transformation image is an image in which hair shadows located in non-hair areas in the first transformation image are removed.

[0090] In summary, the apparatus provided in this embodiment can generate a second transformation image without hair shadows based on the first transformation image, and finally generate a final transformation image with hair shadows removed from the first transformation image based on the first and second transformation images. By removing hair shadows, the hair color transformation effect can be effectively improved.

[0091] In some implementations, the second transformation map generation module 404 is specifically used to: crop the facial region of the first transformation map to obtain a facial region map in the first transformation map; and input the facial region map into a preset portrait generation model to obtain a second transformation map similar to the facial region map.

[0092] In some implementations, the final transformation image generation module 406 is specifically used to: obtain the target region to be corrected in the first transformation image; the target region includes hair shadows located in non-hair regions; and based on the target region, fuse the second transformation image and the first transformation image to obtain the final transformation image.

[0093] In some implementations, the final transformation map generation module 406 is specifically used to: obtain a mask map corresponding to the target region, and fuse the second transformation map and the first transformation map based on the mask map to obtain a final transformation map; wherein, the pixels corresponding to the target region in the final transformation map are the pixels corresponding to the target region in the second transformation map, and the pixels corresponding to the non-target region in the final transformation map are the pixels corresponding to the non-target region in the first transformation map.

[0094] In some implementations, the final transformation map generation module 406 is specifically used to: obtain a first forehead prediction region and a second forehead prediction region of the target object in the first transformation map; and take the region between the first hairline corresponding to the first forehead prediction region and the second hairline corresponding to the second forehead prediction region as the target region to be corrected.

[0095] In some implementations, the final transformation map generation module 406 is specifically used to: obtain the facial key points of the target object in the first transformation map; perform interpolation processing based on the facial key points, and obtain the first forehead prediction region and the second forehead prediction region of the target object according to the interpolation result.

[0096] In some embodiments, the facial key points include multiple eyebrow key points; the final transformation image generation module 406 is specifically used to: determine a first bottom edge line of a first forehead prediction region based on the lines connecting the multiple eyebrow key points; set a preset number of interpolation points on the first bottom edge line, and obtain a perpendicular line to the first bottom edge line corresponding to each interpolation point; obtain a first hairline corresponding to the first forehead prediction region of the target object based on the intersection of each perpendicular line with the edge of the hair area of ​​the target object; determine the first forehead prediction region based on the first bottom edge line and the first hairline, and generate a second forehead prediction region based on the first forehead prediction region; wherein the hairline corresponding to the second forehead prediction region is lower than the hairline corresponding to the first forehead prediction region.

[0097] In some implementations, the final transformation map generation module 406 is specifically used to: determine a second bottom edge line of the second forehead prediction region based on the first bottom edge line; wherein the second bottom edge line coincides with the first bottom edge line; determine a second hairline corresponding to the second forehead prediction region based on the distance between the first hairline and the first bottom edge line; wherein the second hairline is located between the first hairline and the first bottom edge line; and determine the second forehead prediction region based on the second bottom edge line and the second hairline.

[0098] In some implementations, the final transformation map generation module 406 is specifically used to: input the first transformation map into a pre-trained hair shadow recognition model and obtain the hair shadow recognition result output by the hair shadow recognition model; and determine the target region to be corrected in the first transformation map based on the hair shadow recognition result.

[0099] In some embodiments, the device further includes a model training module for training a preset neural network model based on the original image and the final transformation image corresponding to the original image, and using the neural network model at the end of training as a hair color transformation model; wherein the hair color transformation model is used to perform hair color change processing on the target object in the target image and output the final transformation image of the target image, and the final transformation image of the target image does not contain hair shadows located in non-hair areas.

[0100] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0102] This disclosure provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-described image processing method.

[0103] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 500 includes one or more processors 501 and memory 502.

[0104] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0105] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the image processing method of the embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0106] In one example, the electronic device 500 may also include an input device 503 and an output device 504, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0107] In addition, the input device 503 may also include, for example, a keyboard, a mouse, etc.

[0108] The output device 504 can output various information to the outside, including determined distance information, direction information, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0109] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 500 may include any other suitable components depending on the specific application.

[0110] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the image processing method provided in the embodiments of this disclosure.

[0111] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0112] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the image processing method provided in embodiments of this disclosure.

[0113] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image processing method of this disclosure.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Obtain the first transformed image corresponding to the original image; wherein, the first transformed image is the image after transforming the hair color of the target object in the original image; A second transformation map is generated based on the first transformation map; wherein, the non-hair region of the target object in the second transformation map does not contain hair shadows; A final transformation image is generated based on the first transformation image and the second transformation image; the final transformation image is an image in which hair shadows located in non-hair areas of the first transformation image are removed. The step of generating the final transformation diagram based on the first transformation diagram and the second transformation diagram includes: Based on the target region to be corrected in the first transformation map, the second transformation map and the first transformation map are fused to obtain the final transformation map; wherein, the target region to be corrected is determined based on the region between the first hairline corresponding to the first forehead prediction region and the second hairline corresponding to the second forehead prediction region of the target object in the first transformation map, the first forehead prediction region and the second forehead prediction region of the target object are obtained by interpolation processing based on the facial key points of the target object in the first transformation map, the first forehead prediction region includes shadows, and the second forehead prediction region does not include shadows.

2. The method according to claim 1, characterized in that, The step of generating a second transformation graph based on the first transformation graph includes: The facial region is cropped from the first transformed image to obtain the facial region image in the first transformed image; The facial region map is input into a preset portrait generation model to obtain a second transformation map.

3. The method according to claim 1 or 2, characterized in that, The step of generating a final transformation graph based on the first transformation graph and the second transformation graph includes: Obtain the target region to be corrected in the first transformation image; the target region includes hair shadows located in non-hair areas; Based on the target region, the second transformation map and the first transformation map are fused to obtain the final transformation map.

4. The method according to claim 3, characterized in that, The step of fusing the second transformation map and the first transformation map based on the target region to obtain the final transformation map includes: Obtain the mask image corresponding to the target region, and fuse the second transformation image and the first transformation image based on the mask image to obtain the final transformation image; wherein, the pixels corresponding to the target region in the final transformation image are the pixels corresponding to the target region in the second transformation image, and the pixels corresponding to the non-target region in the final transformation image are the pixels corresponding to the non-target region in the first transformation image.

5. The method according to claim 3, characterized in that, The step of obtaining the target region to be corrected in the first transformed image includes: Obtain the first forehead prediction region and the second forehead prediction region of the target object in the first transformation image; The area between the first hairline corresponding to the first forehead prediction area and the second hairline corresponding to the second forehead prediction area is taken as the target area to be corrected.

6. The method according to claim 5, characterized in that, The steps of obtaining the first forehead prediction region and the second forehead prediction region of the target object in the first transformed image include: Obtain the facial key points of the target object in the first transformation image; Interpolation is performed based on the facial key points, and the first and second forehead prediction regions of the target object are obtained based on the interpolation results.

7. The method according to claim 6, characterized in that, The facial key points include multiple eyebrow key points; The steps of performing interpolation based on the facial key points and obtaining the first and second forehead prediction regions of the target object according to the interpolation results include: The first bottom edge line of the first forehead prediction area is determined based on the lines connecting the multiple eyebrow key points. A preset number of interpolation points are set on the first bottom edge line, and a perpendicular line to the first bottom edge line corresponding to each interpolation point is obtained; The first hairline corresponding to the first forehead prediction area of ​​the target object is obtained based on the intersection of each vertical line with the edge of the hair area of ​​the target object; The first forehead prediction region is determined based on the first bottom edge line and the first hairline, and a second forehead prediction region is generated based on the first forehead prediction region; wherein, the second hairline corresponding to the second forehead prediction region is lower than the first hairline.

8. The method according to claim 7, characterized in that, The step of generating a second forehead prediction region based on the first forehead prediction region includes: A second bottom edge line is determined based on the first bottom edge line to define the second forehead prediction region; wherein the second bottom edge line coincides with the first bottom edge line. Based on the distance between the first hairline and the first bottom edge line, the second hairline corresponding to the second forehead prediction area is determined; wherein, the second hairline is located between the first hairline and the first bottom edge line; The second forehead prediction area is determined based on the second bottom edge line and the second hairline.

9. The method according to claim 3, characterized in that, The step of obtaining the target region to be corrected in the first transformed image includes: The first transformation image is input into a pre-trained hair shadow recognition model, and the hair shadow recognition result output by the hair shadow recognition model is obtained; Based on the hair shadow recognition results, the target area to be corrected in the first transformation image is determined.

10. The method according to claim 1, characterized in that, The method further includes: A preset neural network model is trained based on the original image and the corresponding final transformation image, and the neural network model at the end of training is used as the hair color transformation model. The hair color transformation model is used to process the hair color of the target object in the target image and output the final transformation map of the target image. The final transformation map of the target image does not contain hair shadows located in non-hair areas.

11. An image processing apparatus, characterized in that, include: The first transformation image acquisition module is used to acquire the first transformation image corresponding to the original image; wherein, the first transformation image is an image after transforming the hair color of the target object in the original image; The second transformation map generation module is used to generate a second transformation map based on the first transformation map; wherein, the non-hair region of the target object in the second transformation map does not contain hair shadows; The final transformation image generation module is used to generate a final transformation image based on the first transformation image and the second transformation image; the final transformation image is an image in which hair shadows located in non-hair areas of the first transformation image are removed. The step of generating the final transformation diagram based on the first transformation diagram and the second transformation diagram includes: Based on the target region to be corrected in the first transformation map, the second transformation map and the first transformation map are fused to obtain the final transformation map; wherein, the target region to be corrected is determined based on the region between the first hairline corresponding to the first forehead prediction region and the second hairline corresponding to the second forehead prediction region of the target object in the first transformation map, the first forehead prediction region and the second forehead prediction region of the target object are obtained by interpolation processing based on the facial key points of the target object in the first transformation map, the first forehead prediction region includes shadows, and the second forehead prediction region does not include shadows.

12. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image processing method according to any one of claims 1-10.

Images