Skin color migration method and device, storage medium and electronic device

Abstract

The present application relates to the technical field of makeup migration, and provides a skin color migration method and device, a storage medium and an electronic device. The skin color migration method includes: performing image segmentation on a source image and a reference image respectively to obtain a skin color area in the source image and a skin color area in the reference image; dividing the skin color area in the source image and the skin color area in the reference image into multiple sub-areas according to different brightness levels; performing histogram matching on each sub-area in the source image and a sub-area with the same brightness level in the reference image respectively to obtain a migration result image. The method improves the accuracy of histogram matching by using partition matching, and effectively improves the situation where the skin color migration effect is poor due to the large difference between the proportion of the shadow area and / or highlight area in the source image and the proportion of the shadow area and / or highlight area in the reference image.

Description

Technical Field

[0001] The present invention relates to the technical field of makeup migration, and in particular to a skin color migration method and device, a storage medium and an electronic device. Background Art

[0002] Makeup transfer refers to the virtual makeup process of transferring the makeup of a reference image (for example, with makeup) to the source image (without makeup). Compared with the physical makeup process, the virtual makeup process takes less time and does not damage the skin. It can meet the needs of modern people to try a large number of makeup in a short period of time. At the same time, it can be easily integrated into the makeup recommendation function and has broad application prospects.

[0003] Common makeup transfer tasks include skin color transfer, lipstick transfer, eye shadow transfer, etc. Existing methods usually use histogram matching to perform skin color transfer, but when the proportion of shadow area and / or highlight area in the source image is significantly different from that in the reference image, the transfer effect is often poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a skin color migration method and device, a storage medium and an electronic device to improve the above-mentioned technical problems.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a skin color migration method, comprising: performing image segmentation on a source image and a reference image respectively to obtain a skin color area in the source image and a skin color area in the reference image; dividing the skin color area in the source image and the skin color area in the reference image into a plurality of sub-areas according to different brightness levels; performing histogram matching on each sub-area in the source image with a sub-area with the same brightness level in the reference image to obtain a migration result image.

[0007] The above method does not perform histogram matching between the entire skin color area in the source image and the reference image, but further divides the skin color area into multiple sub-areas according to the brightness of the pixels, and only performs histogram matching between the sub-areas with the same brightness level in the source image and the reference image. This method of sub-area matching improves the accuracy of histogram matching, so that the brightness of the pixels in the source image before and after matching will not change across levels, thereby effectively improving the situation where the skin color migration effect is poor due to the large difference between the proportion of the shadow area and / or highlight area in the source image and the proportion of the shadow area and / or highlight area in the reference image.

[0008] In an implementation of the first aspect, dividing the skin color area in the source image and the skin color area in the reference image into multiple sub-areas according to different brightness levels, includes: setting the number of clusters to the total number of brightness levels, and clustering the pixels in the skin color area of ​​the source image and the pixels in the skin color area of ​​the reference image, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into multiple sub-areas.

[0009] In the above implementation, after the number of clusters is set to the total number of brightness levels, the clustering algorithm can be used to automatically complete the region division. The clustering algorithm has good versatility and can be applied to facial images of different skin colors.

[0010] In an implementation of the first aspect, dividing the skin color area in the source image and the skin color area in the reference image into multiple sub-areas according to different brightness levels, includes: classifying the pixels in the skin color area of ​​the source image and the pixels in the skin color area of ​​the reference image according to a preset pixel value threshold, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into multiple sub-areas.

[0011] In the above implementation, pixel values ​​are classified by setting thresholds. The classification algorithm is simpler and more efficient to implement than the clustering algorithm, but its universality is not as good as the clustering algorithm. Different pixel value thresholds may need to be set according to people with different skin colors.

[0012] In an implementation of the first aspect, the multiple sub-regions include at least one sub-region to be corrected. After dividing the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before performing histogram matching on each sub-region in the source image with a sub-region with the same brightness level in the reference image, the method further includes: for each sub-region to be corrected in the source image and the reference image, calculating the connected domains it contains, and for each connected domain therein, determining whether its area is less than an area threshold; if it is less than the area threshold, reclassifying the connected domain as a sub-region in the corresponding image that is adjacent to the brightness level of the sub-region to be corrected.

[0013] Sub-regions may be misclassified. The inventors have found that in most lighting environments, sub-regions appear in large blocks on the face. Therefore, if the area of ​​the connected domain under a sub-region is very small, the sub-region division result of the connected domain is likely to be wrong and should be corrected. Correcting the sub-region division result is conducive to improving the accuracy of subsequent histogram matching of sub-regions.

[0014] The above implementation method provides a sub-region correction strategy, that is, for a certain connected domain, the sub-regions before and after the correction are adjacent in brightness level. This strategy is still based on the brightness of the pixel when performing sub-region correction, that is, only a limited adjustment is made to the original sub-region division result of the connected domain.

[0015] In an implementation of the first aspect, the multiple sub-regions include at least one sub-region to be corrected. After dividing the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before performing histogram matching on each sub-region in the source image with a sub-region of the same brightness level in the reference image, the method further includes: for each sub-region to be corrected in the source image and the reference image, calculating the connected domains it contains, and for each connected domain therein, determining whether its area is less than an area threshold; if it is less than the area threshold, reclassifying the connected domain as a sub-region located around it in the corresponding image.

[0016] The above implementation method provides a sub-region correction strategy, that is, for a certain connected domain, the sub-regions before and after the correction are adjacent in position. This strategy gives priority to the spatial distribution characteristics of the sub-region when performing sub-region correction, that is, the connected domains in the sub-region are usually distributed over a large area, so it is a reasonable choice to merge the connected domain with a smaller area into a surrounding sub-region with a larger area. This strategy allows for a larger adjustment of the original sub-region division results of the connected domain (across multiple brightness levels).

[0017] In an implementation manner of the first aspect, the multiple sub-areas include a shadow area, a normal skin color area, and a highlight area.

[0018] Low brightness (shadow), normal brightness (normal) and high brightness (highlight) are three typical brightness levels, which can effectively describe the brightness distribution in the skin color area in most cases. In addition, since the total number of brightness levels is small, it is also beneficial to reduce the amount of calculation in the skin color migration process.

[0019] In an implementation of the first aspect, after obtaining the migration result image, the method further includes: using a generative network in a pre-trained generative adversarial network to repair the migration result image to obtain a final migration result image.

[0020] The migration result image is only a relatively rough skin color migration result. There are many reasons, such as inaccurate segmentation results of skin color areas, inaccurate segmentation results of sub-areas, etc., which may cause some defects in the migration result image. In the above implementation, the migration result image is reconstructed by generating a network to repair these defects and obtain a more refined skin color migration result.

[0021] In a second aspect, an embodiment of the present application provides a skin color migration device, comprising: a skin color area acquisition module, used to perform image segmentation on a source image and a reference image, respectively, to obtain a skin color area in the source image and a skin color area in the reference image; a sub-area acquisition module, used to divide the skin color area in the source image and the skin color area in the reference image into multiple sub-areas according to different brightness levels; a histogram matching module, used to perform histogram matching on each sub-area in the source image with a sub-area of ​​the same brightness level in the reference image, respectively, to obtain a migration result image.

[0022] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the method provided by the first aspect or any possible implementation of the first aspect is executed.

[0023] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and run by the processor, the method provided by the first aspect or any possible implementation of the first aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A possible process of the skin color migration method provided by an embodiment of the present application is shown;

[0026] Figure 2 A possible structure of the skin color migration device provided in an embodiment of the present application is shown;

[0027] Figure 3 A possible structure of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] Skin color transfer is an important task in makeup transfer, because the skin color may change before and after makeup, so the skin color in the reference image is usually considered as part of the makeup and involved in the transfer process. Experimental results show that when the proportion of shadow area and / or highlight area in the source image is significantly different from that in the reference image, the existing skin color transfer methods often have poor transfer effects.

[0029] After long-term research, the inventors found that the reason for this phenomenon is that the existing methods generally achieve skin color migration by directly performing histogram matching between the source image and the reference image. The main function of the histogram matching algorithm is to make the cumulative histogram of the source image (after matching) as close as possible to the cumulative histogram of the reference image (of course, if the cumulative histograms are similar, the histograms are not necessarily the same). Figure 1 From an intuitive point of view, the skin color of the person in the source image becomes closer to that in the reference image.

[0030] The steps of the conventional histogram matching algorithm are as follows:

[0031] (1) Calculate the cumulative histogram Hs of the source image Is;

[0032] (2) Calculate the cumulative histogram Hr of the reference image Ir;

[0033] (3) For the value corresponding to each gray level g in Hs, calculate the difference (absolute value) between it and the value corresponding to each gray level in Hr. Note that in steps (1) and (2), the value corresponding to each gray level in the cumulative histogram has been mapped to the interval [0,1].

[0034] (4) Find the minimum value of the difference in step (3) and determine the gray level G corresponding to the minimum value in Hr, so that the gray level mapping relationship g→G between Hs and Hr is established;

[0035] (5) Replace the pixels with gray level g in Is with pixels with gray level G to obtain the histogram matching result.

[0036] The source image and reference image in the above steps can be understood as grayscale images. If they are RGB images, just execute the above algorithms separately according to the corresponding color channels.

[0037] If the proportion of shadow areas and / or highlight areas in the source image is significantly different from that in the reference image, for example, there is almost no shadow area in the source image, but there are a large number of shadow areas in the reference image, according to the algorithm steps, a mapping relationship may be established between a large number of pixels of normal brightness (the brightness mentioned here can be represented by grayscale) in the source image and a large number of pixels of low brightness in the reference image, so that after histogram matching, a large number of shadows may appear in the source image, and the shadows are scattered in various parts of the face (because the pixels of normal brightness are distributed in these positions before matching), causing the character to have a "painted face", seriously affecting the skin color migration effect.

[0038] Based on the above findings, the embodiments of the present application provide a skin color migration method and device, a storage medium and an electronic device. The technical solution uses regional histogram matching to replace the traditional histogram matching algorithm, which effectively improves the above technical problems and improves the effect of skin color migration.

[0039] It should be pointed out that, in addition to the technical solution itself, the above analysis of the reasons for the poor skin color migration effect is also the conclusion obtained by the inventor during the research process, rather than the result already existing in the prior art, and therefore should also be regarded as the contribution made by the inventor to the present invention.

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0041] The terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0042] The terms "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and should not be understood as indicating or implying relative importance, nor should they be understood as requiring or implying any such actual relationship or order between these entities or operations.

[0043] Figure 1 A possible process of the skin color migration method provided in the embodiment of the present application is shown. The method can be but is not limited to: Figure 3The electronic device shown is executed, and its specific structure can be referred to in the following Figure 3 Reference Figure 1 , the method comprising:

[0044] Step S110: performing image segmentation on the source image and the reference image respectively to obtain the skin color area in the source image and the skin color area in the reference image.

[0045] The source image can be an image of the user, the reference image can be an image of a model, and the goal of skin color transfer is to transfer the skin color of the reference image to the source image. Ideally, the source image and the reference image will have similar skin colors after the transfer. Skin color transfer can be used as a task in makeup transfer, but it is not excluded that it can be applied to other fields. There is no limit to the acquisition method of the source image and the reference image, and there is no limit to the image type (color or black and white).

[0046] The skin color area refers to the area covered by the skin in the image. The method of segmenting the skin color area from the source image and the reference image is similar. Taking the source image as an example, the segmentation result contains the position information of the skin color area. For example, the segmentation result can be a binary mask, and the pixels with a value of 1 in the mask correspond to the pixels belonging to the skin color area in the source image, and the pixels with a value of 0 correspond to the pixels that do not belong to the skin color area in the source image; for another example, the segmentation result can record the boundary position information of the skin color area, and the position of the skin color area in the source image can be determined based on the boundary position information. The methods for segmenting skin color areas include at least the following categories:

[0047] (1) Traditional image segmentation algorithms; for example, image segmentation algorithms based on color space.

[0048] (2) Image segmentation algorithm based on deep learning; for example, using a pre-trained convolutional neural network to segment the image, the neural network can adopt a structure such as UNet, SegNet, etc.

[0049] (3) Image segmentation algorithm based on feature point detection; for example, first detect the facial feature points in the image, and then connect the facial feature points in a preset order to form a closed area, that is, the skin color area. When connecting the facial feature points, non-skin color areas such as eyes and mouth should be excluded from the skin color area.

[0050] Step S120: Divide the skin color area in the source image and the skin color area in the reference image into a plurality of sub-areas according to different brightness levels.

[0051] The total number of brightness levels is not limited, but should be at least 2 levels, and the brightness levels used to divide the sub-regions are consistent for the source image and the reference image. For example, if the source image is divided into sub-regions according to brightness level 1 and brightness level 2 (two different brightness levels), the reference image must also be divided into sub-regions according to brightness level 1 and brightness level 2 (consistent with the brightness levels used to divide the source image), and the sub-regions cannot be divided according to brightness level 1, brightness level 2, and brightness level 3 (inconsistent with the total number of brightness levels used to divide the source image), nor can the sub-regions be divided according to brightness level 2 and brightness level 3 (consistent with the total number of brightness levels used to divide the source image, but the specific levels are inconsistent).

[0052] Each sub-region obtained by division has different brightness levels. More specifically, it means that the pixels contained in each sub-region have different brightness levels, and the brightness level corresponding to the pixel depends on the pixel value. In most cases, the total number of brightness levels is the same as the total number of sub-regions divided from the skin color region, but it does not rule out that there are no pixels of a certain brightness level in the skin color region. In this case, the corresponding sub-region cannot be divided for this brightness level, so the total number of sub-regions may also be less than the total number of brightness levels. This situation can also be regarded as a special case where one or several sub-regions do not contain pixels. For the sake of simplicity, it will not be the focus of the following text.

[0053] In some implementations, the brightness levels in step S120 include three, namely low brightness, normal brightness and high brightness. According to these three brightness levels, three corresponding sub-areas can be divided from the skin color area of ​​the source image and the reference image, namely the shadow area, the normal skin color area and the highlight area (these three sub-areas are divided in the source image and the reference image). In most cases, the three brightness levels can effectively describe the brightness distribution in the skin color area, and because the total number of brightness levels is small, it is also helpful to reduce the amount of calculation in the skin color migration process. Of course, it is also possible to set more or fewer brightness levels, but for simplicity, the following text mainly takes the above three brightness levels as an example.

[0054] The following are two ways to divide sub-regions. It is understandable that there may be other ways to divide sub-regions:

[0055] (1) The number of clusters is set to the total number of brightness levels, and the pixels in the skin color area of ​​the source image and the pixels in the skin color area of ​​the reference image are clustered respectively, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into multiple sub-areas.

[0056] The specific clustering algorithm is not limited, for example, k-means clustering, Gaussian mixture clustering, hierarchical clustering and other algorithms can be used.

[0057] For example, if the number of clusters is set to 3, the skin color area can be divided into a shadow area, a normal skin color area, and a highlight area.

[0058] The clustering algorithm has good versatility and can be applied to facial images of different skin colors. However, it should be pointed out that the brightness levels of the sub-regions divided by the clustering algorithm are relative. For example, the shadow area refers to the area with relatively low brightness in the skin color area, but its absolute brightness is not necessarily very low.

[0059] (2) According to a preset pixel value threshold, pixels in the skin color area of ​​the source image and pixels in the skin color area of ​​the reference image are classified respectively, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into a plurality of sub-areas.

[0060] For example, if there are three brightness levels, two pixel value thresholds T1 and T2 can be set. Assuming that the brightness value of a pixel in the skin color area (which can be taken as the pixel value or calculated based on the pixel value) is P, if P<T1, it is divided into the shadow area; if T1≤P<T2, it is divided into the normal skin color area; if P>T2, it is divided into the highlight area.

[0061] Compared with clustering algorithms, threshold-based classification algorithms are simpler and more efficient to implement, but they are not as universal as clustering algorithms. Different pixel value thresholds may need to be set according to people with different skin colors. In addition, the brightness levels of the sub-regions divided by the classification algorithm are absolute. For example, the shadow area refers to the area with lower brightness in the skin color area. If the brightness of the entire skin color area is relatively high, the shadow area may not be divided.

[0062] The divided sub-regions can be represented in a similar manner to the skin color region. For example, a corresponding binary mask can be generated for each sub-region in the skin color region of the source image. Pixels with a value of 1 in the mask correspond to pixels in the source image that belong to the sub-region, and pixels with a value of 0 correspond to pixels in the source image that do not belong to the sub-region.

[0063] In some implementations, step S130 may be executed immediately after step S120 is executed; in other implementations, some or all of the sub-regions obtained in step S120 may be corrected before executing step S130. The sub-regions to be corrected may be referred to as sub-regions to be corrected. If the sub-regions to be corrected are not all the sub-regions obtained in step S120, it is possible to specify which sub-regions need to be corrected according to actual needs.

[0064] Correction can be performed on the sub-regions included in the skin color region of the source image and the sub-regions included in the skin color region of the reference image. Optionally, the sub-region to be corrected specified in the source image and the sub-region to be corrected specified in the reference image have the same brightness level, that is, the processing method of the source image and the reference image is consistent. For example, the shadow area and the highlight area in the source image and the reference image are corrected, and the normal skin color area in the source image and the reference image is not corrected. For simplicity, when introducing the correction method of the sub-region below, it is not distinguished whether the sub-region belongs to the source image or the reference image, because the processing method of the two is similar.

[0065] First, the concept of connected domain in the field of image processing is introduced. A connected domain is a region composed of pixels in an image. Any two pixels in the same connected domain are connected. The so-called connection between two pixels means that starting from one of the pixels, you always look for the next pixel from the adjacent pixels, and you can get at least one path to the other pixel. The common adjacency relationships between pixels are 4-adjacency and 8-adjacency. 4-adjacency means that a pixel is only adjacent to the 4 pixels above, below, left and right, and 8-adjacency means that a pixel is adjacent to the 8 pixels surrounding it. The basic unit of sub-region correction is the connected domain contained in the sub-region. That is, to correct a sub-region, it is actually to correct one or more connected domains currently contained in the sub-region. The process is: for the sub-region to be corrected, first determine the connected domains it contains, and then for each connected domain, determine whether it meets the preset correction conditions. If the correction conditions are met, reset the sub-region to which the connected domain belongs. It can be understood that each sub-region may contain multiple connected domains. For example, the highlight area on the human face may include four connected domains, namely the highlight on the forehead, the highlight on the left cheek, the highlight on the right cheek, and the highlight on the tip of the nose. If the highlight area is selected as the sub-region to be corrected, it is necessary to examine whether these four connected domains meet the correction conditions.

[0066] By performing connectivity detection on the pixels inside the sub-region, the connected domain contained in the sub-region can be determined. For example, if the sub-region is represented by a binary mask, algorithms such as Two-Pass and Seed-Filling can be used to detect all connected domains in the sub-region.

[0067] The reason for sub-region correction is that the sub-region may be misdivided. The inventors have found that in most lighting environments, sub-regions appear in large blocks on the face. For example, when a strong light source shines on the face from the left, a large shadow will appear on the right face, that is, the shadow area contains multiple large connected domains. Therefore, if the area of ​​a connected domain under the sub-region is very small, so that it is less than a certain area threshold, then it is likely that the sub-region division result of the connected domain is wrong and should be corrected. The "less than a certain area threshold" here refers to the correction condition mentioned above. The area threshold can be a value in an absolute sense, for example, 100 pixels, or it can be a value in a relative sense, such as 5% of the skin color area, and so on.

[0068] There are many reasons for the occurrence of misdivision: for example, some pixels in the image are originally noise, and their pixel values ​​cannot represent the true skin color; for another example, there is a spot on the skin of the face, and its color is darker than the surrounding skin. Therefore, based only on the pixel value of the spot, the connected domain where the spot is located will be judged as a shadow area, but in fact the spot is not necessarily in the shadow area on the face, etc. Misdivision of sub-regions may affect the accuracy of subsequent histogram matching, so it can be corrected in advance.

[0069] After introducing the basic concept of sub-region correction, two specific correction strategies are given below. It is understandable that other strategies may be adopted to correct sub-regions:

[0070] (1) If the area of ​​a connected domain in a sub-region to be corrected is smaller than the area threshold, the connected domain is reclassified as a sub-region in the image (referring to the image where the sub-region to be corrected is located) whose brightness level is adjacent to that of the sub-region to be corrected.

[0071] The brightness levels corresponding to the multiple sub-regions may be sorted in order from low brightness to high brightness (or vice versa). After the sorting, for any sub-region, its adjacent sub-region in terms of brightness level may be determined.

[0072] For example, for a source image, if the area of ​​a connected domain in the shadow region is smaller than an area threshold, the connected domain is reclassified as a normal skin color region in the source image, because the normal skin color region and the shadow region are adjacent in terms of brightness level.

[0073] For another example, for a source image, if the area of ​​a connected domain in the highlight region is smaller than an area threshold, the connected domain is reclassified as a normal skin color region in the source image, because the normal skin color region and the highlight region are adjacent in terms of brightness level.

[0074] For another example, for a source image, if the area of ​​a connected domain in the normal skin color area is smaller than the area threshold, the connected domain is reclassified as a shadow area or a highlight area in the source image, because the shadow area and the highlight area are adjacent to the normal skin color area in terms of brightness level. As for whether to classify as a shadow area or a highlight area, further judgment can be made: for example, the judgment can be made using the method in strategy (2); for another example, judging whether the brightness of the current connected domain (which can be defined as the average brightness of each pixel in the connected domain) is closer to the brightness of the shadow area or the brightness of the highlight area. If it is closer to the brightness of the shadow area, the connected domain is reclassified as a shadow area in the source image; otherwise, the connected domain is reclassified as a highlight area in the source image.

[0075] In strategy (1), for a certain connected domain, the sub-regions to which it belongs before and after correction are adjacent in terms of brightness level. This indicates that strategy (1) is still based on the brightness of the pixels when performing sub-region correction, that is, the original sub-region division result (divided according to the brightness of the pixels) is only adjusted to a limited extent. Because according to the inventor's research, even if there are errors in the original sub-region division result, they are generally not too serious, and there are rarely errors in the division across brightness levels. For example, it is possible to mistakenly classify a connected region that originally belongs to the shadow area as a normal skin color area, but it is very rare to mistakenly classify the connected region as a highlight area. Therefore, correction across brightness levels should not be performed lightly.

[0076] (2) If the area of ​​a connected domain in a sub-region to be corrected is smaller than the area threshold, the connected domain is reclassified as a sub-region located around it in the image (referring to the image where the sub-region to be corrected is located).

[0077] For example, for the source image, if the area of ​​a connected domain in the shadow area is smaller than the area threshold, and the connected domain is surrounded by a normal skin color area (a small connected domain is most likely surrounded by a sub-region), then the connected domain is reclassified as a normal skin color area in the source image.

[0078] For another example, for a source image, if the area of ​​a connected domain in the normal skin color area is smaller than the area threshold and the connected domain is surrounded by a highlight area, the connected domain is reclassified as a highlight area in the source image.

[0079] For another example, for a source image, if the area of ​​a connected domain in the shadow area is smaller than the area threshold and the connected domain is surrounded by a highlight area, the connected domain is reclassified as a highlight area in the source image.

[0080] For another example, for a source image, if the area of ​​a connected domain in the shadow region is smaller than the area threshold, and the connected domain is adjacent to both the shadow region and the highlight region, then the connected domain is reclassified as the shadow region or the highlight region in the source image. As for whether to classify it as a shadow region or a highlight region, further judgment can be made, and the relevant description in strategy (1) can be referred to.

[0081] In strategy (2), for a connected domain, the sub-regions to which it belongs before and after correction are adjacent in position. This shows that strategy (2) gives priority to the spatial distribution characteristics of the sub-region when performing sub-region correction, that is, the connected domains in the sub-region are usually distributed over a large area, so it is a reasonable choice to merge a connected domain with a smaller area into a sub-region with a larger surrounding area. This strategy allows for a larger adjustment of the original sub-region division results of the connected domain (across multiple brightness levels, such as correcting the shadow area to the highlight area, crossing the brightness level corresponding to the normal skin color area).

[0082] Step S130: performing histogram matching on each sub-region in the source image and the sub-region with the same brightness level in the reference image to obtain a migration result image.

[0083] For example, in the case where the skin color areas in the source image and the reference image both include shadow areas, normal skin color areas, and highlight areas, step S130 refers to matching the shadow area of ​​the source image with the shadow area of ​​the reference image, matching the normal skin color area of ​​the source image with the normal skin color area of ​​the reference image, and matching the highlight area of ​​the source image with the highlight area of ​​the reference image.

[0084] The histogram matching algorithm between each two regions is the histogram matching algorithm introduced in the previous article, and will not be repeated here. Each sub-region in the source image corresponds to a histogram matching result. The histogram matching result maintains the original shape of the sub-region, but the pixel values ​​therein have been mapped. The image composed of all the histogram matching results is the migration result image of skin color migration. It should be pointed out that from the storage perspective, in different implementation methods, the migration result image may be the same image as the source image (of course, the pixel values ​​have changed after skin color migration), or it may be a newly created image (the pixel values ​​in the source image remain unchanged after skin color migration).

[0085] In summary, the skin color migration method provided in the embodiment of the present application does not perform histogram matching between the entire skin color area in the source image and the reference image, but further divides the skin color area into multiple sub-areas according to the brightness of the pixels, and only performs histogram matching between the sub-areas with the same brightness level in the source image and the reference image. This method of sub-area matching improves the accuracy of histogram matching, so that the brightness of the pixels in the source image before and after matching will not change across levels, thereby effectively improving the situation where the skin color migration effect is poor due to the large difference between the proportion of the shadow area and / or the highlight area in the source image and the proportion of the shadow area and / or the highlight area in the reference image.

[0086] Continuing with the example mentioned above, there are almost no shadow areas in the source image, but there are a large number of shadow areas in the reference image. According to the region-based histogram matching algorithm in this application, a large number of pixels with normal brightness in the source image will only establish a mapping relationship with pixels with normal brightness in the reference image, but will not establish a mapping relationship with a large number of pixels with low brightness in the reference image. Therefore, after the histogram matching, there will not be a large number of shadows in the source image, and the skin color migration effect is better.

[0087] Furthermore, the migration result image in step S130 is only a relatively rough skin color migration result. There are many reasons, such as inaccurate segmentation results of skin color areas (such as misclassification or omission), inaccurate segmentation results of sub-areas (such as misclassification), etc., which may cause some defects in the migration result image. Therefore, in some implementations, after executing step S130, the generative network in the pre-trained generative adversarial network (GAN) can be used to repair these defects in the migration result image to obtain the final migration result image. Compared with the migration result image, the final migration result image is a more detailed skin color migration result, which can be used to display to the user.

[0088] Among them, the generative adversarial network includes a generative network and a discriminator network. Both parts are included during training, but after training, only the generative network is used. The generative network takes the migration result image as input and outputs the repaired final migration result image.

[0089] Figure 2 FIG. 2 shows a functional module diagram of a skin color migration device 200 provided in an embodiment of the present application. Figure 2 , the skin color migration device 200 comprises:

[0090] A skin color region acquisition module 210, configured to perform image segmentation on a source image and a reference image respectively, to obtain a skin color region in the source image and a skin color region in the reference image;

[0091] A sub-region acquisition module 220, configured to divide the skin color region in the source image and the skin color region in the reference image into a plurality of sub-regions according to different brightness levels;

[0092] The histogram matching module 230 is used to perform histogram matching on each sub-region in the source image and a sub-region with the same brightness level in the reference image to obtain a migration result image.

[0093] In one implementation of the skin color migration device 200, the sub-region acquisition module 220 divides the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, including: setting the number of clusters to the total number of the brightness levels, and clustering the pixels in the skin color region of the source image and the pixels in the skin color region of the reference image, so that the skin color region in the source image and the skin color region in the reference image are respectively divided into multiple sub-regions.

[0094] In one implementation of the skin color migration device 200, the sub-region acquisition module 220 divides the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, including: classifying the pixels in the skin color region of the source image and the pixels in the skin color region of the reference image according to a preset pixel value threshold, so that the skin color region in the source image and the skin color region in the reference image are respectively divided into multiple sub-regions.

[0095] In one implementation of the skin color migration device 200, the multiple sub-regions include at least one sub-region to be corrected, and the device also includes: a sub-region correction module, which is used to calculate the connected domains contained in each sub-region to be corrected in the source image and the reference image after the sub-region acquisition module 220 divides the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before the histogram matching module 230 performs histogram matching on each sub-region in the source image with the sub-region of the same brightness level in the reference image, and for each connected domain, determine whether its area is less than an area threshold; if it is less than the area threshold, reclassify the connected domain as a sub-region in the corresponding image that is adjacent to the brightness level of the sub-region to be corrected.

[0096] In one implementation of the skin color migration device 200, the multiple sub-regions include at least one sub-region to be corrected, and the device also includes: a sub-region correction module, which is used to calculate the connected domains contained in each sub-region to be corrected in the source image and the reference image after the sub-region acquisition module 220 divides the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before the histogram matching module 230 performs histogram matching on each sub-region in the source image with the sub-region of the same brightness level in the reference image, and for each connected domain, determine whether its area is less than an area threshold, and if so, reclassify the connected domain as a sub-region located around it in the corresponding image.

[0097] In one implementation of the skin color migration device 200, the multiple sub-areas include a shadow area, a normal skin color area, and a highlight area.

[0098] In one implementation of the skin color migration device 200, the device also includes: a migration result repair module, which is used to repair the migration result image using a generative network in a pre-trained generative adversarial network after the histogram matching module 230 obtains the migration result image to obtain a final migration result image.

[0099] The skin color migration device 200 provided in the embodiment of the present application, its implementation principle and the technical effects produced have been introduced in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the method embodiment.

[0100] Figure 3 FIG. 3 shows a possible structure of an electronic device 300 provided in an embodiment of the present application. Figure 3 The electronic device 300 includes: a processor 310, a memory 320 and a communication interface 330. These components are interconnected and communicate with each other through a communication bus 340 and / or other forms of connection mechanisms (not shown).

[0101] Among them, the processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 310 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a dedicated processor, including a graphics processor (GPU), a neural network processor (NPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. In addition, when there are multiple processors 310, one of them can be a general-purpose processor, and the other part can be a dedicated processor.

[0102] The memory 320 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor 310 and other possible components may access the memory 320 and read and / or write data therein.

[0103] In particular, one or more computer program instructions may be stored in the memory 320, and the processor 310 may read and execute these computer program instructions to implement the skin color migration method provided in the embodiment of the present application.

[0104] The communication interface 330 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices to exchange data. The communication interface 330 can include an interface for wired and / or wireless communication.

[0105] Understandably, Figure 3 The structure shown is for illustration only. The electronic device 300 may also include Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown. Figure 3 Each component shown in can be implemented by hardware, software or a combination thereof. The electronic device 300 may be a physical device, such as a PC, a laptop, a tablet computer, a mobile phone, a server, an embedded device, etc., or a virtual device, such as a virtual machine, a virtualized container, etc. Moreover, the electronic device 300 is not limited to a single device, but may also be a combination of multiple devices or a cluster consisting of a large number of devices.

[0106] The present application also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a computer processor, the skin color migration method provided by the present application is executed. For example, the computer-readable storage medium can be implemented as Figure 3 The memory 320 in the electronic device 300.

[0107] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A skin color migration method, characterized in that: include: Performing image segmentation on the source image and the reference image respectively to obtain a skin color area in the source image and a skin color area in the reference image; Dividing the skin color area in the source image and the skin color area in the reference image into a plurality of sub-areas according to different brightness levels respectively; Histogram matching is performed on each sub-region in the source image and a sub-region with the same brightness level in the reference image to obtain a migration result image.

2. The skin color migration method according to claim 1, characterized in that: The step of dividing the skin color area in the source image and the skin color area in the reference image into a plurality of sub-areas according to different brightness levels respectively includes: The number of clusters is set to the total number of the brightness levels, and the pixels in the skin color area of ​​the source image and the pixels in the skin color area of ​​the reference image are clustered respectively, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into multiple sub-areas.

3. The skin color migration method according to claim 1, characterized in that: The step of dividing the skin color area in the source image and the skin color area in the reference image into a plurality of sub-areas according to different brightness levels respectively includes: According to the pixel value threshold, the pixels in the skin color area of ​​the source image and the pixels in the skin color area of ​​the reference image are classified respectively, so that the skin color area in the source image and the skin color area in the reference image are respectively divided into a plurality of sub-areas.

4. The skin color migration method according to any one of claims 1 to 3, characterized in that: The multiple sub-regions include at least one sub-region to be corrected. After dividing the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before performing histogram matching on each sub-region in the source image and a sub-region with the same brightness level in the reference image, the method further includes: For each sub-region to be corrected in the source image and the reference image, the connected domains it contains are calculated. For each connected domain, whether its area is less than the area threshold is determined. If it is less than the area threshold, the connected domain is reclassified as a sub-region in the corresponding image that is adjacent to the brightness level of the sub-region to be corrected.

5. The skin color migration method according to any one of claims 1 to 3, characterized in that: The multiple sub-regions include at least one sub-region to be corrected. After dividing the skin color region in the source image and the skin color region in the reference image into multiple sub-regions according to different brightness levels, and before performing histogram matching on each sub-region in the source image and a sub-region with the same brightness level in the reference image, the method further includes: For each sub-region to be corrected in the source image and the reference image, the connected domains it contains are calculated. For each connected domain, whether its area is less than an area threshold is determined. If it is less than the area threshold, the connected domain is reclassified as a sub-region located around it in the corresponding image.

6. The skin color migration method according to any one of claims 1 to 3, characterized in that: The plurality of sub-areas include a shadow area, a normal skin color area, and a highlight area.

7. The skin color migration method according to any one of claims 1 to 3, characterized in that: After obtaining the migration result image, the method further includes: The migration result image is repaired by using a generative network in a pre-trained generative adversarial network to obtain a final migration result image.

8. A skin color migration device, characterized in that: include: A skin color region acquisition module, used to perform image segmentation on a source image and a reference image respectively, to obtain a skin color region in the source image and a skin color region in the reference image; A sub-region acquisition module, used for dividing the skin color region in the source image and the skin color region in the reference image into a plurality of sub-regions according to different brightness levels; The histogram matching module is used to perform histogram matching on each sub-region in the source image and the sub-region with the same brightness level in the reference image to obtain a migration result image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and executed by the processor, the method according to any one of claims 1 to 7 is executed.

Images