Human skin segmentation method and related device

Abstract

The invention provides a human skin segmentation method and a related device. According to the method, region division is carried out on a human face part through detected human face key points, and pixel information of each region is counted. And the pixel information of the face region which is slightly different from the pixel information of the whole face part is selected as the accurate skin color information, so that the accuracy of the skin color information is improved. Moreover, for the discrete skin color Mask blocks, comparing whether the pixel information of each discrete skin color Mask block and the accurate skin color information are in a tolerance threshold range, and if so, determining that the Mask block is a skin color area and retaining the Mask block; and if not, further clustering the Mask block to obtain a near-skin-color region and / or a difference region. And clustering the difference regions again to obtain final near-skin-color regions and / or difference regions, retaining the near-skin-color regions, and excluding the difference regions. And after all the discrete Mask blocks are processed, all the skin color Mask blocks possibly having wrong segmentation are excluded, so that an accurate skin color region is obtained, and the accuracy of skin color segmentation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a human skin segmentation method and related device. BACKGROUND

[0002] Terminal equipment has become one of the widely used shooting tools by users. In order to improve the user's shooting experience, the clarity and beautification processing of the face region have become the focus of the manufacturers of terminal equipment. Therefore, the terminal equipment can provide a more high-definition and aesthetic portrait photo, which has a great effect on the user's shooting experience.

[0003] At present, a processing method for improving image effects is to perform corresponding processing on the human skin region in the image, such as enhancing and beautifying the pixels in the skin region, and the noise and background of the non-skin region remain unchanged. This processing method needs to segment the human skin in the image first. However, the widely used segmentation algorithm based on neural network may have the problem of missegmentation, for example, dividing the non-skin color region in the image into a skin color region. This erroneous segmentation will cause the subsequent processing algorithm to perform corresponding effect improvement processing on the missegmented region, resulting in a patch effect (such as the pixels in the region being brighter and whiter) in the missegmented region. Therefore, there is an urgent need for a skin color segmentation method with high accuracy. SUMMARY

[0004] Therefore, the present application provides a human skin segmentation method and related device to solve at least part of the above problems. The technical solutions disclosed in the present application are as follows:

[0005] In a first aspect, the present application provides a human skin segmentation method applied to an electronic device, the method comprising: performing skin segmentation on a to-be-processed image containing a face to obtain a skin color Mask image; performing face key point detection on the to-be-processed image, dividing the detected face key points into regions, and obtaining pixel information (such as including the mean gray value and the gray variance) of each face region; obtaining the average value (such as the mean average value and the variance average value) of the pixel information based on the pixel information of each face region, and determining the pixel information corresponding to the face region with the minimum difference between the pixel information and the average value of the pixel information as the accurate skin color information; identifying discrete skin color Mask blocks in the skin color Mask image that are not in the face region based on the face key point detection result; for any discrete skin color Mask block, judging whether the entire discrete skin color Mask block belongs to the skin color region based on the accurate skin color information and the pixel information of the discrete skin color Mask block, and if so, retaining the discrete skin color Mask block; if not, clustering each pixel point in the discrete skin color Mask block to obtain a first near-skin color region and / or a first difference region; re-clustering each pixel point in the first difference region to obtain a second near-skin color region and / or a second difference region; excluding the second difference region from the skin color region of the skin color Mask image to obtain an accurate skin color region.

[0006] It can be seen that the method divides the face part into regions based on the detected face key points, and further obtains the average value of the pixel information of the entire face part based on the pixel information of each region. Selecting the pixel information of the face region with a smaller difference from the average value of the pixel information of the entire face part as the accurate skin color information can avoid the influence of the large skin color difference between different regions of the face part caused by the occlusion of the face part region, such as the user wearing a mask, thereby improving the accuracy of the skin color information.

[0007] Moreover, the method identifies discrete skin color Mask blocks in the skin color Mask image that are not in the face region based on the face key point detection result. For the discrete skin color Mask blocks, the pixel information of each discrete skin color Mask block is compared with the accurate skin color information to determine whether they are within a tolerance threshold, and if so, the Mask block is determined to be a skin color region and retained; if not, the Mask block is further binarized and clustered to obtain a near-skin color region and / or a difference region. The difference region is re-clustered to obtain the final near-skin color region and / or difference region, and the near-skin color region is retained and the difference region is excluded. After all discrete Mask blocks are processed, all skin color Mask blocks that may be misclassified are excluded, thereby obtaining an accurate skin color region, i.e., the method improves the accuracy of skin color segmentation. Further, the processing effect of other processes based on the skin color region is improved.

[0008] In a possible implementation manner of the first aspect, the region is divided according to the detected face key points, including: connecting the face key points in the preset positions in the face to obtain the face region corresponding to the preset positions, the preset positions including eye preset ranges, cheeks, chins and / or side faces. In this way, after the region is divided, it can be avoided that the skin color difference of different regions of the face is large due to the fact that part of the face region is blocked, for example, the user wears a mask, and the influence on the skin color information.

[0009] In a possible implementation manner of the first aspect, the pixel information of each face region is obtained, including: calculating the average value of the gray values of the pixels in each face region to obtain the pixel information of each face region. The average value of the gray values of the pixels can reflect the color information of the pixels in the face region, and the variance of the gray values of the pixels can reflect the difference between different pixels in the region. Therefore, the average value of the gray values of the pixels and the variance of the gray values of the pixels can comprehensively represent the information of the pixels in the face region.

[0010] In a possible implementation manner of the first aspect, the average value of the pixel information is obtained based on the pixel information of each face region, including: calculating the average value of the average values of the gray values of the pixels in each face region to obtain the average value of the average values (i.e., the average value of the average values of the gray values of the pixels in each face region), and calculating the average value of the variances of the gray values of the pixels in each face region to obtain the average value of the variances (i.e., the average value of the variances of the gray values of the pixels in each face region).

[0011] In a possible implementation manner of the first aspect, the pixel information corresponding to the face region with the minimum difference between the pixel information and the average value of the pixel information is determined as the accurate skin color information, including: sequentially calculating the average value difference between the average value of the gray values of the pixels in each face region and the average value of the average values; sequentially calculating the variance difference between the variance of the gray values of the pixels in each face region and the average value of the variances; and determining the pixel information corresponding to the face region with the minimum average value difference and the variance difference within a preset range as the accurate skin color information. It can be seen that even if part of the region where the face is located is blocked, the blocking is usually local and the area ratio is small, and therefore the average value of the pixel information of the whole face part can reflect the skin color information of the face. Further, the region with the minimum difference between the average value of the gray values of the pixels and the average value of the pixel information of the whole face part is selected as the accurate skin color information, thereby improving the accuracy of the skin color information.

[0012] In a possible implementation manner of the first aspect, the pixel information corresponding to the face region with the minimum difference between the pixel information and the average value of the pixel information is determined as the accurate skin color information, including: determining all candidate face regions with the average value of the variances within a first preset range; and selecting the pixel information corresponding to the candidate face region with the minimum average value difference in the order from small to large according to the average value difference of each candidate face region as the accurate skin color information.

[0013] In a further possible implementation form of the first aspect, determining whether the any discrete skin color Mask block as a whole belongs to the skin color region based on the accurate skin color information and the pixel information of the any discrete skin color Mask block comprises: calculating a first difference between a mean gray value in the accurate skin color information and a mean gray value of the any discrete skin color Mask block, and calculating a second difference between a gray variance in the accurate skin color information and a gray variance of the any discrete skin color Mask block; determining whether the first difference is within a second preset range and whether the second difference is within a third preset range, and if so, determining that the any discrete skin color Mask block as a whole belongs to the skin color region; and if the first difference is out of the second preset range and / or the second difference is out of the third preset range, determining that the any discrete skin color Mask block as a whole does not belong to the skin color region. In this way, for each divided region, the difference between the region and the accurate skin color information is compared as a whole, and if the difference is small, the region can be directly retained. If the difference is large, the pixel points in the region can be clustered into a near-skin color region and / or a difference region, thereby improving the recognition efficiency of the discrete skin color Mask block.

[0014] In a further possible implementation form of the first aspect, clustering the pixel points in the any discrete skin color Mask block to obtain the first near-skin color region and / or the first difference region comprises: randomly selecting a pixel point with a gray value close to the mean gray value in the accurate skin color information as a seed point from the any discrete skin color Mask block; for any other pixel point in the any discrete skin color Mask block except the seed point, comparing a difference between a gray value of the any other pixel point and a gray value of the seed point with a preset threshold, and if the difference is less than the preset threshold, determining that the any other pixel point belongs to the first near-skin color region, and if the difference is greater than or equal to the preset threshold, determining that the any other pixel point belongs to the first difference region. In this way, by clustering the pixel points in the region with large difference from the accurate skin color information, the part of the region belonging to the skin can be accurately identified, and thus the accuracy of the skin color region is improved.

[0015] In a further possible implementation form of the first aspect, clustering the pixel points in the any discrete skin color Mask block to obtain the first near-skin color region and / or the first difference region comprises: randomly selecting a pixel point with a gray value close to the mean gray value in the accurate skin color information as a seed point from the any discrete skin color Mask block; for any other pixel point in the any discrete skin color Mask block except the seed point, calculating a distance between the any other pixel point and the seed point, and obtaining a weight value matched with the distance; calculating a first gray difference between the any other pixel point and the seed point, and calculating a product of the first gray difference and the weight value to obtain a second gray difference; and if the second gray difference is less than a preset threshold, determining that the any other pixel point belongs to the first near-skin color region, and if the second gray difference is greater than or equal to the preset threshold, determining that the any other pixel point belongs to the first difference region.

[0016] In another possible implementation of the first aspect, excluding the second difference region from the skin color region of the skin color mask image to obtain the accurate skin color region includes: retaining the face region in the skin color mask image, and retaining the first near skin color region and the second near skin color region, excluding the second difference region, to obtain the accurate skin color region in the skin color mask image.

[0017] In a second aspect, this application also provides an electronic device, which includes: one or more processors, a memory, and a touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device implements the human skin segmentation method as described in any of the first aspects.

[0018] Thirdly, this application also provides a computer-readable storage medium having instructions stored thereon that, when executed on an electronic device, cause the electronic device to perform the human skin segmentation method as described in any of the first aspects.

[0019] Fourthly, this application also provides a computer program product, characterized in that it stores instructions that, when the computer program product is run on an electronic device, cause the electronic device to implement the human skin segmentation method as described in any of the first aspects.

[0020] Fifthly, this application also provides a chip system, comprising: at least one processor and an interface, the interface being used to receive code instructions and transmit them to the at least one processor; the at least one processor executing the code instructions to implement the human skin segmentation method of any one of the first aspects. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the processing procedure of a human skin segmentation method provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart of a human skin segmentation method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram showing a comparison of human skin segmentation effects provided in an embodiment of this application;

[0026] Figure 6 This is another comparative diagram of human skin segmentation effects provided in the embodiments of this application;

[0027] Figure 7 This is another comparative diagram of human skin segmentation effects provided in the embodiments of this application. Detailed Implementation

[0028] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0029] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] The human skin segmentation method provided in this application is applicable to electronic devices, which may include handheld devices with display functions, in-vehicle devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, and augmented reality (AR) devices; this application does not limit these to a specific type. Furthermore, in this application, the electronic device can also be a terminal device in an Internet of Things (IoT) system. IoT is an important component of future information technology development, and its main technical characteristic is connecting objects to networks through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0031] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 1 As shown, the electronic device may include a processor 101, a memory 102, a camera 103, a display screen 104, a sensor module 105, a battery 106, a power management module 107, a universal serial bus interface 108, a wireless communication module 109, and a mobile communication module 110.

[0032] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0033] Processor 101 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), digital signal processors (DSPs), and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors.

[0034] The processor 101 may include one or more interfaces through which it can connect to and communicate with other devices to perform corresponding functions. For example, the processor 101 may include a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface.

[0035] The processor 101 can connect to peripheral devices such as the display screen 104 and the camera 103 via a MIPI interface. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 101 and the camera 103 communicate via the CSI interface to enable the electronic device to capture images. The processor 101 and the display screen 104 communicate via the DSI interface to enable the electronic device to display images.

[0036] The GPIO interface can be configured via software. The GPIO interface can be configured as either control signals or data signals.

[0037] A USB interface is an interface that conforms to the USB standard specification, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc.

[0038] The memory 102 can be used to store computer executable program code (such as instructions). The processor 101 executes various functional applications and data processing of the electronic device by running the instructions stored in the memory 102.

[0039] The display screen 104 is used to display images, videos, etc. The display screen 104 includes a display panel. In some embodiments, the electronic device may include one or N display screens 104, where N is a positive integer greater than 1.

[0040] Camera 103 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 103, where N is a positive integer greater than 1. Incident light passing through a lens can be converged to the focal point of the lens, causing the object being photographed to be imaged on the image sensor. The image sensor can convert the light signal into an electrical signal, which is then transmitted to the ISP in the processor 101 for processing and conversion into a digital image signal. The digital image signal is further processed and converted into a standard image signal in formats such as RGB and YUV, which can then be transmitted to the display screen 103 for display.

[0041] Sensor module 105 includes an image sensor, a touch sensor, etc. A touch sensor is also called a "touch device." The touch sensor can be disposed on display screen 104, and the touch sensor and display screen 104 together form a touchscreen, also called a "touchscreen." The touch sensor is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 104. In other embodiments, the touch sensor may also be disposed on the surface of the electronic device, in a different location than display screen 104.

[0042] The power management module 107 connects the battery 106 to the processor 101. The power management module 107 receives charging input from a charger to charge the battery 106. The charger can be a wireless charger or a wired charger. For example, the power management module 107 can receive charging input from a wired charger via a USB interface. While charging the battery 106, the power management module 107 can also supply power to electronic devices. The power management module 107 receives input from the battery 106 to supply power to the processor 101, memory 102, camera 103, display screen 104, wireless communication module 109, mobile communication module 110, etc.

[0043] The wireless communication module 109 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0044] The mobile communication module 110 can provide solutions for wireless communication applications, including 2G / 3G / 4G / 5G / 6G, in electronic devices.

[0045] In addition, an operating system runs on top of the aforementioned components. For example, operating system, Open source operating system Operating system, etc. Applications can be installed and run on this operating system.

[0046] This application uses the layered architecture of the Android system as an example to illustrate the software structure of an electronic device. Figure 2 This is an architectural block diagram of an electronic device according to an embodiment of this application.

[0047] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the driver layer.

[0048] The application layer includes a series of application packages. For example... Figure 2 As shown, applications can include camera, gallery, and other similar applications.

[0049] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. For example, in this embodiment, the application framework layer may include a camera access interface, which includes camera management.

[0050] The HAL layer provides a set of device function interface specifications for implementing a virtual hardware platform to abstract the hardware, hide hardware interface details, and thus make the code hardware-independent. For example, in this embodiment, the HAL layer may include a camera hardware abstraction layer (camera HAL), which can connect the camera access interface of the application framework layer to the underlying camera driver and hardware. The camera hardware abstraction layer may include camera devices, camera algorithms, etc.

[0051] The driver layer is the layer between hardware and software, through which electronic devices control various hardware components. For example, in this embodiment, the driver layer includes a camera device driver, which controls the camera device (such as a camera and an image signal processor).

[0052] like Figure 4 As shown, when the camera application receives a user's touch to capture the image, it generates a shooting command and transmits it sequentially to the image sensor in the hardware layer via the camera access interface, the camera hardware abstraction layer, and the camera device driver. The image sensor then transmits the obtained image electrical signal to the image signal processor (ISP) for processing, such as converting it into a digital image signal. The image signal processed by the ISP can then be transmitted to the camera algorithm in the camera hardware abstraction layer via the camera device driver. The camera algorithm further processes the image, such as the human skin segmentation method provided in this application and the aforementioned enhancement and beautification processing of pixels in the skin region. The image processed by the camera algorithm is then transmitted to the camera and the image library via the camera access interface.

[0053] The following will combine Figures 3-7 This application introduces the human skin segmentation method provided in this application.

[0054] Figure 3 This is a schematic diagram of a human skin segmentation process provided in an embodiment of this application. Figure 4 This is a flowchart of a human skin segmentation method provided in an embodiment of this application.

[0055] like Figure 4 As shown, the method may include the following steps:

[0056] S101, Receive the image to be processed.

[0057] In one exemplary embodiment, the image to be processed refers to the image passed to the camera algorithm by the image signal processor. For example, it is the image signal output by the image sensor after the user clicks the shutter button on the camera interface, which is then processed by the image signal processor. This image is then passed by the camera device driver to the camera algorithm in the camera hardware abstraction layer to execute the human skin segmentation method provided in this application.

[0058] S102, determine whether the image contains a human face; if so, execute S103 and S104 respectively;

[0059] Existing face detection algorithms can be used to detect whether an image contains a face; these will not be detailed here. If no face is found, the current process ends. Ending the current process means not performing any further steps. Figure 4 The follow-up processing steps shown can be modified for other processing steps in practical applications when the data does not include a face.

[0060] S103, use a skin segmentation algorithm to segment the skin color of the image to obtain the corresponding skin color mask image.

[0061] For example, skin color segmentation can be performed on an image using a neural network-based segmentation algorithm, which will not be described in detail in this application.

[0062] A mask image is a binary image in which pixels are labeled black or white to represent the shape, location, or attributes of certain regions or objects. Mask images are commonly used for object segmentation, image segmentation, image fusion, background removal, and more. For example, in skin segmentation, a mask image can be used to label the areas containing skin, such as labeling pixels in the skin area as white and non-skin pixels as black, thus dividing the image into skin and non-skin regions for subsequent processing.

[0063] For example, such as Figure 3 As shown, skin segmentation is performed on the input image to obtain a skin color mask image, where pixels in the skin area are white and pixels in the non-skin area are black.

[0064] S104, Perform facial landmark detection on the image.

[0065] Facial landmark detection refers to locating the key regions of a face given an image, such as eyebrows, eyes, nose, mouth, and facial contours. The location information of the key points can be relative to the entire image, or it can be relative to the face bounding box (marking the position of the face in the entire image).

[0066] Existing facial landmark detection algorithms can be used to obtain information on 64, 128, or 256 key points of a face.

[0067] In some embodiments, S103 and S104 are steps performed after detecting that the image contains a face. These two steps can be performed simultaneously or sequentially, such as performing S103 first and then S104, or performing S104 first and then S103, or performing S104 to S106 first and then S103. This application does not limit the specific execution order of S103 and S104.

[0068] S105, divides the facial key points into regions and counts the pixel information of each facial region.

[0069] The obtained facial key points are divided into regions. For example, fixed-location points can be connected to divide the key points into regions such as the cheeks, chin, profile, and around the eyes. Figure 3 The example shown is divided into n regions, namely S1, S2, S3, S4, ..., Sn.

[0070] Furthermore, the pixel information of each region obtained by statistical division is calculated, such as the average gray value of each pixel in the region (which can be called the gray mean) and the variance of the gray values ​​(which can be called the gray variance). For example, S1: Aves1, Stds1; S2: Aves2, Stds2; S3: Aves3, Stds3; S4: Aves4, Stds4; ..., Sn: Avesn, Stdsn;

[0071] S106, obtain accurate skin color information of the face region based on the pixel information of each face region.

[0072] After obtaining the mean (Avesi) and variance (Stdsi) of grayscale values ​​for each region, the average of the mean grayscale values ​​and the average of the variance grayscale values ​​for each region are calculated. For example, if n face regions are divided into S1 to Sn, and the mean grayscale values ​​of these n regions are Aves1 to Avesn, the average of these n mean grayscale values ​​(Ave) is further calculated. The variance grayscale values ​​of these n regions are Stds1 to Stdsn, and the average of these n variances (Std) is further calculated. Finally, an average mean (Ave) and an average variance (Std) are obtained for each of the n regions. Grayscale variance reflects the differences between pixels within a region; a larger variance indicates a greater difference, and a smaller variance indicates a smaller difference. Therefore, combining the average mean and average variance of each region determines accurate skin tone information.

[0073] For example, the grayscale mean (Avesi) and grayscale variance (Stdsi) of each region are compared with the mean (Ave) and the mean (Std). The grayscale mean and grayscale variance corresponding to the region with the smallest difference from the mean (Ave) and the mean (Std) are selected as the accurate skin color information.

[0074] For example, in an exemplary embodiment, all candidate face regions whose average variance falls within a preset range can be identified from the divided face regions. Then, the pixel information corresponding to the candidate face region with the smallest mean difference is selected as the precise skin color information, arranged in ascending order of mean difference. Finally, the grayscale mean and grayscale variance of the selected face region are relatively close to the corresponding mean and variance.

[0075] In another exemplary embodiment, the regions can be arranged in ascending order of the difference between the mean grayscale value Avesi and the average grayscale value Ave. First, it is determined whether the difference between the grayscale variance Stdsi and the average variance Std of the region with the smallest difference is less than a preset threshold. If so, the mean grayscale value and grayscale variance of the region with the smallest difference are determined to be accurate skin color information. If not, it is determined whether the difference between the grayscale variance and the average variance of the next region is less than the preset threshold according to the above arrangement order. This process is repeated to determine the regions with small differences between the mean grayscale value and the average grayscale value, and also with small differences between the grayscale variance and the average variance.

[0076] The face is divided into regions based on facial landmarks. The pixel information of each region is comprehensively considered, and the pixel information of the region with the smallest difference from the average pixel information of the entire face is selected as the accurate skin color information. This avoids the impact of large differences in skin color between different areas of the face due to occlusion, such as when the user wears a mask. Therefore, by dividing the face into regions and selecting the pixel information of the region with the smallest difference from the average pixel information of each region as the accurate skin color information, i.e., the accurate skin color information, can be denoted as Ave0 and Std0, thus improving the accuracy of skin color information.

[0077] S107, Identify discrete skin color mask blocks in the skin color mask image that are not located in the face region based on the face key point detection results.

[0078] In some scenarios, the captured image includes areas such as the face, arms, and hands. Therefore, the segmented skin color mask image includes not only the face region (specifically, the skin areas near the face, such as the face and neck), but also areas farther from the face, such as the arms and hands, as well as potentially missegmented areas. These skin areas outside the face region in the skin color mask image are called discrete skin color mask blocks. For example, Figure 3 In the example shown, the discrete skin color mask blocks are identified as M1, M2, M3 and M4.

[0079] In the embodiments of this application, the facial key point detection results obtained in S104, i.e. the location information of the facial region, can be used to identify the skin color mask blocks in the skin color mask image that are not in the facial region, which can be called discrete skin color mask blocks.

[0080] For example, after performing facial landmark detection on an image containing a face, the position coordinates (such as pixel coordinates) of the area where the face is located are obtained. The position coordinates of each skin color mask block in the skin color mask image are compared to see if they are within the position range of the area where the face is located. If not, the skin color mask block is determined to be a discrete skin color mask block.

[0081] S108, compare whether the difference between the discrete skin color mask block and the precise skin color information is within the tolerance threshold; if yes, proceed to S109; otherwise, proceed to S110.

[0082] There may be missegmented regions in the discrete skin color mask blocks. Therefore, in this embodiment, it is further determined whether each discrete skin color mask block is indeed a skin color region based on accurate skin color information, that is, whether each discrete skin color mask block is a missegmented region.

[0083] In some embodiments, after identifying discrete skin tone mask blocks, the pixel information of each discrete skin tone mask block is statistically analyzed, which may include the average and variance of the grayscale values ​​of the pixels within the mask block. For example, Figure 3 In the example shown, the pixel information of the discrete skin color blocks M1 to M4 is as follows: M1: Avem1, Stdm1; M2: Avem2, Stdm2; M3: Avem3, Stdm3;

[0084] M4: Avem4, Stdm4. Then, select any discrete skin tone mask block and compare the difference between the pixel information of this mask block and the precise skin tone information to see if it is within the tolerance threshold range. For example, calculate the difference between the mean and variance of this mask block and the mean and variance of the precise skin tone information (or the mean difference and variance difference). If both the mean difference and variance difference are within the tolerance threshold range, then the mask block is determined to belong to the skin tone region. If either of the above two differences exceeds the tolerance threshold range, then the mask block is determined not to belong to the skin tone region.

[0085] For example, Figure 3In the example shown, the differences between the mean and variance of block M1 and the mean and variance of the precise skin color information are calculated. If the differences corresponding to the mean and variance are both within the tolerance threshold, block M1 is determined to be a skin color region, and the mask block is retained. If either the difference corresponding to the mean and variance is not within the tolerance threshold, block M1 is determined not to be a skin color region, and the pixels within the mask block are further clustered. This process is repeated to determine whether other discrete skin color mask blocks belong to skin color regions.

[0086] In some embodiments, tolerance thresholds corresponding to the mean and variance can be set separately, or the mean and variance can use the same tolerance threshold; this application does not limit this. Furthermore, the tolerance threshold can be set according to actual needs; this application does not limit this.

[0087] S109, retain the discrete skin tone mask block.

[0088] If a discrete skin tone mask block is determined to belong to a skin tone region, then that mask block is retained. Continue to determine whether the next discrete skin tone mask block belongs to a skin tone region.

[0089] S110, cluster the discrete skin color mask blocks that do not belong to the skin color region to obtain the clustering result.

[0090] In this embodiment of the application, for Mask blocks that exceed the tolerance threshold range, a clustering algorithm is used to perform binarization clustering of the pixels within the block. The clustering results include near-skin color regions and / or difference regions.

[0091] For example, a pixel with a grayscale value close to the average value Ave0 in the precise skin color information can be randomly selected from the discrete skin color mask block as a seed point, and then clustering algorithms such as Kmeans (FCM) can be performed.

[0092] In some embodiments, the clustering process may include: assuming a Mask block comprises n pixels, randomly selecting a pixel whose grayscale value is close to Ave0 in the precise skin color information as a seed point. Further, the differences between the grayscale values ​​of the n-1 pixels in the Mask block and the grayscale value of the seed point are calculated. If the difference is less than a set threshold, the pixel is determined to be a near-skin color region; if the difference is greater than or equal to the set threshold, the pixel is determined to be a difference region.

[0093] In other embodiments, a clustering algorithm incorporating spatial information constraints can be used to perform binarized clustering of each pixel within the Mask block. When calculating the differences between each of the n-1 pixels in the Mask block and the seed point, the grayscale difference between the two pixels is weighted according to the distance between them. Then, the pixel is determined to belong to a near-skin color region or a region of difference based on the weighted grayscale difference. For example, for a single pixel, the clustering process incorporating spatial information constraints may include:

[0094] (1) For pixel m, calculate the distance (e.g., Euclidean distance) between pixel m and seed point o, and obtain the weight value that matches this distance. For example, a mapping relationship between distance and weight can be pre-defined. After calculating the distance between two pixels, the corresponding weight value is obtained by querying the mapping relationship. The larger the distance between pixels, the smaller the corresponding weight value, and the smaller the distance, the larger the corresponding weight value. The distance between pixels is negatively correlated with the weight.

[0095] (2) Calculate the grayscale difference between pixel m and seed point o, and calculate the product of the weight value and the grayscale difference as the final grayscale difference between the two pixels. If the difference is less than the set threshold, the pixel is determined to be a near-skin color region; if the difference is greater than or equal to the set threshold, the pixel is determined to be a difference region.

[0096] Repeat steps (1) and (2) above until all pixels in the Mask block have been processed. After clustering all pixels in a discrete skin color Mask block, connect pixels belonging to the near-skin color region to obtain the near-skin color region, and connect pixels belonging to the difference region to obtain the difference region. If the Mask block only includes the near-skin color region, then the entire region is determined to be a skin color region, and the Mask block is retained. If the Mask block only includes the difference region, then proceed to the next step S111. If the Mask block includes both the near-skin color region and the difference region, then the near-skin color region is retained, and the next step S111 is performed for the difference region.

[0097] S111, For pixels in the difference region, cluster them again to obtain the near-skin color region and / or difference region.

[0098] For the regions of difference identified in the first clustering, a second clustering operation is performed. The second clustering operation is the same as the first clustering process, and will not be described again here. Performing two clustering operations on the regions of difference can improve the accuracy of the clustering results.

[0099] S112, all discrete skin color mask blocks have been evaluated; if so, proceed to S113; otherwise, return to S108.

[0100] After identifying the near-skin color region and / or difference region of a discrete skin color mask block, determine whether all discrete skin color mask blocks have been identified. If there are still unidentified mask blocks, return to execute S108; if there are no unidentified mask blocks, continue to execute S113.

[0101] For example, Figure 3 In the example shown, the image includes four discrete skin color mask blocks M1 to M4. After identifying whether block M1 belongs to the skin color region, the process continues to identify block M2 until all four discrete skin color mask blocks have been identified.

[0102] S113 preserves areas close to skin tone and excludes areas of difference to obtain accurate skin tone areas.

[0103] For discrete skin tone mask blocks, after performing the process described in steps S108 to S112 above, near-skin tone regions that belong to the skin tone region and / or, difference regions that do not belong to the skin tone region can be obtained from these discrete skin tone mask blocks. For all discrete skin tone mask blocks, the retained discrete skin tone mask blocks and the near-skin tone regions in the mask blocks all belong to the skin tone region, while the difference regions are excluded, finally obtaining the accurate discrete skin tone region.

[0104] Within the same Mask block, there can be both near-skin color regions and different regions. That is, some regions in the Mask block belong to skin color regions, while some regions do not. In other words, some regions in the Mask block are incorrectly segmented.

[0105] S114, determine the precise skin tone region of the image as the face region, and retain the discrete skin tone mask blocks and near-skin tone regions.

[0106] The precise skin color region includes the face region, the preserved discrete skin color region and the near-skin color region. In other words, it excludes regions that are mistakenly segmented as skin color, thus improving the accuracy of the final skin color region.

[0107] For example, Figure 5 (1) is an image containing a human face, the skin-colored area of ​​which is the area where the face, neck and two hands are located. Figure 5 (2) is a skin color mask image obtained based on a skin segmentation algorithm. This skin color mask image includes the skin regions of the face, neck, and hands, as well as three mis-segmented skin color regions, M1, M2, and M3. After using the human skin segmentation method provided in this application, the mis-segmented skin color regions are eliminated, and the accurate skin color regions are obtained, i.e. Figure 5 The white area shown in (3).

[0108] Figure 6(1) is an image containing a human face. The skin-colored region of this image consists of the face region and the neck region not covered by clothing. The skin segmentation algorithm is used to obtain... Figure 6 The skin color mask image shown in (2) includes the face, neck, and discrete skin color mask blocks M1 to M4, in comparison. Figure 6 (1) and Figure 6 As can be seen from (2), the discrete skin color Mask blocks M1 to M4 are mis-segmented skin color regions. Figure 6 (3) is the precise skin color region obtained using the human skin segmentation method of this application.

[0109] Similarly, Figure 7 The skin-colored areas in the image shown in (1) are the face area and the neck area not covered by clothing. Figure 7 (2) shows the skin color mask image obtained using a skin segmentation algorithm. This mask image includes not only the face and neck regions, but also discrete skin color mask blocks M1. (Comparison) Figure 7 (1) and Figure 7 As can be seen from (2), the discrete skin color Mask block M1 is a mis-segmented skin color region. Figure 7 (3) is the precise skin color region obtained using the human skin segmentation method of this application.

[0110] The human skin segmentation method provided in this application, after determining that the input image contains a face, uses a skin color segmentation algorithm to segment the image to obtain a skin color mask image. Simultaneously, a facial landmark detection algorithm is used to detect facial landmarks in the image, and the facial landmarks are divided into regions, with pixel information for each region being statistically analyzed. Then, accurate skin color parameters, i.e., precise skin color information, are obtained based on the pixel information of each facial region. By dividing different regions based on facial landmark information and comparing the pixel information of each region, the pixel information corresponding to the region with the smallest difference from other regions is determined as the precise skin color information of the facial region. This avoids the impact of large skin color differences in different regions of the face due to partial occlusion, such as when the user wears a mask, on the skin color information, thus improving the accuracy of the skin color pixel information.

[0111] Furthermore, this method utilizes facial landmark detection results to identify discrete skin tone mask blocks in the skin tone mask image that are not located within the face region. For each discrete skin tone mask block, the pixel information of each discrete skin tone mask block is compared with the accurate skin tone information to see if it is within a tolerance threshold. If so, the mask block is determined to be a skin tone region and is retained; otherwise, the mask block is further binarized and clustered to obtain near-skin tone regions and / or difference regions. The difference regions are clustered again to obtain the final near-skin tone regions and / or difference regions. Near-skin tone regions are retained, and difference regions are excluded. After all discrete mask blocks have been processed, all skin tone mask blocks that may be mis-segmented are excluded, thus obtaining accurate skin tone regions. Therefore, this method improves the accuracy of skin tone segmentation. Furthermore, it improves the processing effect of other skin tone region-based processing procedures.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A human skin segmentation method, characterized by, The method is applied to an electronic device, and the method comprises: skin segmentation is performed on a to-be-processed image containing a face to obtain a skin color Mask image; face key point detection is performed on the to-be-processed image, and each face region is divided according to the detected face key points and the pixel information of each face region is planned; an average value of the pixel information is obtained based on the pixel information of each face region, and the pixel information corresponding to a face region with the minimum difference between the pixel information and the average value of the pixel information is determined as accurate skin color information; discrete skin color Mask blocks in the skin color Mask image that are not in the face region are identified based on the face key point detection result; for any discrete skin color Mask block, it is judged whether the whole discrete skin color Mask block belongs to a skin color region based on the accurate skin color information and the pixel information of the discrete skin color Mask block, and if yes, the discrete skin color Mask block is retained; if no, each pixel point in the discrete skin color Mask block is clustered to obtain a first near-skin color region and / or a first difference region; each pixel point in the first difference region is clustered again to obtain a second near-skin color region and / or a second difference region; the second difference region is excluded from the skin color region of the skin color Mask image to obtain an accurate skin color region.

2. The method of claim 1, wherein, the division of each face region according to the detected face key points comprises: connecting the face key points at a preset position in the face to obtain a face region corresponding to the preset position, and the preset position includes an eye preset range, a cheek, a chin and / or a side face.

3. The method according to claim 1 or 2, characterized in that, the planning of the pixel information of each face region comprises: statistically obtaining the average gray value and the gray variance of each pixel point in the same face region to obtain the pixel information of the same face region.

4. The method of claim 3, wherein, the obtaining of the average value of the pixel information based on the pixel information of each face region comprises: calculating the average value of the average gray value of each face region to obtain an average average value, and calculating the average value of the gray variance of each face region to obtain a variance average value.

5. The method of claim 4, wherein, the determination of the pixel information corresponding to the face region with the minimum difference between the pixel information and the average value of the pixel information as the accurate skin color information comprises: sequentially calculating the average value difference between the average gray value of each face region and the average average value; sequentially calculating the variance difference between the gray variance of each face region and the variance average value; determining the pixel information corresponding to the face region with the variance difference within a preset range and the minimum average value difference as the accurate skin color information.

6. The method of claim 5, wherein, the determination of the pixel information corresponding to the face region with the variance difference within a preset range and the minimum average value difference as the accurate skin color information comprises: determining all candidate face regions with the variance average value within a first preset range; selecting the pixel information corresponding to the candidate face region with the minimum average value difference as the accurate skin color information in the order of the average value difference from small to large.

7. The method according to any one of claims 1 to 6, characterized in that, the judgment of whether the whole discrete skin color Mask block belongs to a skin color region based on the accurate skin color information and the pixel information of the discrete skin color Mask block comprises: calculating a first difference between a mean gray value in the accurate skin color information and a mean gray value of the any discrete skin color Mask block, and calculating a second difference between a gray variance in the accurate skin color information and a gray variance of the any discrete skin color Mask block; determining whether the first difference is within a second preset range and whether the second difference is within a third preset range, and if so, determining that the any discrete skin color Mask block as a whole belongs to a skin color region; if the first difference is out of the second preset range and / or the second difference is out of the third preset range, determining that the any discrete skin color Mask block as a whole does not belong to a skin color region.

8. The method according to claim 1 or 7, characterized in that, performing clustering on each pixel point in the any discrete skin color Mask block to obtain a first near-skin color region and / or a first difference region, including: randomly selecting a pixel point with a gray value close to the mean gray value in the accurate skin color information as a seed point from the any discrete skin color Mask block; for any other pixel point in the any discrete skin color Mask block except the seed point, comparing a difference between a gray value of the any other pixel point and a gray value of the seed point with a preset threshold, and if the difference is less than the preset threshold, determining that the any other pixel point belongs to a first near-skin color region, and if not, determining that the any other pixel point belongs to a first difference region.

9. The method according to claim 1 or 7, characterized in that, performing clustering on each pixel point in the any discrete skin color Mask block to obtain a first near-skin color region and / or a first difference region, including: randomly selecting a pixel point with a gray value close to the mean gray value in the accurate skin color information as a seed point from the any discrete skin color Mask block; for any other pixel point in the any discrete skin color Mask block except the seed point, calculating a distance between the any other pixel point and the seed point, and obtaining a weight value matched with the distance; calculating a first gray difference between the any other pixel point and the seed point, and calculating a product of the first gray difference and the weight value to obtain a second gray difference; if the second gray difference is less than the preset threshold, determining that the any other pixel point belongs to a first near-skin color region, and if the second gray difference is greater than or equal to the preset threshold, determining that the any other pixel point belongs to a first difference region.

10. The method of claim 1, wherein, excluding the second difference region from a skin color region of the skin color Mask graph to obtain an accurate skin color region, including: retaining a face region in the skin color Mask graph, and retaining the first near-skin color region and the second near-skin color region, and excluding the second difference region to obtain an accurate skin color region in the skin color Mask graph.

11. An electronic device, comprising: The electronic device includes one or more processors, a memory, and a touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device implements the human skin segmentation method as claimed in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, instructions stored thereon, when executed on an electronic device, cause the electronic device to perform the human skin segmentation method as claimed in any one of claims 1 to 10.

13. A computer program product, characterised in that, A computer program product, having instructions stored thereon, which when executed on an electronic device, cause the electronic device to implement the human skin segmentation method according to any one of claims 1-10.

14. A chip system, characterized by Comprising: at least one processor and an interface for receiving code instructions and transmitting to the at least one processor; the at least one processor executes the code instructions to implement the human skin segmentation method according to any one of claims 1-10.

Images