An image processing method, an electronic device, and a readable storage medium

Abstract

The application discloses an image processing method and device and a computer readable storage medium. The method comprises the following steps: acquiring an image to be processed; performing matching detection on pixel values of each pixel in the image to be processed based on a preset pixel condition to obtain an image mask corresponding to the image to be processed; inputting the image to be processed and the image mask into a skin region identification model; performing skin color identification processing on the image to be processed based on the image mask to obtain a skin region image of the image to be processed; and the image mask has been in a general range in which the skin region is located, so that the skin region identification model can invest limited computing capacity into the general range, identify whether each part in the range is a skin region at a high accuracy, and then obtain a skin region image at a high accuracy by using the limited computing capacity.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, and computer-readable storage medium. Background Technology

[0002] In fields such as live video streaming, users often use beautification features like skin whitening and smoothing to enhance their images. These features require detecting and identifying skin areas in the original image before applying whitening or other enhancements. Since whitening reduces image sharpness, poor accuracy in skin area selection can lead to misidentifying non-skin images as skin. Whitening these misidentified skin areas will not improve the whitening effect but will instead reduce the overall image sharpness. Skin recognition is performed by the live streaming terminal. Due to the limited computing power of these terminals, large-scale image segmentation models with high computational requirements cannot be deployed. Typically, lightweight image segmentation models or traditional pixel-based segmentation methods are used for skin identification. However, lightweight image segmentation models are limited by computational performance, resulting in poor accuracy, and traditional methods also have low accuracy. This leads to lower image sharpness after whitening. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide an image processing method, electronic device and computer-readable storage medium that can obtain skin region images with high accuracy.

[0004] To address the aforementioned technical problems, in a first aspect, this application provides an image processing method, comprising:

[0005] Obtain the image to be processed;

[0006] Based on preset pixel conditions, the pixel values ​​of each pixel in the image to be processed are matched and detected to obtain the image mask corresponding to the image to be processed.

[0007] The image to be processed and the image mask are input into the skin region recognition model. The skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain the skin region image of the image to be processed.

[0008] Optionally, the step of matching and detecting the pixel values ​​of each pixel in the image to be processed based on preset pixel conditions to obtain the image mask corresponding to the image to be processed includes:

[0009] Extract the pixel value of the pixel in each color channel, and determine whether the pixel values ​​satisfy any pixel value relationship specified by the preset pixel conditions;

[0010] If any of the pixel value relationships are not satisfied, then the detection result of the pixel is determined to be non-skin color;

[0011] If the target pixel value relationship is satisfied, then determine whether each pixel value is within the pixel value range corresponding to the target pixel value relationship;

[0012] If the pixel value falls within the specified range, then the detection result for that pixel is determined to be skin color.

[0013] If the pixel value is not within the specified range, then the detection result of the pixel is determined to be non-skin color.

[0014] The pixel value of the pixel whose detection result is skin color in the image to be processed is determined as the first pixel value, and the pixel value of the pixel whose detection result is non-skin color in the image to be processed is determined as the second pixel value, thus obtaining the image mask; wherein the first pixel value and the second pixel value are different.

[0015] Optionally, the skin region recognition model includes an image adjustment network. The skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain a skin region image of the image to be processed, including:

[0016] The image adjustment network is used to resize the image to be processed and the image mask to obtain a preprocessed image; the number of pixels on both the long and wide sides of the preprocessed image is a multiple of 4.

[0017] The preprocessed image is input into the multi-layer network of the skin region recognition model to obtain the skin region image.

[0018] Optionally, the skin region recognition model includes a multi-layer network, which includes a feature extraction sub-network. The feature extraction sub-network includes several convolutional modules. The convolutional modules have an inverted residual structure and an attention mechanism structure. The inverted residual structure uses depthwise separable convolution.

[0019] Optionally, the number of input channels and the number of output channels of the convolution module are multiples of 4.

[0020] Optionally, the skin region recognition model includes a multi-layer network, which includes an upsampling sub-network with several adjacent alternating scaling layers and deconvolution layers.

[0021] Optionally, it also includes:

[0022] The skin area image is processed to whiten the skin tone, resulting in a whitened image corresponding to the image to be processed.

[0023] Optionally, the skin region recognition model is trained using a training dataset, and the process of obtaining the training dataset includes:

[0024] Acquire multiple initial images;

[0025] The initial image is input into the human body parsing and segmentation model to obtain multiple candidate label results corresponding to each initial image;

[0026] The initial image is labeled using the target labeling results to obtain an initial training image. The initial training image is used as a training image mask, and the color images corresponding to the initial image in several color channels constitute training data. The training data is then added to the training dataset. The target labeling results are the candidate labeling results related to skin.

[0027] Optionally, the training process of the skin region recognition model includes:

[0028] According to the preset training batch size, select multiple target training data from the training dataset;

[0029] The target training data is input into the initial model to obtain the output training skin region image;

[0030] Based on the training skin region image and the initial training image, the network parameters of the initial model are adjusted using an optimizer according to a preset learning rate.

[0031] If the training completion condition is met, the initial model with adjusted parameters is determined as the skin region recognition model.

[0032] Optionally, the step of inputting the target training data into the initial model to obtain the output training skin region image includes:

[0033] The target color image and the initial target training image in the target training data are input into the feature extraction subnetwork of the initial model to obtain a multi-channel feature map.

[0034] The multi-channel feature map is input into the upsampling subnetwork of the initial model to obtain the training skin region image.

[0035] Secondly, this application also provides an electronic device, including a memory and a processor, wherein:

[0036] The memory is used to store computer programs;

[0037] The processor is used to execute the computer program to implement the image processing method described above.

[0038] Thirdly, this application also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described image processing method.

[0039] The image processing method provided in this application obtains an image to be processed; based on preset pixel conditions, it performs matching detection on the pixel values ​​of each pixel in the image to be processed to obtain an image mask corresponding to the image to be processed; it inputs the image to be processed and the image mask into a skin region recognition model, and the skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain a skin region image of the image to be processed.

[0040] As can be seen, this method combines traditional skin recognition methods with image segmentation algorithms, enabling accurate skin region recognition even on live streaming terminals with limited computing power. After acquiring the image to be processed, skin color is detected from the perspective of pixel values ​​using preset pixel conditions, obtaining detection results that reflect whether the pixel values ​​represent skin color. The detection results can be used to generate a corresponding image mask, which marks the skin color portions in the image to be processed. Since the range of skin color is wide, and some non-skin objects also have skin color, the image mask also marks objects in the image to be processed that are not skin. However, the preprocessed image also excludes parts in the image to be processed that are clearly not skin, initially defining the range where skin is located. The skin region recognition model deployed on the terminal is limited by the terminal's computing power, resulting in low accuracy in directly recognizing the image to be processed. In this application, after obtaining the image mask, it is input together with the image to be processed into the skin region recognition model. The skin region recognition model can further recognize the image to be processed based on the image mask, further carefully identifying the initially defined skin regions, and finally obtaining the skin region image. Since the image mask has already defined the approximate range of the skin region, the skin region recognition model can devote its limited computing power to this approximate range to accurately identify whether each part within this range is a skin region. This allows the model to obtain a high-accuracy skin region image using limited computing power, thus solving the problem of low accuracy in determining the skin region in related technologies, which leads to poor image clarity after whitening and other processing.

[0041] In addition, this application also provides an electronic device and a computer-readable storage medium that have the same beneficial effects as described above. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 A schematic diagram of the hardware framework applicable to an image processing method provided in an embodiment of this application;

[0044] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0045] Figure 3 This application provides a live streaming terminal display interface as an embodiment.

[0046] Figure 4 A flowchart of image processing during live streaming is provided as an embodiment of this application;

[0047] Figure 5 A flowchart for image mask generation provided in this application embodiment;

[0048] Figure 6 A processing effect diagram provided for an embodiment of this application;

[0049] Figure 7 A specific skin region recognition model structure diagram provided in this application embodiment;

[0050] Figure 8 A schematic flowchart illustrating a specific image processing method provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of an initial image and candidate marker results provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] To facilitate understanding, the hardware framework used in the image processing method provided in this application embodiment will be described first. Please refer to... Figure 1 , Figure 1This is a schematic diagram of the hardware framework applicable to an image processing method provided in an embodiment of this application. The electronic device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input / output (I / O) interface 104, and a communication component 105.

[0054] The processor 101 controls the overall operation of the electronic device 100 to complete all or part of the steps in the image processing method. The memory 102 stores various types of data to support the operation of the electronic device 100. This data may include, for example, instructions for any application or method operating on the electronic device 100, as well as application-related data. The memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In this embodiment, the memory 102 stores at least programs and / or data for implementing the following functions:

[0055] Obtain the image to be processed;

[0056] Based on preset pixel conditions, the pixel values ​​of each pixel in the image to be processed are matched and detected to obtain the image mask corresponding to the image to be processed.

[0057] The image to be processed and the image mask are input into the skin region recognition model. The skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain the skin region image of the image to be processed.

[0058] Multimedia component 103 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 102 or transmitted via communication component 105. The audio component also includes at least one speaker for outputting audio signals. I / O interface 104 provides an interface between processor 101 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 105 is used for wired or wireless communication between electronic device 100 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of these. Therefore, the corresponding communication component 105 may include a Wi-Fi component, a Bluetooth component, or an NFC component.

[0059] Electronic device 100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform image processing methods.

[0060] certainly, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device 100 may include more than Figure 1 More or fewer components as shown, or combinations of certain components.

[0061] Based on the above explanation, please refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The method in this embodiment includes:

[0062] S101: Obtain the image to be processed.

[0063] The image to be processed refers to the image that needs to be used for skin region recognition. There are no restrictions on its specific number, format, or size. The image to be processed can be an image showing human skin areas, or it can be an image without human skin.

[0064] The images to be processed can be acquired in real time using image acquisition devices such as cameras, or pre-stored images can be retrieved from a specified storage location. The images to be processed can be independent images, or several video frames can be selected from a video.

[0065] This embodiment does not limit the timing of acquiring the image to be processed. It is understood that the image to be processed needs to undergo skin region recognition, which is typically a preliminary step before processing such as skin whitening. Therefore, in one implementation, it is possible to detect whether image processing functions such as skin whitening have started working, for example, by detecting whether an instruction to enable image processing functions has been detected. If so, the acquired image can be identified as the image to be processed. Specifically, the user can click a function activation button on the terminal they are using. This button can be a physical or virtual button for selecting to enable a function. After the user clicks this button, the terminal (e.g., mobile phone, tablet, computer, etc.) can determine that skin region recognition is required, and thus identify the subsequently acquired image as the image to be processed. Please refer to [reference needed]. Figure 3 , Figure 3 This application provides a live streaming terminal display interface. It includes a virtual "skin beautification" button, which, when clicked, appears with a border. After the user clicks this virtual button, the terminal determines that the skin beautification function needs to be activated. In this embodiment, the skin beautification function includes a whitening process, therefore, skin area recognition is required, and the subsequently acquired image can be identified as the image to be processed.

[0066] For details, please refer to Figure 4 , Figure 4This application provides a flowchart of image processing during live streaming. In live video applications, after the broadcaster starts broadcasting using a live streaming terminal with the beauty function enabled, they use the image acquisition component on the live streaming terminal, or an image acquisition device that interacts with the live streaming terminal to acquire image frames, completing the step of inputting the image frames (i.e., the images to be processed) into the live streaming terminal. The live streaming terminal processes each image frame by skin color segmentation (i.e., skin region recognition), skin color smoothing, and skin color whitening, and then outputs it to the central CDN (Content Delivery Network). The viewer's terminal (i.e., the viewer's end) pulls the data stream from the central CDN and displays it on the viewer's end, completing the live streaming data transmission process. Since skin color whitening and other processing can cause a decrease in image clarity, if the skin region cannot be accurately selected and a large portion of the non-skin region is whitened, it results in a meaningless decrease in clarity, leading to a poor viewing experience for the viewer.

[0067] S102: Based on preset pixel conditions, perform matching detection on the pixel values ​​of each pixel in the image to be processed to obtain the image mask corresponding to the image to be processed.

[0068] Preset pixel conditions refer to restrictions placed on pixel values ​​to ensure that pixels can represent skin tones. It's understandable that different people have different skin tones, therefore human skin tones correspond to a color range.

[0069] The specific content of the preset pixel conditions is not limited; the preset pixel conditions can vary depending on the color mode of each pixel in the image to be processed. The color mode can specifically be RGB, CMYK, or YUV. RGB color mode is an industry standard that obtains various colors by varying the three color channels (Red, G, and Blue) and their superposition. RGB represents the colors of the red, green, and blue channels, and this standard covers almost all colors perceptible to human vision, making it one of the most widely used color systems. CMYK refers to the four-color printing mode, a color mixing mode used in color printing. It utilizes the mixing principle of the three primary colors of pigments, plus black ink, resulting in a total of four colors mixed and superimposed to form so-called full-color printing. The four standard colors are: C, Cyan (also known as sky blue or azure); M, Magenta (also known as magenta); Y, Yellow; and K, black. The abbreviation uses the last letter K instead of the first B to avoid confusion with Blue. CMYK mode is a subtractive color model, while the corresponding RGB mode is an additive color model. YUV is a color encoding method commonly used in various video processing components. Here, Y represents luminance, which is the grayscale value; U and V represent chrominance, which describes the color and saturation of an image and is used to specify the color of a pixel.

[0070] For all color modes, the preset pixel conditions specify the range in which the pixel values ​​of each color channel should fall when a pixel wants to represent skin tone; that is, the range of pixel values ​​for skin tone in each color channel. For some color modes, to represent skin tone, the pixel values ​​of each color channel should also have a certain correlation, such as a difference relationship or a sum relationship. This correlation can be called a pixel value relationship.

[0071] Regarding the determination of preset pixel conditions: In one embodiment, all images to be processed are generated using a fixed color mode, therefore the preset pixel conditions can also be fixed. In another embodiment, multiple candidate skin tone conditions can be set. After acquiring the image to be processed, its color mode can be analyzed, and the corresponding condition can be selected as the preset pixel condition based on the analysis results. After determining the preset pixel conditions, they are used to match and detect the pixel values ​​of each pixel in the image to be processed, determining whether the pixel value matches the preset pixel condition, and obtaining the detection result. It can be understood that by determining whether the pixel value matches the preset pixel condition, it is essentially determining whether the pixel represents skin tone, so the detection result can indicate whether the pixel represents skin tone. The detection result can correspond to the entire image to be processed, that is, a detection result records whether all pixels in the image to be processed represent skin tone, or it can correspond to a single pixel in the image to be processed; this embodiment does not limit this.

[0072] After obtaining the detection results, an image mask corresponding to the image to be processed is generated. An image mask is an image used to represent the skin color portion of the image to be processed. Since the detection result only has two possibilities: skin color or not skin color, the pixels in the image mask only have two values, representing whether the image at that pixel location in the image to be processed is skin color or not.

[0073] Understandably, although this color range is relatively small compared to the entire color space, a large number of non-skin objects still fall within it. Therefore, the resulting image mask covers not only the skin area but also some of the non-skin objects. This image mask is the result of traditional segmentation methods for skin region recognition. In practical applications, this method's mIOU (Mean Intersection over Union, a metric used to evaluate the accuracy of image segmentation tasks; a higher value indicates higher accuracy) only reaches 63%, indicating low accuracy.

[0074] For details, please refer to Figure 5 , Figure 5This document provides a flowchart for generating an image mask according to an embodiment of the present application. After obtaining an image to be processed with a resolution of 720*1280, the counter `index` is reset to zero. While traversing each pixel, the counter value is compared with 720*1280. If the value is less than 720*1280, the pixel value of the pixel pointed to by the counter value is read, and it is determined whether it meets a preset pixel condition, thus determining whether the pixel is classified as skin. If so, the pixel is set to pure white; otherwise, it is set to pure black. The above process is repeated until the counter value is no longer less than 720*1280, thus obtaining the image mask. Figure 5 The recorded process consists of steps S102 and S103. The classification result is the detection result, and the process of setting pixels to pure white or pure black based on the classification is the image mask generation process, ultimately yielding a black and white image mask. From Figure 5 As can be seen, the final image mask not only records the skin area but also many non-skin areas. If the image mask is used as the final recognition result and whitening is performed based on it, it will lead to a decrease in the clarity of the processed image.

[0075] For example, please refer to Figure 6 , Figure 6 This is a processing effect diagram provided for an embodiment of this application. Figure 6 The left image is the image before processing, and the right image is the processed image. It's clear that the curtain area on the side of the face has also been whitened, reducing its clarity and making it blurry. It should be noted that using a lightweight deep learning model for skin region recognition is limited by computational power; the model's structure is relatively simple, and its recognition accuracy is also relatively low. The results obtained using it for skin region recognition and whitening are... Figure 6 Similar.

[0076] S104: Input the image to be processed and the image mask into the skin region recognition model. The skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain the skin region image of the image to be processed.

[0077] In this application, to improve the accuracy of skin region recognition, traditional skin region recognition and deep learning models are combined. After processing the image to be processed using preset pixel conditions, such as matching and detection, the resulting image mask covers the entire skin region, filtering out non-skin areas. In this case, the image to be processed and the image mask are input into the skin region recognition model. The skin region recognition model can then allocate its limited computing power to the suspected area, further identifying whether the area covered by the image mask is skin. This avoids wasting computing power by identifying areas determined to be non-skin, thus preventing low accuracy. In other words, skin color recognition processing is performed on the image to be processed based on the image mask to obtain the skin region image. This embodiment does not limit the specific architecture and type of the skin region recognition model; for example, it can be a model adapted from lightweight neural network models such as UNet or MobileNet.

[0078] After obtaining the skin region image, skin tone whitening processing can be performed on the skin region image. The processed image is then used to replace the skin region image in the image to be processed, resulting in the whitened image corresponding to the image to be processed.

[0079] The image processing method provided in this application combines traditional skin recognition methods with image segmentation algorithms, enabling accurate skin region recognition even on live streaming terminals with limited computing power. After acquiring the image to be processed, skin color is detected from the perspective of pixel values ​​using preset pixel conditions. That is, the pixel values ​​of pixels in the image to be processed are detected based on preset pixel conditions to obtain a detection result reflecting whether the pixel value represents skin color. The detection result can be used to generate a corresponding image mask, which is an image that marks the skin color parts in the image to be processed. Since the range of skin color is wide, and some non-skin objects also have skin color, the image mask also marks objects in the image to be processed that are not skin. However, the preprocessed image also excludes parts in the image to be processed that are obviously not skin, and initially delineates the range where skin is located. The skin region recognition model deployed on the terminal is limited by the terminal's computing power, and its accuracy in directly recognizing the image to be processed is low. In this application, after obtaining the image mask, it is input together with the image to be processed into the skin region recognition model. Skin region recognition models can further identify skin regions in an image based on an image mask, carefully distinguishing the initially defined skin regions to obtain a final skin region image. Since the image mask already defines the approximate area of ​​the skin region, the skin region recognition model can allocate limited computing power to this approximate area, accurately identifying whether each part within this area is a skin region. This allows for the generation of a high-accuracy skin region image with limited computing power, solving the problem of low accuracy in identifying skin regions in related technologies, which leads to poor image clarity after whitening and other processing.

[0080] Based on the above embodiments, this embodiment provides a detailed description of some steps in the above embodiments. In one implementation, the preset pixel conditions may specify pixel value relationships. In this case, the process of matching and detecting the pixel values ​​of each pixel in the image to be processed based on the preset pixel conditions to obtain the image mask corresponding to the image to be processed may include the following steps:

[0081] Step 11: Extract the pixel values ​​of each color channel and determine whether the pixel values ​​satisfy any pixel value relationship specified by the preset pixel conditions.

[0082] Step 12: If any pixel value relationship is not satisfied, then the detection result of the pixel is determined to be non-skin color.

[0083] Step 13: If the target pixel value relationship is satisfied, determine whether each pixel value is within the pixel value range corresponding to the target pixel value relationship.

[0084] Step 14: If it is within the pixel value range, then the detection result of the pixel is determined to be skin color.

[0085] Step 15: If the pixel value is not within the range, the detection result of the pixel is determined to be non-skin color.

[0086] Step 16: Determine the pixel value of the pixel in the image to be processed that is detected as skin color as the first pixel value, and determine the pixel value of the pixel in the image to be processed that is detected as non-skin color as the second pixel value to obtain the image mask; wherein the first pixel value and the second pixel value are different.

[0087] During matching detection, pixel values ​​are read according to color channels. For example, if the image to be processed is in RGB color mode, the pixel values ​​corresponding to the R, G, and B channels can be read respectively. In practical applications, a certain color mode may have multiple combinations of color channel values ​​to represent different skin tones. In this case, preset pixel conditions may record multiple combinations of pixel value relationships and pixel value ranges. Pixel value relationships refer to the correlation between individual pixel values, such as magnitude relationships or difference relationships; pixel value ranges refer to the selectable range of pixel values ​​corresponding to a pixel when representing skin tones.

[0088] After obtaining the pixel values, it is first determined whether the pixel values ​​satisfy any pixel value relationship specified by the preset pixel conditions. If no pixel value relationship is satisfied, the detection result of the pixel can be directly determined as non-skin color. If the target pixel value relationship is satisfied, it can be further determined whether the pixel values ​​are all within the pixel value range specified by the conditions. If not, the detection result of the pixel is determined as non-skin color. If it is, the result is determined as skin color.

[0089] It should be noted that in the above embodiments, there is only one target pixel value relationship. In another embodiment, the ranges defined by the various pixel value relationships may overlap. In this case, when it is determined that a pixel value satisfies the target pixel value relationship but is not within the corresponding pixel value interval, it can be determined whether there is a pass among the matching results corresponding to all pixel value relationships. A pass means that the pixel value relationship is satisfied and the pixel value is within the corresponding pixel value interval. If it exists, the detection result corresponding to the pixel is determined to be isomorphic; if it does not exist, it is determined to be non-skin color.

[0090] For example, in RGB color mode, there are two types of pixel value relationships. The first type is:

[0091] (abs(RG)>15)and(R>G)and(R>B)and(max(R,G,B)-min(R,G,B)>15)

[0092] That is, the absolute value of RG is greater than 15, and R>G, and R>B, and the difference between the maximum and minimum values ​​of the three is greater than 15. Under these conditions, the corresponding set of pixel value intervals is R∈(95,220), G∈(40,210), B∈(20,170).

[0093] The second type is:

[0094] (abs(RG)>15)and(R>G)and(R>B)and(max(R,G,B)-min(R,G,B)≤15)

[0095] That is, the absolute value of RG is greater than 15, and R>G, and R>B, and the difference between the maximum and minimum values ​​of the three is no greater than 15. Under these conditions, the corresponding set of pixel value intervals is R∈(220,255), G∈(210,255), B∈(170,255).

[0096] The two pixel value relationships mentioned above do not overlap. Therefore, if a certain pixel value relationship is satisfied but the pixel value is not in the corresponding pixel value range, the detection result can be determined to be non-skin color.

[0097] In another implementation, the second pixel value relationship is:

[0098] (abs(RG)>15)and(R>G)and(R>B)

[0099] In this case, the two pixel value relationships have an intersection. Therefore, if the second pixel value relationship is satisfied and the pixel value is not in the corresponding pixel value range, the detection result cannot be directly determined as non-skin color. Instead, the detection result needs to be determined based on the satisfaction of the first pixel value relationship and the pixel value range judgment.

[0100] In summary, under RGB color mode, the process of determining the detection result and generating an image mask on the image to be processed is as follows:

[0101] if(abs(RG)>15)and(R>G)and(R>B):

[0102] if(R>95)and(G>40)and(B>20)and(max(R,G,B)-min(R,G,B)>15):

[0103] imgskin[r,c] = (255,255,255) # This pixel represents skin color; setting this pixel to white.

[0104] elif(R>220)and(G>210)and(B>170):

[0105] imgskin[r,c] = (255,255,255) # This pixel represents skin color; setting this pixel to white.

[0106] In practical applications, since there are two possible detection results, two different pixel values ​​can be used to replace the original pixels in the image to be processed, thus generating an image mask. Specifically, two different pixel values, the first and second, are used to replace the original pixel values ​​of the pixels respectively. The resulting image is a two-color image mask, with the two pixel values ​​representing the two detection results.

[0107] The process of obtaining the detection results and using them to generate an image mask is executed cyclically. That is, whenever a detection result for a pixel is obtained, it is immediately used in the generation of the image mask, for example, replacing pixels in the image to be processed to obtain an initial image mask. After multiple iterations, once all pixels have been processed, the initial image mask is converted into an actual image mask. In another implementation, the steps of obtaining the detection results and generating the image mask are not executed cyclically, but rather separately. That is, after all the detection results are obtained, they are used uniformly to generate the image mask.

[0108] Based on the above embodiments, in order to maximize processing speed and fully utilize the computing resources of mobile devices, the skin region recognition model includes an image adjustment network used to adjust the size of the input image. Therefore, the process of using the skin region recognition model to perform skin color recognition processing on the image to be processed based on an image mask to obtain a skin region image may include the following steps:

[0109] Step 31: Use an image adjustment network to resize the image to be processed and the image mask to obtain a preprocessed image.

[0110] Step 32: Input the preprocessed image into the multi-layer network of the skin region recognition model to obtain the skin region image.

[0111] It should be noted that the number of pixels on both the long and wide sides of the preprocessed image is a multiple of 4. Since the granularity of register read / write operations is 4 data units, setting the number of pixels on the long and wide sides of the processed image to a multiple of 4 allows for fast data read / write operations and simultaneously ensures full utilization of GPU (graphics processing unit) resources. This embodiment does not limit the specific method of size adjustment processing; for example, it could be cropping the long or wide side. Furthermore, the data input range for the skin region recognition model can be (0,1) to improve model training speed and facilitate rapid convergence. In this case, in addition to size adjustment, pixel value normalization of the image is also required.

[0112] A multilayer network refers to a network used to process images to obtain skin region images; its specific content is not limited. In a preferred embodiment, to improve data processing speed, the multilayer network includes a feature extraction sub-network, which comprises several convolutional modules. These convolutional modules have inverse residual structures and attention mechanism structures, with the inverse residual structure employing depthwise separable convolution. The combination of the inverse residual structure and the lightweight attention mechanism, along with the use of depthwise separable convolution, allows for faster feature map extraction. Furthermore, the number of input and output channels of the convolutional modules is also a multiple of 4 to fully utilize GPU resources.

[0113] The multi-layer network also includes an upsampling sub-network, which is used to generate the final skin region image based on the feature map obtained by the feature extraction sub-network. The upsampling sub-network has several adjacent alternating scaling (resize) layers and deconvolution (convTranspose) layers, which improve the data processing speed by alternating processing.

[0114] Please refer to Figure 7 , Figure 7 This document presents a specific structure diagram of a skin region recognition model provided in an embodiment of this application. Parameter I represents the number of input channels, parameter O represents the number of output channels, and parameter S represents the convolution stride. Module 1 (a 2D convolutional layer, Conv2D layer) integrates the three-channel (R, G, B) image to be processed and a single-channel image mask into four-channel data. Then, feature extraction is performed using a feature extraction sub-network composed of multiple modules 2 (convolutional modules, ConvBlock). Finally, an upsampling sub-network composed of modules 3 (deconvolutional layer, Deconv) and 4 (scaling layer, Resize) is used for data processing, ultimately outputting a single-channel image, which is the skin region image. Module 2 specifically includes an inverse residual structure and a lightweight attention mechanism structure. Pointwise convolution is equivalent to the PointWiSe operation, depthwise convolution is equivalent to the DepthWiSe operation, average pooling is equivalent to the Avg Pool layer, and 1×1 2D convolution is equivalent to a 1×1 Conv2D layer. Figure 7 The plus sign operator in the code is the concat operator.

[0115] Please refer to Figure 8 , Figure 8This is a flowchart illustrating a specific image processing method provided in an embodiment of this application. After outputting the skin region image, the skin region image and the image to be processed can be integrated using OpenGL's mix operation to obtain the final processed image. OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics. This interface consists of nearly 350 different function calls used to draw everything from simple graphics bits to complex 3D scenes. In one implementation, this integration can be performed by a skin region recognition model, and the skin region image finally output by the skin region recognition model is the integrated image.

[0116] It should be noted that the skin region recognition model is a converged model that has been trained. It can be trained based on a training dataset, and the process of obtaining the training dataset may include the following steps:

[0117] Step 41: Obtain multiple initial images.

[0118] Step 42: Input the initial image into the human body parsing and segmentation model to obtain multiple candidate label results corresponding to each initial image.

[0119] Step 43: Use the target labeling results to label the initial image to obtain the initial training image. Use the initial training image as a training image mask and combine it with the color images corresponding to several color channels of the initial image to form training data. Add the training data to the training dataset. The target labeling results are candidate labeling results related to skin.

[0120] The initial image refers to unlabeled training images, the number of which is not limited, for example, it could be 30,000. The human body segmentation model is used to identify, segment, and label various human body parts in the image. Its specific type and structure are not limited; for example, it could be the Self-correction for human parsing (SFHP model, released in 2020). The SFHP model receives the initial image as input and can provide a mask (i.e., candidate labeling results) for each human body part. Since not every candidate labeling result is related to skin, target labeling results can be selected. Target labeling results are candidate labeling results related to skin. These are used to label the initial image to obtain the initial training image, which, together with the corresponding initial image mask, ultimately constitutes the training data. Multiple training data sets form the training dataset. Please refer to [reference needed]. Figure 9 , Figure 9This is a schematic diagram of an initial image and candidate labeling results provided in an embodiment of this application. Among the multiple candidate labeling results, only the candidate labeling results corresponding to Face (i.e., face), left / right-arm (i.e., left arm and right arm), and left / right-leg (i.e., left leg and right leg) are the target labeling results.

[0121] Specifically, the training process of the skin region recognition model includes:

[0122] Step 51: Select multiple target training data from the training dataset according to the preset training batch size.

[0123] Step 52: Input the target training data into the initial model to obtain the output training skin region image.

[0124] Step 53: Based on the training skin region images and the initial training images, use the optimizer to adjust the network parameters of the initial model according to the preset learning rate.

[0125] Step 54: If the training completion condition is met, the initial model with adjusted parameters is determined as the skin region recognition model.

[0126] Furthermore, the process of inputting the target training data into the initial model to obtain the output training skin region image may include:

[0127] Step 61: Input the target color image and the initial target training image from the target training data into the feature extraction subnetwork of the initial model to obtain a multi-channel feature map.

[0128] Step 62: Input the multi-channel feature map into the upsampling subnetwork of the initial model to obtain the training skin region image.

[0129] During training, the batch size (batch_size) can be set to 32, the learning rate (preset learning rate) to 0.01, and the Adam optimizer can be used. The Adam (adaptive moment estimation) optimizer is a method for calculating the adaptive learning rate for each parameter. The training data size can be 96x128x4 (where 96x128x3 is the original RGB image and 96x128x1 is the corresponding image mask), and the output size is 96x128x1. The specific conditions for training completion are not limited; for example, they could be the number of training rounds or the model's recognition accuracy. Through a feature extraction sub-network, features can be extracted from the color images of each color channel and the corresponding image masks (i.e., the initial training images). After obtaining the multi-channel feature maps, they are upsampled to generate the corresponding training skin region images.

[0130] The computer-readable storage medium provided in the embodiments of this application is described below. The computer-readable storage medium described below can be referred to in correspondence with the image processing method described above.

[0131] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image processing method described above.

[0132] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0134] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0136] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0137] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; Based on preset pixel conditions, the pixel values ​​of each pixel in the image to be processed are matched and detected to obtain the image mask corresponding to the image to be processed. The color channels of the image to be processed and the single channels of the image mask are integrated, and the integrated data is input into the skin region recognition model. The skin region recognition model performs skin color recognition processing on the image to be processed based on the image mask to obtain the skin region image of the image to be processed. The skin recognition model is a deep learning model.

2. The image processing method according to claim 1, characterized in that, The step of matching and detecting the pixel values ​​of each pixel in the image to be processed based on preset pixel conditions to obtain the image mask corresponding to the image to be processed includes: Extract the pixel value of the pixel in each color channel, and determine whether the pixel values ​​satisfy any pixel value relationship specified by the preset pixel conditions; If any of the pixel value relationships are not satisfied, then the detection result of the pixel is determined to be non-skin color; If the target pixel value relationship is satisfied, then determine whether each pixel value is within the pixel value range corresponding to the target pixel value relationship; If the pixel value falls within the specified range, then the detection result for that pixel is determined to be skin color. If the pixel value is not within the specified range, then the detection result of the pixel is determined to be non-skin color. The pixel value of the pixel whose detection result is skin color in the image to be processed is determined as the first pixel value, and the pixel value of the pixel whose detection result is non-skin color in the image to be processed is determined as the second pixel value, thus obtaining the image mask; wherein the first pixel value and the second pixel value are different.

3. The image processing method according to claim 1, characterized in that, The skin region recognition model includes an image adjustment network. Based on the image mask, the skin region recognition model performs skin color recognition processing on the image to be processed to obtain a skin region image of the image to be processed, including: The image adjustment network is used to resize the image to be processed and the image mask to obtain a preprocessed image; the number of pixels on both the long and wide sides of the preprocessed image is a multiple of 4. The preprocessed image is input into the multi-layer network of the skin region recognition model to obtain the skin region image.

4. The image processing method according to claim 1, characterized in that, The skin region recognition model includes a multi-layer network, which includes a feature extraction sub-network. The feature extraction sub-network includes several convolutional modules. The convolutional modules have an inverted residual structure and an attention mechanism structure. The inverted residual structure uses depthwise separable convolution.

5. The image processing method according to claim 4, characterized in that, The number of input channels and the number of output channels of the convolution module are multiples of 4.

6. The image processing method according to claim 1, characterized in that, The skin region recognition model includes a multi-layer network, which includes an upsampling sub-network with several adjacent alternating scaling layers and deconvolution layers.

7. The image processing method according to claim 1, characterized in that, Also includes: The skin area image is processed to whiten the skin tone, resulting in a whitened image corresponding to the image to be processed.

8. The image processing method according to claim 1, characterized in that, The skin region recognition model is trained using a training dataset, and the process of obtaining the training dataset includes: Acquire multiple initial images; The initial image is input into the human body parsing and segmentation model to obtain multiple candidate label results corresponding to each initial image; The initial image is labeled using the target labeling results to obtain an initial training image. The initial training image is used as a training image mask, and the color images corresponding to the initial image in several color channels constitute training data. The training data is then added to the training dataset. The target labeling results are the candidate labeling results related to skin.

9. The image processing method according to claim 8, characterized in that, The training process of the skin region recognition model includes: According to the preset training batch size, select multiple target training data from the training dataset; The target training data is input into the initial model to obtain the output training skin region image; Based on the training skin region image and the initial training image, the network parameters of the initial model are adjusted using an optimizer according to a preset learning rate. If the training completion condition is met, the initial model with adjusted parameters is determined as the skin region recognition model.

10. The image processing method according to claim 9, characterized in that, The step of inputting the target training data into the initial model to obtain the output training skin region image includes: The target color image and the initial target training image in the target training data are input into the feature extraction subnetwork of the initial model to obtain a multi-channel feature map. The multi-channel feature map is input into the upsampling subnetwork of the initial model to obtain the training skin region image.

11. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the image processing method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the image processing method as described in any one of claims 1 to 10.

Images