Skin region extraction model training method, live image processing method and device

Abstract

The application relates to the fields of image processing and live broadcast technology, and provides a skin region extraction model training method, a method and device for processing skin regions in live broadcast images, equipment and a medium. The application can train a model that can accurately and stably extract skin regions. The method comprises the following steps: performing enhancement processing on a sample image to obtain an enhanced sample image and a labeled image thereof, performing first and second brightness enhancement processing on the enhanced sample image to obtain first and second model input images, inputting the first and second model input images into a skin region extraction model to be trained to obtain first and second model output images, obtaining a first model loss representing the consistency of the first and second model output images, a second model loss representing the consistency of the first model output image and the labeled image, and a third model loss representing the consistency of the second model output image and the labeled image, and training the model according to the first, second and third model losses.

Description

Technical Field

[0001] This application relates to the fields of image processing and live streaming technology, and in particular to a training method for a skin region extraction model, a method for processing skin regions in live streaming images, an apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of artificial intelligence technology, deep learning-based image processing has been applied to various fields to perform corresponding image processing tasks, including using deep learning-based skin region processing models to process skin regions in images. However, in current technology, such skin region processing models are mostly focused on applications such as dermatopathology, lesions, or skin analysis. These models are easily affected by complex scenes, resulting in low stability and accuracy in skin region extraction. Summary of the Invention

[0003] Therefore, it is necessary to provide a training method for a skin region extraction model, a method for processing skin regions in live images, an apparatus, an electronic device, and a computer-readable storage medium to address the aforementioned technical problems.

[0004] Firstly, this application provides a method for training a skin region extraction model. The method includes:

[0005] Obtain a sample image set containing skin regions and a labeled image set for annotating the skin regions;

[0006] For the sample images in the sample image set, based on the sample image set and the labeled image set, the sample images are enhanced according to the image scene to obtain enhanced sample images and their labeled images;

[0007] Based on the enhanced sample image and its labeled image, the enhanced sample image is subjected to a first brightness enhancement process and a second brightness enhancement process targeting the brightness of the skin region to obtain a first model input image and a second model input image;

[0008] Input the first model input image and the second model input image into the skin region extraction model to be trained, and obtain the first model output image corresponding to the first model input image and the second model output image corresponding to the second model input image output by the skin region extraction model to be trained;

[0009] Obtain a first model loss that characterizes the consistency between the first model output image and the second model output image, and obtain a second model loss and a third model loss that characterize the consistency between the first model output image and the second model output image and the labeled image, respectively.

[0010] The skin region extraction model to be trained is trained based on the first model loss, the second model loss, and the third model loss.

[0011] In one embodiment, the enhancement processing of the sample image for the image scene includes: obtaining an auxiliary image of the sample image from the sample image set; enlarging the size of the sample image and dividing the enlarged sample image into an auxiliary image accommodating region; placing the auxiliary image into the auxiliary image accommodating region of the enlarged sample image, and performing size restoration processing on the sample image containing the auxiliary image.

[0012] In one embodiment, the enhancement processing of the sample image for the image scene includes: obtaining an auxiliary image of the sample image from the sample image set; scaling the sample image and the auxiliary image according to a certain probability, and then adding the skin region in the scaled auxiliary image to the scaled sample image.

[0013] In one embodiment, the enhancement processing of the sample image for the image scene includes:

[0014] The labeled image of the sample image is subjected to erosion processing; the edge skin region of the sample image is determined based on the labeled image after erosion processing and the labeled image before erosion processing; the labeled image of the sample image is subjected to dilation processing; the edge region to be filled of the sample image is determined based on the labeled image after dilation processing and the labeled image before dilation processing; the edge region to be filled of the sample image is filled based on the pixel values ​​of the edge skin region.

[0015] In one embodiment, the enhancement processing of the sample image for the image scene includes: determining the skin region of the sample image based on the labeled image of the sample image; obtaining skin-like noise information based on the pixel value statistics of the skin region of the sample image; and adding the skin-like noise information to the sample image.

[0016] In one embodiment, the first brightness enhancement process or the second brightness enhancement process includes: determining the edge skin region of the enhanced sample image based on the labeled image of the enhanced sample image; determining the image block to be adjusted based on the edge skin region of the enhanced sample image; and performing brightness enhancement processing on the image block to be adjusted.

[0017] In one embodiment, the first brightness enhancement process or the second brightness enhancement process includes: determining the edge skin region of the enhanced sample image based on the labeled image of the enhanced sample image; and performing brightness enhancement processing on the edge skin region.

[0018] Secondly, this application provides a method for processing skin regions in live-streamed images. This method is applied to a terminal. The method includes:

[0019] A trained skin region extraction model is obtained; the skin region extraction model is trained according to the method described above.

[0020] After obtaining the live image to be processed, the live image is input into the skin region extraction model. Based on the skin region prediction image output by the skin region extraction model, the skin region in the live image is determined.

[0021] Add skin-processing effects to the skin areas in the live stream image.

[0022] Thirdly, this application provides a training device for a skin region extraction model. The device includes:

[0023] The image set acquisition module is used to acquire a sample image set containing skin regions and a labeled image set for labeling the skin regions;

[0024] The first enhancement processing module is used to perform scene-specific enhancement processing on the sample images in the sample image set based on the sample image set and the labeled image set, to obtain enhanced sample images and their labeled images.

[0025] The second enhancement processing module is used to perform a first brightness enhancement processing and a second brightness enhancement processing on the enhanced sample image based on the enhanced sample image and its labeled image, targeting the brightness of the skin region, to obtain a first model input image and a second model input image.

[0026] The image input module is used to input the first model input image and the second model input image into the skin region extraction model to be trained, and to obtain the first model output image corresponding to the first model input image and the second model output image corresponding to the second model input image output by the skin region extraction model to be trained;

[0027] The loss acquisition module is used to acquire a first model loss characterizing the consistency between the first model output image and the second model output image, and to acquire a second model loss and a third model loss characterizing the consistency between the first model output image and the second model output image and the labeled image, respectively.

[0028] The training processing module is used to train the skin region extraction model to be trained based on the first model loss, the second model loss, and the third model loss.

[0029] Fourthly, this application provides an apparatus for processing skin regions in live-streamed images. It is applied to a terminal. The apparatus includes:

[0030] The model acquisition module is used to acquire a trained skin region extraction model; the skin region extraction model is trained according to the method described above.

[0031] The region determination module is used to obtain the live image to be processed, input the live image into the skin region extraction model, and determine the skin region in the live image based on the skin region prediction image output by the skin region extraction model.

[0032] The effects addition module is used to add skin processing effects to the skin areas in the live stream image.

[0033] Fifthly, this application provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0034] A set of sample images containing skin regions and a set of labeled images for annotating the skin regions are obtained. For the sample images in the sample image set, enhancement processing is performed on the sample images based on the sample image set and the labeled image set to obtain enhanced sample images and their labeled images. Based on the enhanced sample images and their labeled images, a first brightness enhancement processing and a second brightness enhancement processing are performed on the enhanced sample images to target the brightness of the skin regions, resulting in a first model input image and a second model input image. The first model input image and the second model input image are input into a skin region extraction model to be trained, obtaining a first model output image corresponding to the first model input image and a second model output image corresponding to the second model input image. A first model loss characterizing the consistency between the first model output image and the second model output image is obtained, as well as a second model loss and a third model loss characterizing the consistency between the first model output image and the second model output image and their respective labeled images. The skin region extraction model to be trained is trained based on the first model loss, the second model loss, and the third model loss.

[0035] Sixthly, this application provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0036] A trained skin region extraction model is obtained; the skin region extraction model is trained according to the method described above; after obtaining the live image to be processed, the live image is input into the skin region extraction model, and the skin region in the live image is determined according to the skin region prediction image output by the skin region extraction model; skin processing effects are added to the skin region in the live image.

[0037] Seventhly, this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0038] A set of sample images containing skin regions and a set of labeled images for annotating the skin regions are obtained. For the sample images in the sample image set, enhancement processing is performed on the sample images based on the sample image set and the labeled image set to obtain enhanced sample images and their labeled images. Based on the enhanced sample images and their labeled images, a first brightness enhancement processing and a second brightness enhancement processing are performed on the enhanced sample images to target the brightness of the skin regions, resulting in a first model input image and a second model input image. The first model input image and the second model input image are input into a skin region extraction model to be trained, obtaining a first model output image corresponding to the first model input image and a second model output image corresponding to the second model input image. A first model loss characterizing the consistency between the first model output image and the second model output image is obtained, as well as a second model loss and a third model loss characterizing the consistency between the first model output image and the second model output image and their respective labeled images. The skin region extraction model to be trained is trained based on the first model loss, the second model loss, and the third model loss.

[0039] Eighthly, this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] A trained skin region extraction model is obtained; the skin region extraction model is trained according to the method described above; after obtaining the live image to be processed, the live image is input into the skin region extraction model, and the skin region in the live image is determined according to the skin region prediction image output by the skin region extraction model; skin processing effects are added to the skin region in the live image.

[0041] The aforementioned training method, apparatus, electronic device, and computer-readable storage medium for skin region extraction models involve acquiring a set of sample images containing skin regions and a set of labeled images for annotating these skin regions. Based on the sample image set and the labeled image set, the sample images are enhanced according to the image scene to obtain enhanced sample images and their labeled images. Based on the enhanced sample images and their labeled images, the enhanced sample images undergo first and second brightness enhancement processing targeting the skin region brightness to obtain first and second model input images. These first and second model input images are input into the skin region extraction model to be trained, resulting in first and second model output images. A first model loss characterizing the consistency between the first and second model output images, and second and third model losses characterizing the consistency between each of the first and second model output images and their labeled images are obtained. The skin region extraction model is then trained based on these first, second, and third model losses. This scheme, based on sequential enhancement processing of sample images in terms of image scene and brightness, combined with supervised training using corresponding losses, can train a skin region extraction model that can stably and accurately extract complete skin regions from images. Attached Figure Description

[0042] Figure 1 This is a diagram illustrating the application scenarios of the relevant methods in the embodiments of this application;

[0043] Figure 2 This is a flowchart illustrating the training method of the skin region extraction model in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of the enhancement process in the embodiments of this application;

[0045] Figure 4 This is a schematic diagram of the skin region extraction model in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the process of supervised training of model loss in the embodiments of this application;

[0047] Figure 6 This is a flowchart illustrating the method for processing skin regions in live-streamed images in an embodiment of this application.

[0048] Figure 7 This is a structural block diagram of the training device for the skin region extraction model in the embodiments of this application;

[0049] Figure 8 This is a structural block diagram of the apparatus for processing skin regions in live images according to embodiments of this application;

[0050] Figure 9 This is a diagram showing the internal structure of the electronic device in an embodiment of this application;

[0051] Figure 10 This is an internal structural diagram of an electronic device according to another embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] The training method for the skin region extraction model and the method for processing skin regions in live images provided in this application can be applied to, for example... Figure 1 In the application scenario shown, the scenario may include a terminal and a server. The terminal communicates with the server via a network. The server can be used to execute the training method of the skin region extraction model provided in this application, and send the trained skin region extraction model to the terminal for deployment. The terminal can be used to execute the method for extracting skin regions from live images provided in this application, and apply the trained skin region extraction model to extract skin regions from live images and perform subsequent processing. The terminal may be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server may be a standalone server or a server cluster consisting of multiple servers.

[0054] The training method of the skin region extraction model and the method of processing skin regions in live images provided in this application will be described in turn with reference to the various embodiments and corresponding figures.

[0055] In one embodiment, such as Figure 2 As shown, a training method for a skin region extraction model is provided. This method can be achieved by, for example, training a skin region extraction model. Figure 1 The server execution method shown may include the following steps:

[0056] Step S201: Obtain a sample image set containing skin regions and a labeled image set for labeling skin regions.

[0057] In this step, a set of sample images containing skin regions and a set of labeled images for annotating skin regions can be obtained from the training sample database. The sample image set may include multiple sample images; the image that annotates the skin regions of the corresponding sample image is called its labeled image. The set of images composed of these labeled images is the aforementioned labeled image set. Specifically, the labeled images included in this labeled image set can be mask images used to annotate the skin regions in the corresponding sample images. Correspondingly, the labeled image set can be the mask image set corresponding to the set of sample images containing skin regions.

[0058] Step S202: For the sample images in the sample image set, based on the sample image set and the labeled image set, perform enhancement processing on the sample images according to the image scene to obtain enhanced sample images and their labeled images.

[0059] Step S203: Based on the enhanced sample image and its labeled image, perform a first brightness enhancement process and a second brightness enhancement process on the enhanced sample image targeting the brightness of the skin region to obtain a first model input image and a second model input image.

[0060] Combination Figure 3 Steps S202 and S203 above involve sequentially performing scene enhancement processing and first or second brightness enhancement processing on the sample images in the sample image set. That is, the sample image is first enhanced to obtain an enhanced sample image, and then the enhanced sample image is subjected to first brightness enhancement processing to obtain a first model input image, or the enhanced sample image is subjected to second brightness enhancement processing to obtain a second model input image. The first and second model input images will be used to input the skin region extraction model to be trained in step S204.

[0061] In step S202, scene enhancement processing can be performed on each sample image in the sample image set. Specifically, for each sample image, scene enhancement processing is performed based on the sample image set and the labeled image set. The resulting image is called an enhanced sample image. Scene enhancement processing refers to the process of enriching / enhancing the scene in the image. This can be done in various ways. For example, for a sample image, it can be fused / combined with another sample image in the sample image set. The processed sample image can then be used as an enhanced sample image.

[0062] Furthermore, in one embodiment, the enhancement processing of the sample image for the image scene in step S202 may include:

[0063] Obtain auxiliary images from the sample image set; enlarge the sample image and divide the auxiliary image area into regions within the enlarged sample image; place the auxiliary image into the corresponding auxiliary image area within the enlarged sample image and perform size restoration processing on the sample image placed in the auxiliary image.

[0064] In this embodiment, four auxiliary images can be randomly obtained from the sample image set. Let the size of each sample image in the set be H×W, where H represents the image height and W represents the image width. Let this size also represent the size of the input image required by the skin region extraction model to be trained. Therefore, the sample image can be enlarged to four times its original size, resulting in an enlarged sample image of size 4H×4W. Then, a center point can be randomly determined in the middle third of the enlarged sample image. Based on this center point, the enlarged sample image is divided into four regions, which are the auxiliary image accommodating regions. Then, the four auxiliary images are placed into their corresponding center points within these four auxiliary image accommodating regions. Finally, the sample images placed into the auxiliary images are scaled back to their original size of H×W. This embodiment enriches the image scene of the sample images and avoids the problem of large areas without images appearing in corners due to some images being too small in ordinary mosaic enhancement. It ensures that the enhanced sample image contains the entire region, improving the model training effect.

[0065] Furthermore, in one embodiment, the enhancement processing of the sample image for the image scene in step S202 may include:

[0066] Obtain auxiliary images of the sample images from the sample image set; after scaling the sample images and auxiliary images according to a certain probability, add the skin region from the scaled auxiliary images to the scaled sample images.

[0067] In this embodiment, for a sample image in the sample image set, an auxiliary image of the sample image can be randomly obtained from the sample image set, thereby obtaining the sample image and its auxiliary image. Then, the sample image and its auxiliary image can be scaled with a certain probability and a scaling factor of [0.8, 1.2] to obtain the scaled auxiliary image and the scaled sample image. Then, the skin region in the scaled auxiliary image can be extracted and added to the scaled sample image, thereby enabling some fragmented skin regions to be obtained in the sample image, achieving the purpose of enriching the image scene of the sample image and improving the model training effect.

[0068] Furthermore, in one embodiment, the enhancement processing of the sample image for the image scene in step S202 may include:

[0069] The annotation image of the sample image is subjected to erosion processing; the edge skin region of the sample image is determined based on the annotation image after erosion processing and the annotation image before erosion processing; the annotation image of the sample image is subjected to dilation processing; the edge region to be filled in the sample image is determined based on the annotation image after dilation processing and the annotation image before dilation processing; the edge region to be filled in the sample image is filled based on the pixel value of the edge skin region.

[0070] In this embodiment, for a sample image, on one hand, an erosion operation is performed on the labeled image of the sample image (denoted as the labeled image before erosion) to obtain an eroded labeled image. Then, the difference between the eroded labeled image and the labeled image before erosion is calculated, and the edge skin region of the sample image is determined based on this difference, and the pixel values ​​of the edge skin region of the sample image are obtained. On the other hand, a dilation operation is performed on the labeled image of the sample image (denoted as the labeled image before dilation) to obtain a dilated labeled image. Then, the difference between the dilated labeled image and the labeled image before dilation is calculated, and the region corresponding to this difference is determined in the sample image and denoted as the edge region to be filled. Finally, the edge region to be filled is filled based on the pixel values ​​of the edge skin region obtained in the first aspect. Specifically, the edge region to be filled can be filled with the average value of the pixel values ​​of the aforementioned edge skin region. The scheme of this embodiment allows the subsequent model to learn the semantic information of skin shape, and can achieve accurate skin edge extraction even in scenarios where the background color and skin color are similar.

[0071] Furthermore, in another embodiment, the enhancement processing of the sample image for the image scene in step S202 may include:

[0072] Based on the labeled image of the sample image, determine the skin region of the sample image; based on the pixel value statistics of the skin region of the sample image, obtain skin-like noise information; add the skin-like noise information to the sample image.

[0073] This embodiment introduces skin-like noise information, or skin-like noise enhancement, into the sample image to further enhance the model's ability to recognize skin and background regions. Specifically, the skin region of the sample image is first determined based on the labeled image. Then, the pixel values ​​of the skin region are obtained, and statistical information is obtained from these pixel values. Specifically, the average and variance values ​​of the pixel values ​​in the skin region are obtained. Then, Gaussian noise is obtained as skin-like noise information based on the average and variance values. This skin-like noise information is added to the sample image to obtain the enhanced sample image. Compared to skin-like filling, the skin-like noise enhancement in this embodiment can enhance the model's ability to recognize skin and background noise from a global perspective.

[0074] In practical applications, when performing scene enhancement processing on sample images in a sample image set, the scene enhancement processing provided in one or more of the above embodiments can be performed with a certain probability to obtain enhanced sample images and their labeled images.

[0075] In step S203, based on the obtained enhanced sample image and its labeled image, first and second brightness enhancement processes are performed. The first and second brightness enhancement processes can be two different brightness enhancement processes for the same enhanced sample image. In practical applications, the first brightness enhancement process can be to brighten the entire enhanced sample image to obtain the first model input image, and the second brightness enhancement process can be to keep the brightness of the enhanced sample image unchanged to obtain the second model input image, and vice versa. After the first and second brightness enhancement processes, Gaussian smoothing operations can be performed on each of them. This step can make the model more robust to brightness.

[0076] Furthermore, in one embodiment, the first brightness enhancement process or the second brightness enhancement process may include:

[0077] Based on the labeled image of the enhanced sample image, determine the edge skin region of the enhanced sample image; based on the edge skin region of the enhanced sample image, determine the image block to be adjusted; and perform brightness enhancement processing on the image block to be adjusted.

[0078] The brightness enhancement processing provided in this embodiment can be used as either a first brightness enhancement processing or a second brightness enhancement processing, specifically performing brightness enhancement processing on a localized skin region. In this embodiment, the edge skin region of the enhanced sample image can first be determined based on the labeled image of the enhanced sample image. Specifically, the edge skin region can be determined by referring to the previously described method based on erosion operations. Then, an image block of unlimited size can be randomly determined based on the edge skin region as the image block to be adjusted. For example, an image block of unlimited size can be determined at a position above the edge skin region as the image block to be adjusted. Brightness enhancement processing is then performed on the image block to be adjusted, for example, the image block to be adjusted can be brightened.

[0079] Furthermore, in another embodiment, the first brightness enhancement process or the second brightness enhancement process may include:

[0080] Based on the labeled image of the enhanced sample image, the edge skin region of the enhanced sample image is determined; the edge skin region is then subjected to brightness enhancement processing.

[0081] The brightness enhancement processing provided in this embodiment can be used as either a first brightness enhancement processing or a second brightness enhancement processing, specifically performing brightness enhancement processing on the edges of the skin region. Specifically, in this embodiment, the labeled image of the enhanced sample image can be subjected to erosion and dilation operations respectively to obtain the eroded and dilated labeled images corresponding to the enhanced sample image. Then, the difference between the edge information of the eroded and dilated labeled images can be extracted. Based on this difference, the corresponding edge skin region on the enhanced sample image can be obtained, and then the brightness enhancement processing of this edge skin region can be performed, for example, brightening the edge skin region.

[0082] In practical applications, when performing first and second brightness enhancement processing on the enhanced sample image targeting the brightness of the skin region, the brightness enhancement processing targeting the brightness of the skin region provided in one or more of the above embodiments can be set in the first and second brightness enhancement processing with a certain probability, so that the first and second brightness enhancement processing can respectively form different brightness enhancement processing for the enhanced sample image, and respectively obtain the first model input image and the second model input image.

[0083] Step S204: Input the first model input image and the second model input image into the skin region extraction model to be trained, and obtain the first model output image corresponding to the first model input image and the second model output image corresponding to the second model input image.

[0084] In this step, the model input images for the skin region extraction model to be trained can include a first model input image and a second model input image. The skin region extraction model to be trained outputs a first model output image for the first model input image and a second model output image for the second model input image. That is, the model output image of the skin region extraction model to be trained can include both the first model output image and the second model output image. Figure 4 In practical applications, the skin region extraction model to be trained can be a deep learning network, which can consist of two branches. Specifically, it can include a shallow network branch and a deep network branch. The former can be used to extract image detail information of the input image of the model, and the latter can be used to extract high semantic information of the input image of the model. The features extracted by the two branches of the deep learning network can be fused together to obtain the output of the model.

[0085] Step S205: Obtain a first model loss that represents the consistency between the first model output image and the second model output image, and obtain a second model loss and a third model loss that represent the consistency between the first model output image and the second model output image and the labeled image, respectively.

[0086] This step involves obtaining the model loss; specifically, it combines... Figure 5 The model loss includes a first model loss, a second model loss, and a third model loss. The first and second model input images are input into the skin region extraction model to be trained. The model outputs a first model output image corresponding to the first model input image and a second model output image corresponding to the second model input image. The first model loss can be obtained based on the consistency between the first and second model output images. The second model loss can be obtained based on the consistency between the first model output image and the labeled image of the aforementioned augmented sample image. The third model loss can be obtained based on the consistency between the second model output image and the labeled image. Specifically, the first model loss can be calculated based on the KL divergence loss function, using the first and second model output images as a consistency constraint loss. Similarly, the second and third model losses can be calculated based on the cross-entropy loss function, using the first model output image and the labeled image.

[0087] Step S206: Train the skin region extraction model to be trained based on the first model loss, the second model loss, and the third model loss.

[0088] In this step, specifically, in combination Figure 5 The total model loss can be obtained by summing the losses of the first, second, and third models. Based on this total loss, backpropagation is used to adjust the model parameters of the skin region extraction model to be trained. This allows the model to learn key semantic features unaffected by data augmentation. As an example, a well-trained skin region extraction model can be obtained when the total model loss is less than or equal to a set loss threshold.

[0089] The training method for the skin region extraction model in this embodiment involves acquiring a set of sample images containing skin regions and a set of labeled images for annotating these skin regions. Based on the sample image set and the labeled image set, the sample images are subjected to scene-specific enhancement processing to obtain enhanced sample images and their labeled images. Based on the enhanced sample images and their labeled images, the enhanced sample images undergo first and second brightness enhancement processing targeting the skin region brightness to obtain first and second model input images. These first and second model input images are input into the skin region extraction model to be trained, resulting in first and second model output images. A first model loss characterizing the consistency between the first and second model output images, and second and third model losses characterizing the consistency between each of the first and second model output images and their respective labeled images are obtained. The skin region extraction model to be trained is then trained based on these first, second, and third model losses. This scheme, based on sequential enhancement processing of sample images in terms of image scene and brightness, combined with corresponding loss-supervised training, can train a skin region extraction model that can stably and accurately extract complete skin regions from images.

[0090] In one embodiment, such as Figure 6 As shown, a method for processing skin regions in live stream images is provided. This method can be used by... Figure 1 The method, executed on the terminal shown, may include the following steps:

[0091] Step S601: Obtain the trained skin region extraction model.

[0092] In this step, the server can obtain a trained skin region extraction model according to the training method of the skin region extraction model provided in any of the above embodiments, which is denoted as the trained skin region extraction model. Then the server can send the trained skin region extraction model to the terminal for deployment, so that the terminal obtains the trained skin region extraction model provided by the server.

[0093] Step S602: After obtaining the live image to be processed, input the live image into the skin region extraction model, and determine the skin region in the live image based on the skin region prediction image output by the skin region extraction model.

[0094] In this step, the terminal can capture the user's live stream image with the user's authorization. The user can also authorize the terminal to add skin processing effects, such as beautification, makeup, or AR effects. The terminal can then use the captured live stream image as the image to be processed and input it into the skin region extraction model. This is the application phase of the skin region extraction model. In the application phase, no enhancement processing is needed on the input image, but the live stream image to be processed can be normalized to the image input size specified by the skin region extraction model before being input. Then, the skin region extraction model outputs a corresponding skin region prediction image based on the input live stream image. After obtaining this skin region prediction image, the terminal can determine the skin region in the live stream image accordingly.

[0095] Step S603: Add skin processing effects to the skin area in the live image.

[0096] In this step, the terminal can add skin processing effects, such as beauty filters, makeup, or AR effects, to the skin area in the live broadcast image according to the user's desired skin processing effect.

[0097] The solution in this embodiment can apply the skin region extraction model to live streaming scenarios, enabling the terminal to accurately and stably extract complete skin regions from live streaming images and add skin processing effects using the skin region extraction model.

[0098] The proposed solution provides a lightweight skin region extraction model that can be deployed on the terminal. It learns skin semantic information under varying scenarios, improves the model's robustness through targeted enhancement processing, and enhances the model's semantic recognition ability of skin regions through supervised constraints and consistency constraints. This enables real-time fine segmentation of the skin region of people in live images, obtaining the real-time skin region of people in live scenes. It can achieve efficient, stable, and accurate extraction of complete skin regions in live scenes, which can be used for precise skin beautification, makeup, and special effects generation. In other words, the obtained fine skin region can be used for beautification, makeup, or AR special effects, and it has strong robustness to lighting and skin integrity prediction. It saves the cost of manual image cutout, thereby improving the efficiency of live image processing, and can meet the needs of real-time segmentation of complete skin regions in live streaming and other scenarios. At the same time, it is easy to promote and apply on a large scale.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] Based on the same inventive concept, this application also provides a related apparatus for implementing the aforementioned methods. The solution provided by this apparatus is similar to the implementation schemes described in the above methods; therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the related methods described above, and will not be repeated here.

[0101] In one embodiment, such as Figure 7 As shown, a training device for a skin region extraction model is provided, which can be applied to a server. The device 700 may include:

[0102] Image set acquisition module 701 is used to acquire a sample image set containing skin regions and a labeled image set for labeling the skin regions;

[0103] The first enhancement processing module 702 is used to perform scene-specific enhancement processing on the sample images in the sample image set based on the sample image set and the labeled image set, to obtain enhanced sample images and their labeled images.

[0104] The second enhancement processing module 703 is used to perform a first brightness enhancement processing and a second brightness enhancement processing on the enhanced sample image based on the enhanced sample image and its labeled image, targeting the brightness of the skin region, to obtain a first model input image and a second model input image.

[0105] Image input module 704 is used to input the first model input image and the second model input image into the skin region extraction model to be trained, and obtain the first model output image corresponding to the first model input image and the second model output image corresponding to the second model input image output by the skin region extraction model to be trained;

[0106] The loss acquisition module 705 is used to acquire a first model loss characterizing the consistency between the first model output image and the second model output image, and to acquire a second model loss and a third model loss characterizing the consistency between the first model output image and the second model output image and the labeled image, respectively.

[0107] The training processing module 706 is used to train the skin region extraction model to be trained based on the first model loss, the second model loss, and the third model loss.

[0108] In one embodiment, the first enhancement processing module 702 is configured to obtain an auxiliary image of the sample image from the sample image set; enlarge the sample image and divide the enlarged sample image into an auxiliary image accommodating region; place the auxiliary image into the auxiliary image accommodating region of the enlarged sample image, and perform size restoration processing on the sample image containing the auxiliary image.

[0109] In one embodiment, the first enhancement processing module 702 is used to obtain an auxiliary image of the sample image from the sample image set; after scaling the sample image and the auxiliary image according to a certain probability, the skin region in the scaled auxiliary image is added to the scaled sample image.

[0110] In one embodiment, the first enhancement processing module 702 is configured to perform erosion processing on the labeled image of the sample image; determine the edge skin region of the sample image based on the labeled image after erosion processing and the labeled image before erosion processing; perform dilation processing on the labeled image of the sample image; determine the edge region to be filled of the sample image based on the labeled image after dilation processing and the labeled image before dilation processing; and perform filling processing on the edge region to be filled of the sample image based on the pixel values ​​of the edge skin region.

[0111] In one embodiment, the first enhancement processing module 702 is configured to determine the skin region of the sample image based on the labeled image of the sample image; obtain skin-like noise information based on the pixel value statistics of the skin region of the sample image; and add the skin-like noise information to the sample image.

[0112] In one embodiment, the first brightness enhancement process or the second brightness enhancement process includes: determining the edge skin region of the enhanced sample image based on the labeled image of the enhanced sample image; determining the image block to be adjusted based on the edge skin region of the enhanced sample image; and performing brightness enhancement processing on the image block to be adjusted.

[0113] In one embodiment, the first brightness enhancement process or the second brightness enhancement process includes: determining the edge skin region of the enhanced sample image based on the labeled image of the enhanced sample image; and performing brightness enhancement processing on the edge skin region.

[0114] In one embodiment, such as Figure 8 As shown, an apparatus for processing skin regions in live-stream images is provided, applicable to a terminal. The apparatus 800 may include:

[0115] The model acquisition module is used to acquire a trained skin region extraction model; the skin region extraction model is trained according to the method described above.

[0116] The region determination module is used to obtain the live image to be processed, input the live image into the skin region extraction model, and determine the skin region in the live image based on the skin region prediction image output by the skin region extraction model.

[0117] The effects addition module is used to add skin processing effects to the skin areas in the live stream image.

[0118] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0119] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as sample image sets. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements a training method for a skin region extraction model.

[0120] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external devices; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for processing skin regions in live images. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0121] Those skilled in the art will understand that Figure 9 and Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0122] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of training a skin region extraction model, characterized by, The method comprises: obtaining a sample image set containing skin regions and a labeled image set for labeling the skin regions; for a sample image in the sample image set, based on the sample image set and the labeled image set, performing enhancement processing on the sample image for an image scene to obtain an enhanced sample image and a labeled image thereof; based on the enhanced sample image and the labeled image thereof, performing first and second brightness enhancement processing on the enhanced sample image for skin region brightness to obtain first and second model input images; inputting the first and second model input images into a skin region extraction model to be trained to obtain first and second model output images corresponding to the first and second model input images output by the skin region extraction model to be trained; obtaining a first model loss representing consistency of the first and second model output images, and obtaining second and third model losses representing consistency of the first and second model output images with the labeled images respectively; training the skin region extraction model to be trained according to the first, second and third model losses, including: summing the first, second and third model losses to obtain a total model loss, adjusting model parameters of the skin region extraction model to be trained in a back propagation manner according to the total model loss to train the same, and obtaining a trained skin region extraction model when the total model loss is less than or equal to a set loss threshold.

2. The method of claim 1, wherein, The enhancement processing on the sample image for an image scene comprises: obtaining an auxiliary image of the sample image from the sample image set; performing size enlargement processing on the sample image, and dividing an auxiliary image containing area in the sample image after size enlargement; correspondingly placing the auxiliary image into the auxiliary image containing area in the sample image after size enlargement, and performing size recovery processing on the sample image with the auxiliary image placed therein.

3. The method of claim 1, wherein, The enhancement processing on the sample image for an image scene comprises: obtaining an auxiliary image of the sample image from the sample image set; performing scaling processing on the sample image and the auxiliary image according to a certain probability, and adding a skin region in the auxiliary image after scaling processing to the sample image after scaling processing.

4. The method of claim 1, wherein, The enhancement processing on the sample image for an image scene comprises: performing erosion processing on the labeled image of the sample image; determining an edge skin region of the sample image according to the labeled image after erosion processing and the labeled image before erosion processing; performing inflation processing on the labeled image of the sample image; determining a to-be-filled edge region of the sample image according to the labeled image after inflation processing and the labeled image before inflation processing; performing filling processing on the to-be-filled edge region of the sample image according to pixel values of the edge skin region.

5. The method of claim 1, wherein, The enhancement processing on the sample image for an image scene comprises: According to the label image of the sample image, a skin region of the sample image is determined; According to pixel value statistical information of the skin region of the sample image, skin color-like noise information is obtained; The skin color-like noise information is added to the sample image.

6. The method according to any one of claims 1 to 5, characterized in that, The first brightness enhancement processing or the second brightness enhancement processing comprises: According to the label image of the enhanced sample image, an edge skin region of the enhanced sample image is determined; According to the edge skin region of the enhanced sample image, an image block to be adjusted is determined; The image block to be adjusted is subjected to brightness enhancement processing.

7. The method according to any one of claims 1 to 5, characterized in that, The first brightness enhancement processing or the second brightness enhancement processing comprises: According to the label image of the enhanced sample image, an edge skin region of the enhanced sample image is determined; The edge skin region is subjected to brightness enhancement processing.

8. A method of processing a skin region in a live image, characterized in that, The method applied to a terminal comprises: A trained skin region extraction model is obtained; the skin region extraction model is trained according to the method in any one of claims 1 to 7; After obtaining a live image to be processed, the live image is input into the skin region extraction model, and a skin region prediction image output by the skin region extraction model is used to determine a skin region in the live image; A skin processing effect is added to the skin region in the live image.

9. A training device for a skin region extraction model, characterized in that, The device comprises: An image set acquisition module is configured to acquire a sample image set containing a skin region and a label image set used for labeling the skin region; A first enhancement processing module is configured to, for a sample image in the sample image set, perform enhancement processing on the sample image based on the sample image set and the label image set, to obtain an enhanced sample image and a label image thereof; A second enhancement processing module is configured to perform first brightness enhancement processing and second brightness enhancement processing on the enhanced sample image based on the enhanced sample image and the label image thereof, to obtain a first model input image and a second model input image; An image input module is configured to input the first model input image and the second model input image into a skin region extraction model to be trained, to obtain a first model output image corresponding to the first model input image and a second model output image corresponding to the second model input image output by the skin region extraction model to be trained; A loss acquisition module is configured to acquire a first model loss representing consistency of the first model output image and the second model output image, and acquire a second model loss and a third model loss representing consistency of the first model output image and the second model output image with a label image, respectively; A training processing module is configured to train the skin region extraction model to be trained according to the first model loss, the second model loss, and the third model loss, including: summing the first model loss, the second model loss, and the third model loss to obtain a total model loss, adjusting model parameters of the skin region extraction model to be trained in a back propagation manner according to the total model loss to train the skin region extraction model to be trained, and obtaining a trained skin region extraction model when the total model loss is less than or equal to a set loss threshold.

10. An apparatus for processing a skin region in a live image, characterized by Applied to a terminal, the apparatus comprises: a model obtaining module, configured to obtain a trained skin region extraction model; the skin region extraction model is trained according to the method in any one of claims 1 to 7; a region determining module, configured to, after obtaining a live image to be processed, input the live image into the skin region extraction model, and determine a skin region in the live image according to a skin region prediction image output by the skin region extraction model; an effect adding module, configured to add a skin processing effect to the skin region in the live image. 11.An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program comprises the following steps of: The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7 or the method in claim 8.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7 or the method in claim 8.

Images