A face segmentation method, apparatus and device

Abstract

The present application discloses a face segmentation method, device and equipment and a storage medium, which comprises: obtaining an image to be segmented, and inputting the image to be segmented into a pre-trained face segmentation model after normalizing the image; wherein the image to be segmented comprises a face; outputting a face segmentation result corresponding to the image to be segmented through the face segmentation model; wherein the face segmentation model is trained through a pre-constructed main branch network and an edge branch network and based on adaptive adjustment of the weight of a preset loss function. By introducing the edge branch, the model can optimize the edge segmentation accuracy without increasing the inference loss; and by adaptively adjusting the weight of each objective function, the calculation amount of manual search for optimal parameters is reduced, and the segmentation accuracy is optimized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a face segmentation method, apparatus, and device. Background Technology

[0002] Scene parsing is a fundamental task in computer vision. It involves assigning class labels to specific pixels in an image based on semantic information, serving as the foundation for downstream tasks. Full-face segmentation, as a scene parsing task, uses algorithms to segment the facial region in an image, obtaining a facial mask. Specific beautification algorithms, such as makeup and face slimming applications, rely on facial masks obtained through full-face segmentation. With advancements in algorithms, new demands have emerged for the precision of full-face segmentation.

[0003] Currently, mainstream semantic segmentation technologies are all based on deep learning. The performance of deep learning-based semantic segmentation algorithms mainly depends on the data, network structure, objective function, and optimization algorithm. The latest semantic segmentation algorithms improve the performance by modifying one or more of these components. Current semantic segmentation algorithms typically use multi-task supervision as the objective function to optimize the network, such as using a weighted combination of cross-entropy loss and DICE loss. Defining the weights of each loss becomes crucial to the algorithm's performance. The common practice is to manually try different combinations to find the optimal one, which incurs significant computational costs and makes finding the optimal combination difficult. Furthermore, current face segmentation algorithms have relatively coarse edge details, making it difficult to meet the fine-grained segmentation requirements of downstream tasks. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a face segmentation method, apparatus and device to solve the above problems.

[0005] To achieve the above objectives, the present invention provides a face segmentation method, the method comprising:

[0006] The image to be segmented is obtained, and after normalization, it is input into the pre-trained face segmentation model; wherein, the image to be segmented includes a face;

[0007] The face segmentation model outputs a face segmentation result corresponding to the image to be segmented; wherein, the face segmentation model is trained by a pre-constructed main branch network and edge branch network and the weights of a preset loss function are adaptively adjusted.

[0008] Preferably, the loss function includes a first loss function for supervising the face mask output by the main branch network and a second loss function for supervising the face edge mask output by the edge branch network.

[0009] Preferably, the first loss function includes the cross-entropy loss function L of the face mask. ce and the Dice loss function L of the face mask. dice ,in,

[0010]

[0011] In the formula, i represents the pixel index, N = H × W represents the number of pixels, and c represents the pixel category index. This indicates that the i-th pixel corresponds to the true label of the c-th class. This represents the predicted probability that the i-th pixel corresponds to the c-th class.

[0012] Preferably, the second loss function includes the balanced cross-entropy loss function L of the face edge mask. bce , In the formula, j represents the pixel index, g j Let s represent the j-th pixel. j Y represents the predicted probability corresponding to the j-th pixel. + Represents the set of pixels for positive example labels and the corresponding edge pixels, Y - This represents the set of pixels labeled as negative examples and the corresponding background pixels, with β representing the balance factor.

[0013] Preferably, the step of adaptively adjusting the weights of the preset loss function includes:

[0014] During model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective.

[0015] Preferably, during model training, the process of transforming the multi-task optimization of the face segmentation model into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective includes:

[0016] According to the preset formula The multi-task optimization is transformed into multi-objective optimization, where the formula for multi-task optimization is: In the formula, t represents the sequence number of different tasks, θ represents the model parameters, and w t Represents the weights of different tasks, and θ sh θ represents the model parameters shared by all tasks. tThese represent parameters specific to different tasks. This indicates the loss of experience from the task.

[0017] Preferably, the step of adaptively adjusting the weights of the preset loss function further includes:

[0018] The weights are updated using the Frank-Wolfe parser.

[0019] To achieve the above objectives, the present invention also provides a face segmentation device, the device comprising:

[0020] An input unit is used to acquire an image to be segmented, and after normalizing the image to be segmented, input it into the pre-trained face segmentation model; wherein, the image to be segmented includes a face;

[0021] The processing unit is configured to output a face segmentation result corresponding to the image to be segmented through the face segmentation model; wherein the face segmentation model is trained by a pre-constructed main branch network and edge branch network and the weights of a preset loss function are adaptively adjusted.

[0022] To achieve the above objectives, the present invention also proposes an apparatus comprising a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a face segmentation method as described in the above embodiments.

[0023] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a face segmentation method as described in the above embodiments.

[0024] Beneficial effects:

[0025] The above scheme optimizes the edge details of the segmentation results by introducing edge branches as a supervision task. The edge branches are only used as auxiliary supervision during the training phase, which enables the model to optimize the edge segmentation accuracy without increasing inference loss. Furthermore, the scheme reduces the computational cost of manually searching for optimal parameters by adaptively adjusting the weights of each objective function through adaptive multi-objective optimization, while also improving the segmentation accuracy.

[0026] The above solution, using the face segmentation method proposed in this embodiment, can significantly improve the edge accuracy and overall segmentation effect of face segmentation in practical applications.

[0027] The above scheme utilizes an adaptive balanced cross-entropy loss function for edge loss, which helps to more effectively supervise edges, optimize edge segmentation accuracy, and improve segmentation results. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a face segmentation method according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the network structure of a face segmentation model provided in an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram illustrating the update process of the Frank-Wolfe parser provided in an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram of a face segmentation device provided in an embodiment of the present invention.

[0033] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0036] The present invention will be described in detail below with reference to the embodiments.

[0037] Reference Figure 1 The diagram shown is a flowchart of a face segmentation method provided in an embodiment of the present invention.

[0038] In this embodiment, the method includes:

[0039] S11, Obtain the image to be segmented, and input the image to be segmented into the pre-trained face segmentation model after normalization processing; wherein, the image to be segmented includes a face;

[0040] S12, the face segmentation model outputs the face segmentation result corresponding to the image to be segmented; wherein, the face segmentation model is trained by a pre-constructed main branch network and edge branch network and the weights of a preset loss function are adaptively adjusted.

[0041] In this embodiment, as Figure 2 The network structure diagram shown illustrates a face segmentation model comprising a main branch network and edge branch networks. The main branch network is an Encoder-Decoder network structure responsible for predicting the final face segmentation result. The edge branch network predicts the segmentation edges by integrating multi-layer features from the Encoder stage network. The edge branch network is an auxiliary branch that obtains finer segmentation edges; it only participates in network optimization during training and is ignored during inference. Therefore, adding an edge branch does not increase the network's inference time. In other words, during training, the forward path includes both the main branch and the edge branch, with the edge branch obtained by fusing shallow features of different resolutions from the Encoder part of the main branch. During inference, the edge branch is ignored, and the final segmentation result is directly predicted from the main branch. Figure 2 In the diagram, res-s1 and res-s2 represent residual modules with stride lengths of 1 and 2, respectively; the conv 1x1 module is a channel fusion module with a 1x1 convolutional kernel; seg-loss is a combined loss that includes the cross-entropy loss of the face mask and the Dice loss of the face mask; edge-loss represents the edge loss of the face edge mask.

[0042] Furthermore, the loss function includes a first loss function for supervising the face mask output by the main branch network, and a second loss function for supervising the face edge mask output by the edge branch network. The first loss function includes the cross-entropy loss function L of the face mask. ce and the Dice loss function L of the face mask. dice The second loss function includes the balanced cross-entropy loss function L of the face edge mask. bce.

[0043] In this embodiment, a multi-task supervision approach is used to optimize the face segmentation model, mainly comprising two parts: seg-loss and edge-loss, used to supervise the face mask and face edge mask, respectively. The seg-loss consists of cross-entropy loss and Dice loss; the edge-loss uses a cross-entropy loss function that balances positive and negative samples. The overall loss function is as follows:

[0044] L = w0L ce +w1L dice +w2L bce #(2-1)

[0045] Here, w0, w1, and w2 represent the weights corresponding to different loss functions.

[0046] (1)L ce The cross-entropy loss of the face mask is represented by the following formula:

[0047]

[0048] In the formula, i represents the pixel index; N = H × W represents the number of pixels; c represents the pixel category index, and C = 2 represents the number of categories; This indicates that the i-th pixel corresponds to the true label of class c; This represents the predicted probability that the i-th pixel corresponds to the c-th class.

[0049] (2)L dice The Dice loss representing the face mask is expressed by the following formula:

[0050]

[0051] In the formula, i represents the pixel index; N = H × W represents the number of pixels; c represents the pixel category index, and C = 2 represents the number of categories; This indicates that the i-th pixel corresponds to the true label of class c; This represents the predicted probability that the i-th pixel corresponds to the c-th class.

[0052] (3)L bce This represents the balanced cross-wrap loss for face edge masks. Face edge masks suffer from a severe imbalance between positive and negative samples, where the number of pixels corresponding to positive edges is far less than the number of pixels corresponding to negative background. Directly applying cross-wrap loss would cause the background influence to overwhelm the edge influence. To address this issue, an improved balanced cross-wrap loss is proposed, with the following formula:

[0053]

[0054] In the formula, j represents the pixel number; g j s represents the j-th pixel; j Y represents the predicted probability corresponding to the j-th pixel. + Y represents the set of pixels for positive example labels, corresponding to edge pixels; - This represents the set of pixels for negative labeling, corresponding to the background pixels. β represents the balance factor, specifically as follows:

[0055]

[0056] In the formula, |Y|=|Y + |+|Y - | represents the total number of pixels, |Y + | and |Y - | represent the number of edge pixels and the number of background pixels, respectively.

[0057] Furthermore, the method of adaptively adjusting the weights of the preset loss function includes:

[0058] During model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective.

[0059] In this embodiment, in order to achieve the optimal weight combination w t By transforming the multi-task optimization problem into a multi-objective optimization problem, the weights of each task are adaptively adjusted during network training, thereby obtaining an approximate Pareto optimal solution for the network, and thus the corresponding optimal network model. The formula for the multi-task optimization problem is as follows:

[0060]

[0061] In the formula, t represents the sequence number of different tasks; θ represents the model parameters; w t This represents the weight of different tasks, and θ sh This represents the model parameters shared by all tasks; θ t These represent parameters specific to different tasks; This indicates the loss of experience from the task.

[0062] To obtain the optimal weight w t A grid search can be used to find the optimal solution, but this incurs a large computational cost and makes it difficult to find the optimal solution. Therefore, in this embodiment, the multi-task optimization problem is transformed into a multi-objective optimization problem, with the goal of obtaining the Pareto optimal solution for each objective. The formula is as follows:

[0063]

[0064] To simplify the explanation, we consider two scenarios involving different tasks, where the optimization problem can be defined as:

[0065]

[0066] Optimal weight coefficient ( (An intermediate result for optimizing the weights) is the analytical solution to the above formula (2-8):

[0067]

[0068] in[·] + This means cropping the number to [0,1].

[0069] In practice, to improve efficiency, the Frank-Wolfe parser is also used to update the weight coefficients; see [link to relevant documentation]. Figure 3 As shown, the Frank-Wolfe parser's update process for the weight coefficients includes:

[0070] (1) Initialize the weight coefficients:

[0071] (2) Calculation

[0072] (3) Determine whether Or, if the preset number of iterations has been exceeded, return w. 1 ,…,w T And end; otherwise, Until judgment Or it exceeds the preset number of iterations.

[0073] In summary, by introducing a gradient-based multi-objective optimization method into the objective function, the weight coefficients of each objective function can be adaptively adjusted in each iteration, thereby obtaining the optimal face segmentation model. In practical applications, this face segmentation model can significantly improve both edge accuracy and overall segmentation performance.

[0074] Reference Figure 4 The diagram shown is a structural schematic of a face segmentation device provided in an embodiment of the present invention.

[0075] In this embodiment, the device 40 includes:

[0076] Input unit 41 is used to acquire the image to be segmented, and input the image to be segmented into the pre-trained face segmentation model after normalizing the image to be segmented; wherein, the image to be segmented includes a face;

[0077] Processing unit 42 is used to output a face segmentation result corresponding to the image to be segmented through the face segmentation model; wherein, the face segmentation model is trained by a pre-constructed main branch network and edge branch network and the weights of a preset loss function are adaptively adjusted.

[0078] Furthermore, the loss function includes a first loss function for supervising the face mask output by the main branch network, and a second loss function for supervising the face edge mask output by the edge branch network.

[0079] Furthermore, the first loss function includes the cross-entropy loss function L of the face mask. ce and the Dice loss function L of the face mask. dice ,in,

[0080]

[0081] In the formula, i represents the pixel index, N = H × W represents the number of pixels, and c represents the pixel category index. This indicates that the i-th pixel corresponds to the true label of the c-th class. This represents the predicted probability that the i-th pixel corresponds to the c-th class.

[0082] Furthermore, the second loss function includes the balanced cross-entropy loss function L of the face edge mask. bce , In the formula, j represents the pixel index, g j Let s represent the j-th pixel. j Y represents the predicted probability corresponding to the j-th pixel. + Represents the set of pixels for positive example labels and the corresponding edge pixels, Y - This represents the set of pixels labeled as negative examples and the corresponding background pixels, with β representing the balance factor.

[0083] Furthermore, the method of adaptively adjusting the weights of the preset loss function includes:

[0084] During model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective.

[0085] Furthermore, during model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective, including:

[0086] According to the preset formula The multi-task optimization is transformed into multi-objective optimization, where the formula for multi-task optimization is: In the formula, t represents the sequence number of different tasks, θ represents the model parameters, and w t Represents the weights of different tasks, and θ sh θ represents the model parameters shared by all tasks. t These represent parameters specific to different tasks. This indicates the loss of experience from the task.

[0087] Furthermore, the method of adaptively adjusting the weights of the preset loss function also includes:

[0088] The weights are updated using the Frank-Wolfe parser.

[0089] Each unit module of the device 40 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0090] This invention also provides a device comprising the face segmentation apparatus described above, wherein the face segmentation apparatus can employ... Figure 4 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0091] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0092] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0093] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0094] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0095] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The face segmentation method shown is illustrated. The computer-readable storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0096] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0097] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A face segmentation method, characterized in that, The method includes: The image to be segmented is obtained, and after normalization, it is input into a pre-trained face segmentation model; wherein, the image to be segmented includes a face; The face segmentation model outputs a face segmentation result corresponding to the image to be segmented; wherein, the face segmentation model is trained by a pre-constructed main branch network and edge branch network, and the weights are adaptively adjusted based on a preset loss function; the weights adaptively adjusted based on the preset loss function include: During model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective; including: According to the preset formula The multi-task optimization is transformed into multi-objective optimization, where the formula for multi-task optimization is: In the formula Indicates the sequence number of different tasks. Indicates model parameters, Represents the weights of different tasks, and , This represents the model parameters shared by all tasks. These represent parameters specific to different tasks. This indicates the loss of experience during the task; The main branch network is an Encoder-Decoder network structure, responsible for predicting the final face segmentation result; the edge branch network predicts the segmentation edges by integrating multi-layer features from the Encoder stage network, and the edge branch network includes residual modules with strides of 1 and 2, and a channel fusion module with a 1x1 convolution kernel; The loss function includes a first loss function for supervising the face mask output by the main branch network, and a second loss function for supervising the face edge mask output by the edge branch network; wherein, the first loss function includes the cross-entropy loss function of the face mask. and the Dice loss function of the face mask. The second loss function includes the balanced cross-entropy loss function of the face edge mask. The loss function is: , , and These represent the corresponding weights.

2. The face segmentation method according to claim 1, characterized in that, , In the formula Indicates the pixel number. Indicates the number of pixels. Indicates the category number of the pixel. Indicates the first The pixel corresponds to the first The true label of the class, Indicates the first The pixel corresponds to the first The predicted probability of a class.

3. The face segmentation method according to claim 1, characterized in that, In the formula Indicates the pixel number. Indicates the first 1 pixel, Indicates the first The predicted probability corresponding to each pixel. This represents the set of pixels for positive example labels and the corresponding edge pixels. This represents the set of pixels for the negative label and the corresponding background pixels. This represents the balance factor.

4. The face segmentation method according to claim 1, characterized in that, The method of adaptively adjusting the weights of a preset loss function also includes: The weights are updated using the Frank-Wolfe parser.

5. A face segmentation device, characterized in that, The device includes: The input unit is used to acquire the image to be segmented, and after normalizing the image to be segmented, input it into a pre-trained face segmentation model; wherein, the image to be segmented includes a face; The processing unit is configured to output a face segmentation result corresponding to the image to be segmented using the face segmentation model; wherein the face segmentation model is trained by a pre-constructed main branch network and edge branch network, and the weights are adaptively adjusted based on a preset loss function; the weights adaptively adjusted based on the preset loss function include: During model training, the multi-task optimization of the face segmentation model is transformed into multi-objective optimization according to a preset formula to obtain the Pareto optimal solution of the objective; including: According to the preset formula The multi-task optimization is transformed into multi-objective optimization, where the formula for multi-task optimization is: In the formula Indicates the sequence number of different tasks. Indicates model parameters, Represents the weights of different tasks, and , This represents the model parameters shared by all tasks. These represent parameters specific to different tasks. This indicates the loss of experience during the task; The main branch network is an Encoder-Decoder network structure, responsible for predicting the final face segmentation result; the edge branch network predicts the segmentation edges by integrating multi-layer features from the Encoder stage network, and the edge branch network includes residual modules with strides of 1 and 2, and a channel fusion module with a 1x1 convolution kernel; The loss function includes a first loss function for supervising the face mask output by the main branch network, and a second loss function for supervising the face edge mask output by the edge branch network; wherein, the first loss function includes the cross-entropy loss function of the face mask. and the Dice loss function of the face mask. The second loss function includes the balanced cross-entropy loss function of the face edge mask. The loss function is: , , and These represent the corresponding weights.

6. A device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a face segmentation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of a face segmentation method as described in any one of claims 1 to 4.

Images