Lip shape correction system based on deep learning

Abstract

The invention discloses a lip shape correction system based on deep learning. The lip shape correction system comprises an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module and a lip shape correction module. The invention belongs to the field of image processing, and particularly relates to a lip shape correction system based on deep learning, which is characterized in that a dynamic gray scale weighted mask is introduced, and the gradient of a low gray scale region is remarkably amplified; the dynamic weight is suppressed at a high gray level, and an edge sensing item is added into a loss function, so that the sensitivity to the edge of the lip line is enhanced, and the segmentation accuracy of the key region of the lip is improved; the subsequent lip shape correction reliability is further improved; in combination with rotation area representation and endpoint fitting, a lip-shaped key area is accurately depicted; endpoint offset penalty loss and angle fitting loss are introduced to accurately penalize angular point errors, fitting jump is eliminated, the positioning accuracy of lip shape key control points is improved, and accurate anchor points are provided; and the final lip shape correction effect is further improved.

Description

TECHNICAL FIELD

[0001] The application relates to the field of image processing, in particular to a lip shape correction system based on deep learning. BACKGROUND

[0002] The lip shape correction system is based on a lip image, analyzes a lip shape contour and key features, and corrects the problems of asymmetry and edge breakage, so as to optimize the standardization and integrity of the lip shape in the image. However, the general lip shape correction system has the problems that the difference in response of the lip region to the color temperature of a light source is ignored, the brightness is inconsistent, the weak feature region of the lip line and lip wrinkle with low contrast is prone to be missed, and the subsequent lip shape correction reliability is low; the detection ability of the small-scale lip wrinkle details and the large-scale lip shape contour is greatly different, the positioning of the key control points of the lip shape is not accurate due to the corner point offset, and the lip shape correction effect is poor. SUMMARY

[0003] In view of the above problems, the lip shape correction system based on deep learning is provided to overcome the defects of the prior art. The dynamic gray scale weighting mask is introduced to significantly amplify the gradient of the low gray scale region and preferentially learn the tiny details of the lip line and lip wrinkle. The dynamic weight is suppressed at high gray scale to weaken the loss of the lip background region. The edge perception term is added to the loss function to strengthen the sensitivity to the lip line edge, reduce the edge discontinuity and break false negatives, and improve the segmentation accuracy of the key region of the lip. The subsequent lip shape correction reliability is improved. The rotation region representation and the end point fitting are combined to accurately depict the key region of the lip shape at any angle and inclination. The end point offset penalty loss and the angle fitting loss are introduced to accurately punish the corner point error, eliminate the fitting jump caused by the periodicity of the angle, improve the positioning accuracy of the key control points of the lip shape, and provide accurate anchor points. The final lip shape correction effect is improved.

[0004] The technical scheme adopted by the application is as follows: the lip shape correction system based on deep learning provided by the application comprises an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module, and a lip shape correction module.

[0005] The image acquisition module acquires original lip image data.

[0006] The image preprocessing module performs color correction on the original lip image data and segments the main body of the lip.

[0007] The lip key region segmentation model design module converts the preprocessed lip image into grayscale and normalizes it, and realizes lip key region segmentation model design based on a dynamic grayscale weighted mask;

[0008] The lip shape feature detection model design module uses ResNet-50 as the backbone to construct a multi-scale feature pyramid, and realizes lip shape feature detection model design through a comprehensive loss of rotating rectangle fitting and combining endpoint offset penalty to locate the lip shape key control points.

[0009] The lip shape correction module uses the detected control points to complete lip shape correction through geometric correction, edge completion and lip line optimization.

[0010] Further, the image acquisition module is to obtain the original lip image data; and image labeling is performed, including pixel-level mask labeling and region endpoint labeling.

[0011] Further, the image preprocessing module is to perform color temperature correction, solve a 3x3 color temperature mapping matrix using two sets of skin color reference boards as references, enhance local contrast by statistically analyzing the gray scale distribution in a local window, and stretch the dynamic range; lip main region segmentation is performed using a lightweight U-Net model to segment the lip main body to obtain the preprocessed lip image.

[0012] Further, the lip key region segmentation model design module converts the preprocessed lip image into grayscale and linearly normalizes it to [0, 1]; and performs dynamic grayscale weighted masking to define a dynamic grayscale weighted mapping; the model architecture uses a lightweight U-Net as the backbone; the encoder uses a depth separable convolution, and the decoder adds a skip connection; a self-attention module is embedded at each skip connection; an edge perception term is introduced to finally construct a lip segmentation loss; and the lip key region segmentation model is trained through the preprocessed lip image.

[0013] Further, the lip shape feature detection model design module is used to locate the position and direction of the lip shape key control points, output feature coordinates and angle information, and specifically includes the following contents:

[0014] A multi-scale feature fusion unit; a feature pyramid network is constructed with ResNet-50 as the backbone; multi-scale features are fused from bottom to top, low-level features capture lip line details, and high-level features capture overall lip shape contours;

[0015] A rotating region fitting unit; region endpoint labeling is performed to label the lip key region with a rotating rectangle and output the endpoints.

[0016] The loss construction unit; an endpoint offset penalty term is introduced to construct a comprehensive metric; an angle fitting loss is introduced to construct a total loss.

[0017] Further, the lip shape correction module is real-time acquisition of lip image, and the control point coordinates output by the lip shape feature detection model design module are corrected in lip shape; including: lip contour geometry correction, taking the lip midline as the reference, calculating the deviation of the left and right symmetry points, if the distance difference between the left lip peak and the midline and the right lip peak exceeds 2 pixels, the coordinates are corrected through coordinate transformation; lip line edge completion, according to the rotation area angle , the reference direction vector of the lip line is calculated; the end points of the broken part are linearly interpolated and completed along the direction; the lip line detail optimization replaces the isolated noise points in the lip line area with median filtering; finally, the lip shape correction is realized.

[0018] The beneficial effects obtained by the above-mentioned scheme of the present application are as follows:

[0019] (1) For the general lip shape correction system, the difference in response to the light source color temperature of the lip region is ignored, resulting in inconsistent brightness, and the weak feature region of the low-contrast lip line and lip line is easy to be missed, and then the reliability of the subsequent lip shape correction is low, the dynamic gray weight mask is introduced, the gradient of the low gray region is significantly amplified, and the small details of the lip line and lip line are preferentially learned; the dynamic weight is suppressed at high gray, the loss of the lip background region is weakened, the edge perception term is added in the loss function, the sensitivity to the lip line edge is strengthened, the edge discontinuity and broken false negative are reduced, and the segmentation accuracy of the key region of the lip is improved; and then the reliability of the subsequent lip shape correction is improved.

[0020] (2) For the general lip shape correction system, the detection ability of small-scale lip line details and large-scale lip contour is greatly different, the positioning of the key control points of the lip shape is not accurate due to the corner point offset, and then the lip shape correction effect is poor, the rotation area representation and the end point fitting are combined to accurately depict the key region of the lip shape at any angle and inclination; the end point offset penalty loss and the angle fitting loss are introduced to accurately punish the corner point error, eliminate the fitting jump caused by the periodicity of the angle, improve the positioning accuracy of the key control points of the lip shape, and provide accurate anchor points; and then the final lip shape correction effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of a lip shape correction system based on deep learning is provided for the present application;

[0022] Figure 2 A functional unit diagram of the lip shape feature detection model design module.

[0023] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of the specification, illustrate embodiments of the application, and are used to explain the application without limiting the application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0025] In the description of the application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0026] Embodiment one, refer to Figure 1 The application provides a lip shape correction system based on deep learning, which comprises an image acquisition module, an image preprocessing module, a lip key area segmentation model design module, a lip feature detection model design module and a lip shape correction module.

[0027] The image acquisition module acquires original lip image data, and sends the data to the image preprocessing module.

[0028] The image preprocessing module performs color correction on the original lip image data, and segments the lip main body, and sends the data to the lip key area segmentation model design module.

[0029] The lip key area segmentation model design module converts the preprocessed lip image into grayscale and normalizes it, and based on a dynamic grayscale weighted mask, realizes lip key area segmentation model design, and sends the data to the lip feature detection model design module.

[0030] The lip feature detection model design module, for the lip key area segmentation result, takes ResNet-50 as the backbone, constructs a multi-scale feature pyramid, realizes lip feature detection model design through a comprehensive loss of rotating rectangle fitting and combining endpoint offset penalty, and locates the lip key control points, and sends the data to the lip shape correction module.

[0031] The lip shape correction module uses the detected control points to complete lip shape correction through geometric correction, edge completion and lip line optimization.

[0032] Embodiment two, refer to Figure 1 , this embodiment is based on the above embodiment, the image acquisition module is used when the shooting object keeps the face relaxed. The 500 million pixel industrial camera is fixed by using the fixed height support, the camera lens is parallel to the lip plane, the 50mm fixed focus macro lens is adopted, the texture features of the key area of the lip are captured, the original lip image data is obtained; and image annotation is carried out, including pixel level mask annotation and region endpoint annotation; the pixel level mask annotation is to annotate whether the pixel belongs to the lip line lip wrinkle area; the region endpoint annotation is to annotate the rotating rectangular area and the region endpoint for each lip shape key area, including the upper lip line, the lower lip line, the lip peak connecting line and the lip angle to the lip valley.

[0033] Embodiment three, refer to Figure 1 , this embodiment is based on the above embodiment, the image preprocessing module is easy to be affected by skin color difference and light source color temperature. The image characteristics are unified through preprocessing, and the weak contrast features of the lip line and the lip wrinkle are enhanced; the specific operation is as follows: color temperature correction, two groups of skin color reference plates are used as reference to solve the 3*3 color temperature mapping matrix R, and the lip color deviation under different light sources is eliminated, which is expressed as: ; wherein, is the original lip image; is the corrected lip image; local contrast enhancement, for the low contrast area between the lip line and the surrounding skin, the gray scale distribution is counted through the local window, the dynamic range is stretched, the natural texture inside the lip red area is retained, and the noise amplification caused by excessive enhancement is avoided; lip main body segmentation, the lightweight U-Net model is used to segment the lip main body, and the preprocessed lip image is obtained.

[0034] Embodiment four, refer to Figure 1 , this embodiment is based on the above embodiment, the lip key area segmentation model design module converts the preprocessed lip image into gray scale and linearly normalizes it to [0, 1], eliminates the brightness difference under different collection conditions, and unifies the scale; and dynamic gray scale weighting mask, the gray value of the lip line edge and the fine lip wrinkle is usually lower than that of the lip main body. The weight in the loss function is amplified through the weighted mask, so as to force the segmentation model to preferentially learn the weak features, define the dynamic gray scale weighting mapping, which is expressed as: ; ; ; wherein, is the local contrast coefficient; is the normalized gray value of position (i,j); is the average gray value in the local neighborhood; is the normalized local contrast coefficient; maxC is the global maximum contrast; is the smoothing term; is the segmentation loss weight; is a hyperbolic tangent function; is a dynamic sensitivity parameter; when , corresponding to the lip line edge, significantly amplifying the gradient of the corresponding area; when , corresponding to the lip body, avoiding overfitting of the segmentation model to the high gray background, ensuring that the weak features of the lip line and lip lines are accurately segmented; Model architecture: backbone adopts lightweight U-Net; encoder uses deep separable convolution, and decoder adds jump connection; At each jump connection, embed a self-attention module to focus on feature reconstruction of key positions of lip peaks and lip corners; Introduce edge perception term to improve the continuity of the lip line contour and avoid lip line rupture caused by lip line interference, finally build a lip segmentation loss, expressed as: ; ; wherein, is the basic loss; is the segmentation real label, represents the background, represents the lip line and lip line area; is the predicted lip line and lip line area probability; is the gradient; is the edge perception weight; is the loss function of the lip key area segmentation model; through the mask M, it directly guides to learn the lip line details preferentially; the binary threshold , extract the connected domain, remove the noise whose area is less than the preset threshold, realize the lip key area segmentation; train the lip key area segmentation model through the preprocessed lip image.

[0035] By performing the above operations, the general lip shape correction system ignores the response difference of the lip area to the light source color temperature, leading to inconsistent brightness, and the weak feature area of the low-contrast lip line and lip line is easy to be missed, which further leads to low reliability of subsequent lip shape correction. The scheme introduces a dynamic gray weight mask, significantly amplifies the gradient of the low gray area, and preferentially learns the tiny details of the lip line and lip line; through the dynamic weight, the weight is suppressed when the gray is high, the loss of the lip background area is weakened, the edge perception term is added in the loss function, the sensitivity to the lip line edge is strengthened, the edge discontinuity and rupture false negative are reduced, and the accuracy of the lip key area segmentation is improved; Further improve the reliability of subsequent lip shape correction.

[0036] Embodiment five, refer to Figure 1 and Figure 2 , this embodiment is based on the above embodiment, the lip feature detection model design module is used for accurately positioning the position and direction of the lip feature control point, outputting feature coordinates and angle information, providing anchor points for subsequent lip shape correction, which specifically includes the following contents:

[0037] A multi-scale feature fusion unit; taking ResNet-50 as the backbone, a feature pyramid network is constructed; the multi-scale features are fused from bottom to top, the low-level features capture the details of the lip lines, and the high-level features capture the overall contour of the lips; the equal detection capability of the small-scale features of the lip valley micro-depression and the large-scale features of the whole upper lip contour is ensured;

[0038] A rotation region fitting unit; the lip shape key region including the upper lip line between the lip peaks and the lower lip line from the lip corners to the lip valley is often in an inclined state, and the lip corners are raised when smiling, so the position and angle are labeled by the rotation region to realize accurate positioning in any direction; the region endpoint is labeled, the lip key region is labeled by a rotating rectangle, and the format is: , and the output endpoint , is expressed as: ; ; ; ; ; ; ; ; wherein, is the four endpoint coordinates; (x, y) is the center; is the region width and height; is the rotation angle relative to the horizontal axis; the endpoint sorting is to ensure that the endpoint of the predicted region corresponds to the endpoint of the real region one by one, and the endpoints of the real region and the predicted region are sorted in ascending order according to the x coordinate; five tuples are predicted for each anchor point; T is the transpose; the predicted region and the real region are converted into endpoints to ensure one-to-one correspondence; the fitting of improves the positioning accuracy at any angle;

[0039] A loss construction unit; an endpoint offset penalty term is introduced; the standard rotation region overlap rate is represented as: ; wherein, is the polygon region corresponding to the real region; is the polygon region corresponding to the predicted region; the endpoint offset penalty term is introduced and represented as: ; wherein, is the endpoint offset penalty term; is the real region endpoint; is the corresponding endpoint predicted by the model; the comprehensive measure IU is represented as: ; wherein, and are the width and height of the real region respectively; the fitting loss is represented as: ; the angle fitting loss is introduced, and the overall loss is represented as: ; ; wherein, is the standard cross-entropy loss; is the real region angle; and is the loss weight; the comprehensive metric takes into account the overall overlap of the region and the end details, suitable for small region positioning of the lip peak; the angle loss can distinguish the natural trend of the lip line from the lip print interference, reduce false positives, and provide accurate feature coordinates for lip shape correction.

[0040] Embodiment six, refer to Figure 1 , which is based on the above embodiment, the lip shape correction module is to obtain the lip image in real time, and the control point coordinates output by the lip shape feature detection model design module are subjected to lip shape correction; the specific operation is: lip contour geometric correction, taking the lip midline as the reference, calculating the deviation of the left and right symmetrical points, if the distance difference between the left lip peak and the right lip peak exceeds 2 pixels, the coordinates are corrected through coordinate transformation, which is represented as: ; is the original x-coordinate of the lip peak on one side without correction; is the x-coordinate of the lip peak on the other side after correction; when the lip angle height is not symmetrical, the y-coordinate symmetry adjustment is realized in the same way to balance; based on the detected lip line rotation angle, if it deviates from the natural lip shape angle range, the affine rotation correction is performed, all points (x, y) in the lip line region are rotated around the lip valley, which is represented as: ; ; is the standard lip line rotation angle; is the corrected angle; wherein, is the coordinate after affine rotation correction; is the center coordinate of the four end points, corresponding to the lip valley point coordinate; lip line edge completion, according to the rotation region angle , the reference direction vector of the lip line is calculated, which is represented as: ; assuming that the two end points of the broken part are A(x a ,y a ) and B(x b ,y b ), if the deviation between the direction of the line connecting the two end points of the broken part and the reference direction exceeds 10 degrees, the two end points of the broken part are projected onto the straight line in the direction to obtain the corrected end points, and the end points are linearly interpolated along the direction to complete the completion point P(x, y) which satisfies ; ; wherein, and are the coordinates of the two end points of the lip line broken part, and t is the linear interpolation parameter; ensure that the completed edge extends along the direction and smoothly connects with the original lip line; lip print detail optimization, replacing the isolated noise points in the lip print region with median filtering; based on the detected lip print main direction By low layer feature analysis, for the lip line with disordered direction, select the local and overall direction difference greater than 15 degrees, adjust through direction filtering: adopt Gaussian direction filter, main direction , reserve the texture along the direction, suppress the interference in the vertical direction, make the lip line consistent with the lip line, and finally realize the lip shape correction.

[0041] By performing the above operation, for the general lip shape correction system, there is a large difference between the detection ability of small scale lip line details and large scale lip shape contour, the key control point positioning of the lip shape is not accurate due to the corner point offset, and then the lip shape correction effect is poor. The scheme combines the rotation area representation and the endpoint fitting, accurately describes the key area of the lip shape with any angle and inclination; introduce the endpoint offset penalty loss and the angle fitting loss to accurately punish the corner point error, eliminate the fitting jump caused by the periodicity of the angle, improve the positioning accuracy of the key control point of the lip shape, and provide accurate anchor points; and then improve the final lip shape correction effect.

[0042] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application.

[0043] The above describes the present application and its embodiments, which is not restrictive, and the shown in the drawings is only one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution can be designed, which shall belong to the protection scope of the present application.

Claims

1. A deep learning-based lip shape correction system, characterized by: The system comprises an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module and a lip shape correction module. The image acquisition module acquires original lip image data. The image preprocessing module performs color correction on the original lip image data and segments the lip main body. The lip key region segmentation model design module converts the preprocessed lip image into grayscale and normalizes it, and based on a dynamic grayscale weighted mask, realizes lip key region segmentation model design. The lip shape feature detection model design module uses ResNet-50 as the backbone, constructs a multi-scale feature pyramid, and through a comprehensive loss combining rotating rectangle fitting and endpoint offset penalty, realizes lip shape feature detection model design and locates the lip shape key control points. The lip shape correction module uses the detected control points to complete lip shape correction through geometric correction, edge completion and lip line optimization. The lip key region segmentation model design module includes a dynamic gray scale weighting map defined as: ; ; ; wherein, is a local contrast coefficient; is a normalized gray scale value at position (i,j); is a mean gray scale value within a local neighborhood; is a normalized local contrast coefficient; maxC is a global maximum contrast; is a smoothing term; is a segmentation loss weight; is a hyperbolic tangent function; is a dynamic sensitivity parameter.

2. The deep learning-based lip correction system of claim 1, wherein: The image preprocessing module performs color temperature correction, uses two sets of skin color reference plates as references to solve a 3x3 color temperature mapping matrix, enhances local contrast by statistically analyzing the gray scale distribution in a local window, and stretches the dynamic range, and uses a lightweight U-Net model to segment the lip main body to obtain the preprocessed lip image.

3. The deep learning-based lip shape correction system of claim 2, wherein: The lip key region segmentation model design module converts the preprocessed lip image into grayscale and linearly normalizes it to [0,1], and performs dynamic grayscale weighted masking and defines a dynamic grayscale weighted mapping. The model architecture uses a lightweight U-Net as the backbone, uses deep separable convolution as the encoder, and adds a skip connection to the decoder.

4. The lip shape correction system based on deep learning according to claim 3, characterized in that: An attention module is embedded at each skip connection. An edge perception term is introduced to finally construct a lip segmentation loss. The lip key region segmentation model is trained using the preprocessed lip image. The lip shape feature detection model design module is used to locate the position and direction of the lip shape key control points, output feature coordinates and angle information, and specifically includes the following contents: A multi-scale feature fusion unit uses ResNet-50 as the backbone to construct a feature pyramid network.

5. The deep learning-based lip shape correction system of claim 4, wherein: The multi-scale features are fused from bottom to top, with low-level features capturing lip line details and high-level features capturing overall lip shape contours. A rotating region fitting unit labels the lip key region with a rotating rectangle and outputs the end points. A loss construction unit introduces an endpoint offset penalty term to construct a comprehensive metric and an angle fitting loss to construct a total loss.

6. The deep learning-based lip shape correction system of claim 5, wherein: The lip shape correction module is to acquire lip images in real time, correct the control point coordinates output by the lip shape feature detection model design module, and include: lip contour geometry correction, taking the lip midline as a reference, calculating the deviation of left and right symmetry points, if the distance difference between the left lip peak and the midline and the right lip peak exceeds 2 pixels, correcting through coordinate transformation; lip line edge completion, calculating the reference direction vector of the lip line according to the rotation area angle , linearly interpolating and completing the endpoints of the broken part along the direction; lip wrinkle detail optimization, replacing isolated noise points in the lip wrinkle area with median filtering; and finally realizing lip shape correction.

Images