A Deep Learning-Based Lip Correction System

Abstract

This invention discloses a deep learning-based lip correction system, comprising an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip feature detection model design module, and a lip correction module. This invention belongs to the field of image processing, specifically referring to a deep learning-based lip correction system. This scheme introduces a dynamic grayscale weighted mask to significantly amplify the gradient in low grayscale regions; it suppresses weights in high grayscale regions through dynamic weights, adds an edge-aware term to the loss function to enhance sensitivity to lip line edges, and improves the accuracy of lip key region segmentation; thereby improving the reliability of subsequent lip correction; it combines rotation region representation with endpoint fitting to accurately characterize the key lip region; and it introduces endpoint offset penalty loss and angle fitting loss to precisely penalize corner point errors, eliminate fitting jumps, improve the positioning accuracy of key lip control points, and provide precise anchor points; thus improving the final lip correction effect.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a deep learning-based lip correction system. Background Technology

[0002] Lip correction systems analyze lip contours and key features based on lip images to specifically correct asymmetry and edge breaks, optimizing the regularity and integrity of the lip shape in the image. However, typical lip correction systems often ignore the differences in the lip region's response to light source color temperature, leading to inconsistent brightness. Low-contrast lip lines and weak feature areas are easily missed, resulting in low reliability of subsequent lip correction. Furthermore, typical lip correction systems exhibit significant differences in detection capabilities between small-scale lip line details and large-scale lip contours. Corner point offsets cause inaccurate localization of key lip control points, resulting in poor lip correction effects. Summary of the Invention

[0003] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention provides a deep learning-based lip correction system. Addressing the problems of conventional lip correction systems neglecting the varying responses of the lip region to light source color temperature, leading to inconsistent brightness and missed detection of weak feature areas in low-contrast lip lines and wrinkles, thus resulting in low reliability of subsequent lip correction, this solution introduces a dynamic grayscale weighted mask to significantly amplify gradients in low-grayscale regions, prioritizing the learning of minute details in lip lines and wrinkles. By dynamically weighting lip lines at high grayscale levels, weights are suppressed to reduce loss in the lip background region. Furthermore, an edge-aware term is added to the loss function to enhance sensitivity to lip line edges and reduce edge discontinuities. This approach eliminates false negatives related to breakage, improving the accuracy of key lip region segmentation and thus enhancing the reliability of subsequent lip shape correction. Addressing the issue of significant differences in detection capabilities between small-scale lip wrinkle details and large-scale lip contours in general lip shape correction systems, and the inaccurate positioning of key lip control points due to corner point offsets, resulting in poor lip shape correction, this solution combines rotational region representation with endpoint fitting to accurately characterize key lip regions at arbitrary angles and tilts. It introduces endpoint offset penalty loss and angle fitting loss to precisely penalize corner point errors, eliminating fitting jumps caused by angle periodicity, improving the positioning accuracy of key lip control points, and providing precise anchor points; thereby improving the final lip shape correction effect.

[0004] The technical solution adopted by the present invention is as follows: The present invention provides a lip shape correction system based on deep learning, including an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module, and a lip shape correction module;

[0005] The image acquisition module acquires raw lip image data;

[0006] The image preprocessing module performs color correction on the original lip image data and segments the main body of the lips;

[0007] The lip key region segmentation model design module converts the preprocessed lip image to grayscale and normalizes it, and then implements the lip key region segmentation model design based on dynamic grayscale weighted mask.

[0008] The lip feature detection model design module uses ResNet-50 as the backbone to construct a multi-scale feature pyramid based on the lip key region segmentation results. It achieves lip feature detection model design and locates key lip control points through a comprehensive loss combining rotation rectangle fitting and endpoint offset penalty.

[0009] The lip shape correction module uses the detected control points to complete the lip shape correction through geometric correction, edge completion, and lip line optimization.

[0010] Furthermore, the image acquisition module acquires raw lip image data and performs image annotation, including pixel-level mask annotation and region endpoint annotation.

[0011] Furthermore, the image preprocessing module performs color temperature correction by using two sets of skin tone reference plates as references and solving a 3×3 color temperature mapping matrix; enhances local contrast by statistically analyzing grayscale distribution through local windows and stretching the dynamic range; and segments the lip main body region by using a lightweight U-Net model to segment the lip main body to obtain the preprocessed lip image.

[0012] Furthermore, the lip key region segmentation model design module converts the preprocessed lip image to grayscale and linearly normalizes it to [0,1]; and performs dynamic grayscale weighted masking to define dynamic grayscale weighted mapping; model architecture: the backbone adopts lightweight U-Net; the encoder adopts depthwise separable convolution, and the decoder adds skip connections; a self-attention module is embedded at each skip connection; an edge-aware term is introduced, and finally a lip segmentation loss is constructed; the lip key region segmentation model is trained using the preprocessed lip image.

[0013] Furthermore, the lip shape feature detection model design module is used to locate the position and orientation of key control points of the lip shape, and output feature coordinates and angle information, specifically including the following:

[0014] Multi-scale feature fusion unit; using ResNet-50 as the backbone to construct a feature pyramid network; fusing multi-scale features from bottom to top, with low-level features capturing lip texture details and high-level features capturing the overall outline of the lips;

[0015] Rotation region fitting unit; region endpoint labeling, labeling key lip regions with a rotation rectangle, and outputting endpoints;

[0016] Loss building blocks; an endpoint offset penalty term is introduced to build a comprehensive metric; an angle fitting loss is introduced to build the overall loss.

[0017] Furthermore, the lip shape correction module acquires lip images in real time and performs lip shape correction based on the control point coordinates output by the lip shape feature detection model design module; this includes: geometric correction of the lip contour, using the lip midline as a reference, calculating the deviation of the left and right symmetrical points; if the difference between the distance from the left lip peak to the midline and the distance from the right lip peak exceeds 2 pixels, correction is performed through coordinate transformation; and lip line edge completion, based on the rotation area angle. Calculate the reference direction vector of the lip line; along the endpoint of the break. Directional linear interpolation is used for completion; lip line detail optimization is performed by replacing isolated noise points in the lip line area with median filtering; and finally, lip shape correction is achieved.

[0018] The beneficial effects achieved by adopting the above solution are as follows:

[0019] (1) To address the problem that general lip correction systems ignore the differences in the response of the lip region to the color temperature of the light source, resulting in inconsistent brightness and easy omission of weak feature areas of low-contrast lip lines and lip lines, which leads to low reliability of subsequent lip correction, this scheme introduces a dynamic gray-level weighted mask to significantly amplify the gradient of low gray-level regions and prioritize the learning of the minute details of lip lines and lip lines; by suppressing the weights at high gray levels through dynamic weights, the loss of the lip background region is reduced; and an edge-aware term is added to the loss function to enhance the sensitivity to the lip line edge, reduce edge discontinuities and false negatives of breakage, and improve the accuracy of lip key region segmentation; thereby improving the reliability of subsequent lip correction.

[0020] (2) In view of the problem that general lip correction systems have large differences in detection capabilities between small-scale lip wrinkle details and large-scale lip contours, and that the positioning of key lip control points is inaccurate due to corner point offset, resulting in poor lip correction effect, this solution combines rotation region representation and endpoint fitting to accurately characterize key lip regions at arbitrary angles and tilts; it introduces endpoint offset penalty loss and angle fitting loss to accurately penalize corner point errors, eliminate fitting jumps caused by angle periodicity, improve the positioning accuracy of key lip control points, and provide accurate anchor points; thereby improving the final lip correction effect. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a deep learning-based lip correction system provided by this invention;

[0022] Figure 2 A schematic diagram of the functional units of the module for designing the lip feature detection model.

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0025] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0026] Example 1, see Figure 1 The present invention provides a deep learning-based lip shape correction system, including an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module, and a lip shape correction module;

[0027] The image acquisition module acquires raw lip image data and sends the data to the image preprocessing module.

[0028] The image preprocessing module performs color correction on the original lip image data and segments the main body of the lips; and sends the data to the lip key region segmentation model design module.

[0029] The lip key region segmentation model design module converts the preprocessed lip image to grayscale and normalizes it, and then designs the lip key region segmentation model based on a dynamic grayscale weighted mask; and sends the data to the lip shape feature detection model design module.

[0030] The lip feature detection model design module uses ResNet-50 as the backbone to construct a multi-scale feature pyramid based on the lip key region segmentation results. It then uses a comprehensive loss combining a rotated rectangle fitting and an endpoint offset penalty to design the lip feature detection model and locate the key control points of the lip shape. The data is then sent to the lip correction module.

[0031] The lip shape correction module uses the detected control points to complete the lip shape correction through geometric correction, edge completion, and lip line optimization.

[0032] Example 2, see Figure 1 This embodiment is based on the above embodiment. When the subject's face is relaxed, the image acquisition module uses a fixed-height bracket to fix an industrial camera with a resolution of 5 megapixels or higher, ensuring that the camera lens is parallel to the lip plane. A 50mm fixed-focus macro lens is used to focus on capturing the texture features of key areas of the lips to obtain raw lip image data. Image annotation is then performed, including pixel-level mask annotation and region endpoint annotation. The pixel-level mask annotation marks whether a pixel belongs to the lip line and lip texture area. The region endpoint annotation marks the rotating rectangular area and region endpoints for each key lip area, including the upper lip line, lower lip line, the line connecting the cupid's bow, and the line segment from the corner of the lip to the valley of the lip.

[0033] Example 3, see Figure 1 This embodiment is based on the above embodiment. The image preprocessing module addresses the issue of lip images being easily affected by skin tone differences and light source color temperature. It unifies image characteristics through preprocessing while enhancing the weak contrast features of lip lines and lip wrinkles. Specifically, it performs color temperature correction by using two sets of skin tone reference plates to solve for a 3×3 color temperature mapping matrix R, eliminating lip color deviations under different light sources. This is expressed as: ;in, This is the original lip image; This is the corrected lip image; local contrast enhancement targets the low-contrast areas of the lip line and surrounding skin by statistically analyzing the grayscale distribution through local windows, stretching the dynamic range, preserving the natural texture within the vermilion border, and avoiding noise amplification caused by over-enhancement; lip body segmentation uses a lightweight U-Net model to segment the lip body, resulting in the preprocessed lip image.

[0034] Example 4, see Figure 1 This embodiment is based on the above embodiment. The lip key region segmentation model design module converts the preprocessed lip image to grayscale and linearly normalizes it to [0,1] to eliminate brightness differences under different acquisition conditions and unify the scale. It also performs dynamic grayscale weighted masking. The grayscale values ​​of the lip line edge and fine lip lines are usually lower than those of the main lip body. By weighting the mask, their weight in the loss function is amplified, forcing the segmentation model to prioritize learning weak features. The dynamic grayscale weighted mapping is defined as follows: ; ; ;in, It is the local contrast coefficient; It is the normalized gray value at position (i,j); It is the average gray level within a local neighborhood; It is the normalized local contrast ratio; maxC is the global maximum contrast ratio; It is a smoothing term; It is the segmentation loss weight; It is the hyperbolic tangent function; It is a dynamic sensitivity parameter; when Corresponding to the lip line edge, significantly amplifying the gradient of the corresponding area; when Corresponding to the main body of the lips, to avoid the segmentation model overfitting to high grayscale backgrounds, ensuring that weak features of the lip line and lip lines are accurately segmented; Model architecture: The backbone adopts a lightweight U-Net; the encoder uses depthwise separable convolution, and the decoder adds skip connections; Self-attention modules are embedded at each skip connection to focus on feature reconstruction at key positions of the cupid's bow and corners of the lips; An edge-aware term is introduced to improve the continuity of the lip line contour and avoid lip line breakage caused by lip lines; Finally, a lip segmentation loss is constructed, represented as: ; ;in, This is a basic loss; It is to segment the real labels. Indicates background, This indicates the area within the lip line and lip lines; It is the predicted probability of the lip line and lip line area; It is the gradient; It is edge-aware weight; It is the loss function of the lip key region segmentation model; it directly guides the learning of lip line details through a mask M; and it uses a binarization threshold. The system extracts connected components and removes noise points with areas smaller than a preset threshold to achieve lip key region segmentation; the lip key region segmentation model is trained using the preprocessed lip images.

[0035] By performing the above operations, this solution addresses the common problem in general lip correction systems: neglecting the differences in the response of the lip region to the color temperature of the light source, resulting in inconsistent brightness, and the easy omission of weak feature areas of low-contrast lip lines and lip wrinkles, leading to low reliability of subsequent lip correction. This solution introduces a dynamic grayscale weighted mask to significantly amplify the gradient in low grayscale regions, prioritizing the learning of minute details of lip lines and lip wrinkles. Dynamic weights suppress weights at high grayscale levels, reducing the loss in the lip background region. An edge-aware term is added to the loss function to enhance sensitivity to lip line edges, reduce false negatives due to edge discontinuities and breaks, and improve the accuracy of key lip region segmentation; thereby improving the reliability of subsequent lip correction.

[0036] Example 5, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. The lip shape feature detection model design module is used to accurately locate the position and direction of the key control points of the lip shape, output feature coordinates and angle information, and provide anchor points for subsequent lip shape correction. Specifically, it includes the following:

[0037] Multi-scale feature fusion unit; using ResNet-50 as the backbone, a feature pyramid network is constructed; multi-scale features are fused from bottom to top, with low-level features capturing lip line details and high-level features capturing the overall lip shape contour; ensuring that the small-scale features of subtle lip valley depressions have the same detection capability as the large-scale features of the entire upper lip contour.

[0038] Rotation region fitting unit; including the upper lip line between the cupid's bow and the lower lip line from the corner of the lip to the valley of the lip, key lip areas are often tilted, with the corners of the lips rising when smiling. The rotation region is used to mark the position and angle, achieving precise positioning in any direction; region endpoint annotation, using a rotated rectangle to annotate key lip areas, in the following format: and output endpoints , is represented as: ; ; ; ; ; ; ; ;in, These are the coordinates of the four endpoints; (x, y) is the center. It refers to the area's width and height; It is the rotation angle relative to the horizontal axis; endpoint sorting: to ensure a one-to-one correspondence between the endpoints of the predicted region and the actual region, the endpoints of the actual region and the predicted region are sorted in ascending order of the x-coordinate; a quintuple is predicted for each anchor point. T is the transpose; it converts the predicted region and the true region into endpoints, ensuring a one-to-one correspondence; by... Fitting improves positioning accuracy at any angle;

[0039] Loss building blocks; introduction of endpoint offset penalty term; standard rotation region overlap rate is expressed as: ;in, It is the polygonal region corresponding to the real area; It is the polygonal region corresponding to the predicted region; an endpoint offset penalty term is introduced, expressed as: ;in, It is an endpoint offset penalty term; These are the endpoints of the real region; These are the corresponding endpoints predicted by the model; the comprehensive metric IU is expressed as: ;in, and These are the width and height of the real region, respectively; the fitting loss. Represented as: Introducing angle fitting loss Total loss Represented as: ; ;in, It is the standard cross-entropy loss; It is a real-world perspective; and It is a loss weight; the comprehensive measurement takes into account the overall overlap of the region and the details of the endpoints, which is suitable for the small area positioning of the lip peak; the angle loss can distinguish the natural direction of the lip line and the interference of lip lines, reduce false positives, and provide accurate feature coordinates for lip shape correction.

[0040] Example 6, see Figure 1 This embodiment is based on the above embodiment. The lip shape correction module acquires lip images in real time and corrects the lip shape based on the control point coordinates output by the lip shape feature detection model design module. Specifically, it performs geometric correction of the lip contour, using the lip midline as a reference, calculating the deviation of the left and right symmetrical points. If the difference between the distance from the left lip peak to the midline and the distance from the right lip peak exceeds 2 pixels, it is corrected through coordinate transformation, as shown below: ; It is the original x-coordinate of the cupid's bow on one side without correction; This is the x-coordinate of the corrected lip peak; when the lip corner height is asymmetrical, balance is achieved by symmetrical adjustment of the y-coordinate; based on the detected lip line rotation angle, if it deviates from the natural lip shape angle range, affine rotation correction is used, rotating all points (x, y) within the lip line area around the lip valley, as shown below: ; ; It is the standard lip line rotation angle; It is a correction angle; among which, These are coordinates corrected for affine rotation. These are the center coordinates of the four endpoints, corresponding to the coordinates of the lip valley point; lip line edge completion is based on the rotation area angle. Calculate the reference direction vector of the lip line. , is represented as: Let the two endpoints of the fracture be A(x). a ,y a ) and B(x b ,y b If the direction of the line connecting the two ends of the fracture deviates from the reference direction by more than 10 degrees, then the two ends of the fracture will be projected onto... On a straight line in the direction, the corrected endpoints are obtained, and the endpoints are along... Directional linear interpolation completion, the completed point P(x,y) satisfies ; ;in, and These are the coordinates of the two endpoints at the lip line break, and t is the linear interpolation parameter; ensuring the completed edge follows... The direction is extended, smoothly connecting with the original lip line; lip line details are optimized by replacing isolated noise points within the lip line area with median filtering; based on the detected main direction of the lip lines... Based on low-level feature analysis, for lip lines with disordered orientation, those with a local and global orientation difference greater than 15 degrees are selected and adjusted through orientation filtering: a Gaussian orientation filter is used, with the main orientation... Preserve along The texture of the direction suppresses vertical interference, making the lip lines consistent with the direction of the lip lines, and ultimately achieving lip shape correction.

[0041] By performing the above operations, this solution addresses the problem that general lip correction systems have significant differences in detection capabilities between small-scale lip wrinkle details and large-scale lip contours, and that inaccurate positioning of key lip control points due to corner point offset, resulting in poor lip correction effects. This solution combines rotational region representation with endpoint fitting to accurately characterize key lip regions at arbitrary angles and tilts. It introduces endpoint offset penalty loss and angle fitting loss to precisely penalize corner point errors, eliminating fitting jumps caused by angle periodicity, improving the positioning accuracy of key lip control points, and providing precise anchor points; thereby improving the final lip correction effect.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0043] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A deep learning-based lip correction system, characterized in that: The system includes an image acquisition module, an image preprocessing module, a lip key region segmentation model design module, a lip shape feature detection model design module, and a lip shape correction module; The image acquisition module acquires raw lip image data; The image preprocessing module performs color correction on the original lip image data and segments the main body of the lips; The lip key region segmentation model design module converts the preprocessed lip image to grayscale and normalizes it, and then implements the lip key region segmentation model design based on dynamic grayscale weighted mask. The lip feature detection model design module uses ResNet-50 as the backbone to construct a multi-scale feature pyramid based on the lip key region segmentation results. It achieves lip feature detection model design and locates key lip control points through a comprehensive loss combining rotation rectangle fitting and endpoint offset penalty. The lip shape correction module uses the detected control points to complete the lip shape correction through geometric correction, edge completion and lip line optimization; The lip key region segmentation model design module includes defining a dynamic gray-level weighted mapping, represented as: ; ; ;in, It is the local contrast coefficient; It is the normalized gray value at position (i,j); It is the average gray level within a local neighborhood; It is the normalized local contrast ratio; maxC is the global maximum contrast ratio; It is a smoothing term; It is the segmentation loss weight; It is the hyperbolic tangent function; It is a dynamic sensitivity parameter.

2. The lip correction system based on deep learning according to claim 1, characterized in that: The image preprocessing module performs color temperature correction by using two sets of skin tone reference plates as references and solving a 3×3 color temperature mapping matrix; local contrast enhancement by statistically analyzing grayscale distribution through local windows and stretching the dynamic range; and lip main body region segmentation by using a lightweight U-Net model to segment the lip main body to obtain the preprocessed lip image.

3. The lip correction system based on deep learning according to claim 2, characterized in that: The lip key region segmentation model design module converts the preprocessed lip image to grayscale and linearly normalizes it to [0,1]; and performs dynamic grayscale weighted masking, defining dynamic grayscale weighted mapping; Model architecture: The backbone adopts lightweight U-Net; The encoder adopts depthwise separable convolution, and the decoder adds skip connections; Self-attention modules are embedded at each skip connection; An edge-aware term is introduced, and finally, the lip segmentation loss is constructed; A lip key region segmentation model was trained using preprocessed lip images.

4. The lip correction system based on deep learning according to claim 3, characterized in that: The lip shape feature detection model design module is used to locate the position and orientation of key control points of the lip shape, and output feature coordinates and angle information, specifically including the following: Multi-scale feature fusion unit; a feature pyramid network is constructed with ResNet-50 as the backbone; Multi-scale features are integrated from bottom to top, with low-level features capturing lip line details and high-level features capturing the overall lip shape outline. Rotation region fitting unit; region endpoint annotation, using a rotated rectangle to annotate the key lip region and output the endpoints; Loss building block.

5. The lip correction system based on deep learning according to claim 4, characterized in that: The loss construction unit is constructed by introducing an endpoint offset penalty term to build a comprehensive metric and introducing an angle fitting loss to build an overall loss.

6. The lip correction system based on deep learning according to claim 5, characterized in that: The lip shape correction module acquires lip images in real time and corrects the lip shape based on the control point coordinates output by the lip shape feature detection model design module. This includes: geometric correction of the lip contour, calculating the deviation of the left and right symmetrical points based on the lip midline; if the difference between the distance from the left lip peak to the midline and the distance from the right lip peak exceeds 2 pixels, correction is performed through coordinate transformation; and lip line edge completion, based on the rotation area angle. Calculate the reference direction vector of the lip line; along the endpoint of the break. Directional linear interpolation is used for completion; lip line detail optimization is performed by replacing isolated noise points in the lip line area with median filtering; and finally, lip shape correction is achieved.

Images