Nail measuring and beautifying method and device based on deep learning

Abstract

The invention discloses a nail measurement and beautification method and device based on deep learning, and belongs to the field of nail beautification, the method uses a DeepLabV3 + model to effectively segment a nail area through ASPP and decoder networks, and even if the shape of a nail is irregular or the nail is shielded, an accurate segmentation result can be obtained. By using the YOLOv8 model, fingernail key point positions such as fingertips and fingernail center points can be rapidly and accurately detected, and an accurate reference basis is provided for subsequent measurement and beautification. The posture and orientation of the finger can be recognized by using the MMPose model, key information is provided for nail tip rendering, and it is ensured that the manicure map can be accurately attached to the hand image. By segmenting the fingernail and the calibration object and calculating the minimum enclosing rectangle, the actual length and width of the fingernail can be accurately measured without manual measurement by a user, so that the convenience in the whole use process is improved.

Description

Technical Field

[0001] The present invention belongs to the field of nail art, and specifically is a nail measurement and beautification method based on deep learning. Background Art

[0002] With the improvement of living standards, people's pursuit of beauty is getting higher and higher. As a fashionable decoration, manicure has been loved by more and more people. In recent years, people have paid more and more attention to their appearance management, and the scale of manicure users has continued to grow, which has promoted the continuous improvement of the demand for manicure market. As Generation Z gradually enters society, it has become the main consumer in the market. Spiritual consumption, personalization, quality, and appearance have become the characteristics of Generation Z users. Manicure is not just a consumption that makes nails beautiful, but a way of expressing personality and aesthetics. According to data from iiMedia Research, the market size of China's manicure industry has shown an upward trend year by year in recent years. The market size in 2021 is 185.48 billion yuan, a year-on-year increase of 25.3%. It is expected that the market size will reach 266.25 billion yuan in 2027, a year-on-year increase of 5.5%. Consumers' love of beauty and pursuit of fashion are getting stronger and stronger, and there is still room for the development of the manicure market in the future.

[0003] However, traditional nail measurement and beautification methods require users to manually measure the length, width and curvature of nails, and go to a nail salon for nail services, which increases costs and time and affects convenience during use. Summary of the invention

[0004] In order to solve the problem of automatically measuring the length, width and curvature of nails and save cost and time, the present invention provides a nail measurement and beautification method and device based on deep learning.

[0005] A nail measurement and beautification method based on deep learning, the method comprising the following steps:

[0006] S1: Collect and annotate nail segmentation datasets, including nail images of multiple fingers, different shapes and colors;

[0007] Collect and annotate a dataset of nail key points, including the locations of fingertips and nail center key points;

[0008] Collect a diverse dataset of nail art stickers, including nail art images of different colors, patterns, and styles;

[0009] S2: A nail segmentation model is established and trained based on the DeepLabV3+ model for nail segmentation;

[0010] S3: A finger nail side detection model is established and trained based on the YOLOv8 model to detect the key points of the nails; a finger joint point detection model is established and trained based on the MMPose model to detect the key points of the fingers;

[0011] S4: A nail art sticker generation model is established based on the StyleGAN model and trained to generate nail art stickers;

[0012] S5: Use the trained nail segmentation model and finger nail side detection model to perform nail measurement, including nail length, width and curvature measurement;

[0013] S6: Rendering the nail according to the obtained length, width and curvature of the nail;

[0014] S7: The user inputs the rendered nail photo into the trained nail map generation model, and performs nail segmentation, finger joint point detection, length and width measurement, arc measurement and nail map generation operations according to the information in the nail photo;

[0015] S8: Feedback the nail art renderings obtained after processing the nail art mapping generation model to the user, including the nail art renderings and nail size data; the user selects different nail art plans based on the nail art renderings and makes adjustments; the user checks the nail size data and selects appropriate nail art products and services.

[0016] In a preferred embodiment, in step S1, the nail segmentation dataset obtains hand images from the Internet or public datasets, including Finger dataset, Hand dataset; uses a mobile phone or camera to take real hand images, including nails of different shapes, colors and postures; uses 3D modeling software to generate a hand model and renders the nail image;

[0017] Use the image segmentation tool LabelImg or MakeSense to label the nail area and generate a pixel-level mask image.

[0018] In a preferred embodiment, in step S1, during the collection of the nail key point dataset, a hand image containing finger key points is obtained from the Internet or a public dataset, including an MPII Hand Pose dataset, a Handkeypoints dataset, and the positions of the finger key points are marked, including fingertips, nail center points, and joint points.

[0019] In a preferred embodiment, in step S2, the specific steps of training the nail segmentation model include:

[0020] S2.1: preprocess the data in the nail segmentation dataset collected and annotated in step S1, scale and normalize the images, and use them as input for the nail segmentation model;

[0021] The nail segmentation model includes:

[0022] Encoder: Use ResNet, Xception or MobileNet backbone network to extract nail features;

[0023] Decoder: Use Atrous Spatial Pyramid Pooling and the decoder network for feature fusion and upsampling to generate a pixel-level nail segmentation map;

[0024] S2.2: Use the cross entropy loss function to evaluate the performance of the nail segmentation model and calculate the difference between the predicted nail segmentation map and the true segmentation map. The calculation formula of the cross entropy loss function is:

[0025] L=-Σ(y0*log(p1)+(1-y0)*log(1-p1));

[0026] Among them, L is the loss function, y0 is the true label, and p1 is the predicted probability;

[0027] S2.3: The training process of the nail segmentation model is as follows:

[0028] Use the preprocessed nail segmentation dataset for training and set the training nail segmentation model parameters;

[0029] Use the gradient descent algorithm to update the nail segmentation model parameters to minimize the cross entropy loss function in step S2.2;

[0030] Use the validation set to evaluate the generalization ability of the nail segmentation model and obtain the nail segmentation model with the best performance;

[0031] S2.4: Use evaluation indicators to evaluate the segmentation performance of the nail segmentation model. The evaluation indicators include accuracy, recall rate, and F1 value.

[0032] In a preferred embodiment, in step S3, the training process of the finger nail side detection model includes:

[0033] S3.1: pre-processing the data in the data set of nail key points collected and annotated in step S1, scaling and normalizing the images as input to the fingernail side detection model;

[0034] The fingernail side detection model consists of the following parts:

[0035] Backbone: Use CSPDarknet and EfficientNet backbone networks to extract features on the side of fingernails;

[0036] Neck: Use PANet or YOLOv5Neck for feature fusion and upsampling;

[0037] Head: used to predict the position, category and key point position of the nail target box;

[0038] S3.2: Evaluate model performance using a multi-task loss function consisting of position loss, category loss, and keypoint loss.

[0039] The calculation formula of the position loss function is:

[0040] L_loc=Σ(x1-x1') 2 +(y1-y1') 2

[0041] Among them, L_loc is the position loss, (x1, y1) is the coordinate of the center point of the real target box, and (x1', y1') is the coordinate of the center point of the predicted target box;

[0042] The calculation formula of the category loss function is:

[0043] L_cls=-Σ(y2*log(p2)+(1-y2)*log(1-p2))

[0044] Among them, L_cls is the category loss, y2 is the true category label, and p2 is the predicted category probability;

[0045] The calculation formula of the key point loss function is:

[0046] L_kp=Σ(x3-x3') 2 +(y3-y3') 2

[0047] Among them, L_kp is the key point loss, (x3, y3) is the real key point coordinate, (x3', y3') is the predicted key point coordinate;

[0048] S3.3: Use the preprocessed nail key point data set to train the finger nail side detection model and set the training parameters of the finger nail side detection model;

[0049] Use the gradient descent algorithm to update the parameters of the fingernail side detection model to minimize the multi-task loss function in step S3.2;

[0050] Use the validation set to evaluate the generalization ability of the fingernail side detection model and perform model selection;

[0051] S3.4: Use the mAP metric to evaluate the object detection performance of the model.

[0052] In a preferred embodiment, in step S4, the training process of the nail art image generation model includes:

[0053] S4.1: In step S1, the data in the nail art image dataset is collected and preprocessed, and the images are scaled and normalized as inputs of the nail art image generation model;

[0054] The nail art sticker generation model consists of a generator and a discriminator: the generator uses multi-layer convolution and deconvolution networks to convert random noise or latent vectors into nail art stickers; the discriminator is used to determine whether the nail art stickers generated by the generator are real;

[0055] S4.2: Evaluate model performance using a multi-task loss function consisting of adversarial loss, content loss, and style loss.

[0056] The calculation formula of the adversarial loss function is:

[0057] L_adv=-E_{x~p_data}[log(D(x4))]-E_{z~p_z}[log(1-D(G(z1)))]

[0058] Among them, L_adv is the adversarial loss, D is the discriminator, G is the generator, x4 is the real picture, z1 is the random noise; x~p_data means that x is a sample drawn from the real data distribution p_data, z~p_z means that z is a sample drawn from the prior noise distribution p_z, and E_{} represents the expected value, which is used to measure the average or expected value of a random variable;

[0059] The calculation formula of the content loss function is:

[0060] L_content=||F(x4)-F(G(z1))||

[0061] Among them, L_content is the content loss, F is the feature extraction network, x4 is the real picture, and z1 is random noise;

[0062] The calculation formula of the style loss function is:

[0063] L_style=Σ||G_style(z1)-F_style(x4)||

[0064] Among them, L_style is the style loss, G_style and F_style are the style layers of the generator and discriminator respectively;

[0065] S4.3: Use the preprocessed nail art image dataset to train the nail art image generation model, and set the training parameters of the nail art image generation model, including learning rate and batch size;

[0066] Use the gradient descent algorithm to update the parameters of the nail art image generation model to minimize the multi-task loss function in step S4.2;

[0067] The validation set is used to evaluate the generalization ability of the nail art sticker generation model and obtain the nail art sticker generation model with the best performance.

[0068] In a preferred embodiment, in step S5, the length and width measurement specifically comprises:

[0069] ① Segmentation of nails and calibration objects: Use the DeepLabV3+ model to segment nails and calibration objects;

[0070] ② Get the minimum enclosing rectangle: Get the minimum enclosing rectangle of the nail and the calibration object;

[0071] ③ Calculate length and width: Calculate the actual length and width of the nail based on the size and pixel density of the calibration object; use a scale or reference object for calibration; the calculation formula is: actual nail size = nail pixel size / pixel density.

[0072] In a preferred embodiment, in step S5, the arc measurement includes:

[0073] ① Detect nail key points: Use the trained finger nail side detection model to detect the position of nail key points;

[0074] ② Mark five points: Mark five points on the nail contour, excluding the two end points;

[0075] ③Calculate radians:

[0076] By calibrating five points on the nail contour, taking three adjacent points, fitting the equation of the circle through these three points, and then calculating the coordinates of the center of the circle, the calculation formula is: x5 2 +y5 2 +Dx5+Ey5+F=0; where D, E, and F are the parameters of the circle, calculated from the coordinates of three points on the circle, and (x5, y5) represents a point on the circle;

[0077] The center angle of the circle is calculated using the coordinates of the center of the circle and the three points. The angle between the two adjacent points on the circle and the center of the circle is converted into radians. The calculation formula is: radian (rad) = angle (deg) * π / 180. The coordinates of the key points of each frame are calculated, and the radians are calculated using the coordinates of the center of the circle and the angle. The average value of the radians is taken as the radian of the nail.

[0078] In a preferred embodiment, in step S6, the nail plate rendering specifically includes:

[0079] S6.1: Segment nail region: Use the trained nail segmentation model to segment the nail region;

[0080] S6.2: Obtain bounding box direction: Use the trained finger nail side detection model to detect finger key points and obtain the nail bounding box direction;

[0081] S6.3: Texture rendering: The nail art texture is attached to the hand image according to the direction of the bounding box to achieve the nail rendering effect;

[0082] S6.4: Perform perspective transformation on the nail plate image, specifically including:

[0083] S6.4.1: Obtain the width and height of the nail image and define four coordinate points of the nail image;

[0084] S6.4.2: Calculate the perspective transformation matrix and perform perspective transformation on the nail plate image;

[0085] S6.4.3: Fit the transformed nail plate image to the hand image and display the resulting image;

[0086] The perspective transformation formula is:

[0087] x0'=(a*x6+b*y6+c) / (g*x6+h*y6+i),

[0088] Among them, a, b, c, g, h, i are the parameters of the perspective transformation matrix, which are calculated by the coordinates of the finger key points and the nail plate image. (x6, y6) represents the original coordinate point on the nail plate image, and x0' represents the new x coordinate of the corresponding point on the nail plate image after the perspective transformation.

[0089] A nail measurement and beautification device based on deep learning, comprising: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the nail measurement and beautification method based on deep learning.

[0090] Compared with the prior art, the present invention has the following advantages: data enhancement: the diversity of the data set is expanded to better adapt to different scenarios and samples; model lightweight: the model size is reduced to enable it to be deployed on mobile devices or embedded devices; multi-task learning: multiple tasks are integrated into a system framework, including nail segmentation, key point detection and nail map generation at the same time. The details are as follows:

[0091] 1. In the present invention, the DeepLabV3+ model can be used through ASPP and decoder network to effectively segment the nail area, and even if the nail shape is irregular or there is occlusion, accurate segmentation results can be obtained. The YOLOv8 model can be used to quickly and accurately detect the position of key points of the nail, such as fingertips, nail center points, etc., to provide an accurate reference for subsequent measurement and beautification. The MMPose model can be used to identify the posture and orientation of the fingers, provide key information for nail rendering, and ensure that the nail map can be accurately attached to the hand image.

[0092] 2. In the present invention, by segmenting the nail and the calibration object and calculating the minimum circumscribed rectangle, the actual length and width of the nail can be accurately measured without the need for manual measurement by the user. By detecting the key points of the nail and calculating the angle, the curvature of the nail can be accurately measured, providing a reference for nail art design. The StyleGAN model can generate a variety of nail art maps to meet the personalized needs of users, and can be generated and adjusted in real time according to the user's choice.

[0093] 3. The present invention can generate a nail art rendering in real time, allowing users to preview the effects of different nail art schemes and select a satisfactory nail art scheme. Users can adjust the position, size and direction of the nail art map according to the nail art rendering to achieve personalized nail art design. Users can superimpose the nail art rendering with the real hand image to achieve a virtual try-on effect and feel the nail art effect more intuitively. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0096] A method for nail measurement and beautification based on deep learning, the flow chart is as follows Figure 1 As shown, the method comprises the following steps:

[0097] S1: Collect nail segmentation datasets, including nail images of multiple fingers, different shapes and colors, and annotate them using keypoints annotation tools;

[0098] Collect a data set of nail key points, including fingertips, nail center points and other key point locations, and annotate them using the keypoints annotation tool;

[0099] Collect a diverse dataset of nail art stickers, such as nail art images of different colors, patterns, and styles;

[0100] In the present invention, keypoints annotation tools are used for annotation;

[0101] S2: A nail segmentation model is established and trained based on the DeepLabV3+ model. The DeepLabV3+ model includes ASPP and a decoder network for segmenting nails. Accurate segmentation results can be obtained even if the nails are irregularly shaped or occluded.

[0102] S3: A finger nail side (cross-section) detection model is established and trained based on the YOLOv8 model to detect the key points of the nails; a finger joint point detection model is established and trained based on the MMPose model to detect the key points of the fingers;

[0103] S4: A nail art sticker generation model is established based on the StyleGAN model and trained to generate nail art stickers;

[0104] S5: Use the trained nail segmentation model and finger nail side detection model to perform nail measurement, including nail length, width and curvature measurement;

[0105] S6: Rendering the nail according to the obtained length, width and curvature of the nail;

[0106] S7: The user inputs the rendered nail art photo into the trained nail art map generation model, and performs nail segmentation, key point detection, length and width measurement, arc measurement and nail art map generation operations according to the information in the nail art photo;

[0107] S8: Feedback the nail art renderings obtained after processing the nail art mapping generation model to the user, including the nail art renderings and nail size data; the user selects different nail art plans based on the nail art renderings and makes adjustments; the user checks the nail size data and selects appropriate nail art products and services.

[0108] In step S1, the nail segmentation dataset obtains hand images from the Internet or public datasets, including Fingerdataset, Hand dataset; uses a mobile phone or camera to take real hand images, including nails of different shapes, colors and postures; uses 3D modeling software to generate a hand model and render the nail image;

[0109] Use the image segmentation tool LabelImg or MakeSense to label the nail area and generate a pixel-level mask image. During the labeling process, pay attention to distinguish between the nails and surrounding skin, fingers and other areas.

[0110] In the process of collecting the nail key point dataset, we obtain hand images containing finger key points from the Internet or public datasets, including MPII Hand Pose dataset, Hand keypoints dataset, and annotate the positions of finger key points, including fingertips, nail center points, and joints. During the annotation process, we pay attention to the accuracy and consistency of the key points.

[0111] In step S2, the specific process of nail segmentation model training includes:

[0112] S2.1: preprocessing the data in the nail segmentation dataset collected and annotated in step S1, scaling and normalizing the images to make them meet the input requirements of the nail segmentation model;

[0113] The nail segmentation model consists of the following two parts:

[0114] Encoder: Extract features using ResNet, Xception or MobileNet backbone networks;

[0115] Decoder: Use Atrous Spatial Pyramid Pooling (ASPP) and the decoder network for feature fusion and upsampling to generate a pixel-level segmentation map;

[0116] S2.2: Use the cross entropy loss function to evaluate the performance of the nail segmentation model and calculate the difference between the predicted segmentation map and the true segmentation map. The calculation formula of the cross entropy loss function is:

[0117] L=-Σ(y0*log(p1)+(1-y0)*log(1-p1));

[0118] Among them, L is the loss function, y0 is the true label, and p1 is the predicted probability;

[0119] S2.3: The training process of the nail segmentation model is as follows:

[0120] Use the preprocessed nail segmentation dataset for training and set appropriate parameters for training the nail segmentation model;

[0121] Use the gradient descent algorithm to update the nail segmentation model parameters to minimize the cross entropy loss function in step S2.2;

[0122] Use the validation set to evaluate the generalization ability of the nail segmentation model and perform model selection; model selection means that multiple model candidates may be generated during the training process (for example, through different initializations, hyperparameter settings, or training strategies), and the validation set is needed to evaluate the generalization ability of these models and select the model with the best performance. Even when only one model is trained, multiple checkpoints may be generated through different training stages. These checkpoints represent the state of the model at different training stages, and they also need to be selected through the validation set to determine which stage has the best model performance.

[0123] S2.4: Use evaluation indicators to evaluate the segmentation performance of the nail segmentation model. The evaluation indicators include accuracy, recall rate, and F1 value.

[0124] In step S3, the training process of the YOLOv8 fingernail side detection model includes:

[0125] S3.1: preprocessing the data in the data set of nail key points collected and annotated in step S1, scaling and normalizing the images to meet the input requirements of the model;

[0126] The YOLOv8 model is an end-to-end target detection model that consists of the following parts:

[0127] Backbone: Use CSPDarknet and EfficientNet backbone networks to extract features;

[0128] Neck: Use PANet or YOLOv5Neck for feature fusion and upsampling;

[0129] Head: used to predict the location, category and key point location of the target box;

[0130] S3.2: Evaluate model performance using a multi-task loss function consisting of position loss, category loss, and keypoint loss.

[0131] The calculation formula of the position loss function is:

[0132] L_loc=Σ(x1-x1') 2 +(y1-y1') 2

[0133] Among them, L_loc is the position loss, (x1, y1) is the coordinate of the center point of the real target box, and (x1', y1') is the coordinate of the center point of the predicted target box;

[0134] The calculation formula of the category loss function is:

[0135] L_cls=-Σ(y2*log(p2)+(1-y2)*log(1-p2))

[0136] Among them, L_cls is the category loss, y2 is the true category label, and p2 is the predicted category probability;

[0137] The calculation formula of the key point loss function is:

[0138] L_kp=Σ(x3-x3') 2 +(y3-y3') 2

[0139] Among them, L_kp is the key point loss, (x3, y3) is the real key point coordinate, (x3', y3') is the predicted key point coordinate;

[0140] S3.3: Use the preprocessed nail key point dataset to train the finger nail side detection model, and set appropriate training parameters of the finger nail side detection model, such as learning rate, batch size, etc.;

[0141] Use the gradient descent algorithm to update the parameters of the fingernail side detection model to minimize the multi-task loss function in step S3.2;

[0142] Use the validation set to evaluate the generalization ability of the fingernail side detection model and perform model selection;

[0143] S3.4: Use the mAP metric to evaluate the object detection performance of the model.

[0144] In step S4, the training of the nail art image generation model using StyleGAN specifically includes:

[0145] S4.1: In step S1, the data in the nail art texture dataset is collected for preprocessing, and the images are scaled, normalized, and other operations are performed to make them meet the input requirements of the nail art texture generation model.

[0146] The nail art sticker generation model consists of a generator and a discriminator: the generator uses multi-layer convolution and deconvolution networks to convert random noise or latent vectors into nail art stickers; the discriminator is used to determine whether the nail art stickers generated by the generator are real;

[0147] S4.2: Evaluate model performance using a multi-task loss function consisting of adversarial loss, content loss, and style loss.

[0148] The calculation formula of the adversarial loss function is:

[0149] L_adv=-E_{x~p_data}[log(D(x4))]-E_{z~p_z}[log(1-D(G(z1)))]

[0150] Among them, L_adv is the adversarial loss, D is the discriminator, G is the generator, x4 is the real picture, z1 is random noise; x~p_data means that x is a sample drawn from the real data distribution p_data, z~p_z means that z is a sample drawn from the prior noise distribution p_z, E_{} represents the expected value, which is used to measure the average or expected value of a random variable; similarly, x is a real picture, which comes from the real data set that you want the model to learn; in GANs, z is a random vector used as the input of the generator G to generate synthetic pictures.

[0151] The calculation formula of the content loss function is:

[0152] L_content=||F(x4)-F(G(z1))||

[0153] Among them, L_content is the content loss, F is the feature extraction network, x4 is the real picture, and z1 is random noise;

[0154] The calculation formula of the style loss function is:

[0155] L_style=Σ||G_style(z1)-F_style(x4)||

[0156] Among them, L_style is the style loss, G_style and F_style are the style layers of the generator and discriminator respectively;

[0157] S4.3: Use the preprocessed nail art image dataset to train the nail art image generation model, and set appropriate training parameters of the nail art image generation model, such as learning rate and batch size;

[0158] Use the gradient descent algorithm to update the parameters of the nail art image generation model to minimize the multi-task loss function in step S4.2;

[0159] Use the validation set to evaluate the generalization ability of the nail art image generation model and perform model selection.

[0160] In step S5, the length and width measurement specifically includes:

[0161] ① Segmentation of nails and calibration objects: Use the trained DeepLabV3+ model to segment nails and calibration objects (such as ID cards).

[0162] ② Get the minimum enclosing rectangle: Get the minimum enclosing rectangle of the nail and the calibration object.

[0163] ③ Calculate length and width: Calculate the actual length and width of the nail based on the size and pixel density of the calibration object. Use a scale or reference object for calibration. The calculation formula is: Actual nail size = nail pixel size / pixel density

[0164] In step S5, the arc measurement includes:

[0165] ① Detect nail key points: Use the trained YOLOv8 model to detect the position of nail key points.

[0166] ② Mark five points: Mark five points on the nail outline (excluding the two end points).

[0167] ③Calculate radians:

[0168] By calibrating five points on the nail contour, taking three adjacent points, fitting the equation of the circle through these three points, and then calculating the coordinates of the center of the circle, the calculation formula is: x5 2 +y5 2 +Dx5+Ey5+F=0; (x5, y5) represents a point on the circle; D, E, F are the parameters of the circle, which are calculated by the coordinates of the three points on the circle. The calculation process is as follows:

[0169] Suppose the coordinates of the three points are (x 11 ,y 11 ),(x 22 ,y 22 ), and (x 33 ,y 33 ); To fit a circle from these three points, we need to solve the following system of equations:

[0170] (x 11 2 +y 11 2 )+Dx 11 +Ey 11 +F=0;

[0171] (x 22 2 +y 22 2 )+Dx 22 +Ey 22 +F=0;

[0172] (x 33 2 +y 33 2 )+Dx 33 +Ey 33 +F=0

[0173] The above system of equations is a linear system of equations with three unknowns (D, E, F); convert the system of equations into a matrix form of Ax = b, where:

[0174] A=[x 11 2 +y 11 2 *x 11 *y 11 ][x 22 2 +y 22 2 *x 22 *y 22 ][x 33 2 +y 33 2 *x 33 *y 33 ]

[0175] x=[D][E][F]

[0176] b=[-(x 11 2 +y 11 2 )][-(x 22 2 +y 22 2 )][-(x 33 2 +y 33 2 )]

[0177] Solving the system of equations gives x, which is:

[0178] x=A -1 *b;

[0179] If the matrix A is invertible, then A -1 If it exists, D, E, and F can be calculated;

[0180] Using the coordinates of the key points of each frame, the center angle of the circle is calculated through the coordinates of the center of the circle and the three points. The angle between the two adjacent points on the circle and the center of the circle is converted into radians. The calculation formula is: radians (rad) = angle (deg) * π / 180. The average value of the radians is taken as the radian of the nail.

[0181] In step S6, nail plate rendering specifically includes:

[0182] S6.1: Segment the nail area: Use the trained nail segmentation model, that is, the trained DeepLabV3+ model to segment the nail area.

[0183] S6.2: Get the bounding box direction: Use the trained finger joint point detection model, that is, the trained MMPose model to detect the finger key points and get the nail bounding box direction. The MMPose model is a model for human posture estimation and is used to detect the key points of the fingers.

[0184] S6.3: Texture rendering: The nail art texture is attached to the hand image according to the direction of the bounding box to achieve the nail rendering effect.

[0185] S6.4: Perform perspective transformation on the nail plate image, specifically including:

[0186] S6.4.1: Obtain the width and height of the nail image and define the four coordinate points of the nail image.

[0187] S6.4.2: Calculate the perspective transformation matrix and perform perspective transformation on the nail plate image.

[0188] S6.4.3: Fit the transformed nail plate image to the hand image and display the resulting image.

[0189] The perspective transformation formula is:

[0190] x0'=(a*x+b*y+c) / (g*x+h*y+i),

[0191] Among them, a, b, c, g, h, i are the parameters of the perspective transformation matrix, which can be calculated by the coordinates of the finger key points and the nail plate image. (x6, y6) represents the original coordinate point on the nail plate image, and x0' represents the new x coordinate of the corresponding point on the nail plate image after the perspective transformation. The perspective transformation here is to fit the nail plate image to the curved surface of the nail to make it look more natural. The new coordinate point after the transformation will correspond to the position of the nail on the hand image, so as to achieve the correct rendering of the nail plate. Similarly, there will be a corresponding y0' to represent the new y coordinate after the transformation.

[0192] Assuming there is a point with original coordinates (x0, y0), the new coordinates (x0', y0') after perspective transformation can be expressed by the following formula:

[0193] x0'=Ax0+By0+C;

[0194] y0'=Dx0+Ey0+F

[0195] Here A, B, C, D, E, F are the parameters of the perspective transformation. In this formula, to calculate y0', we will use:

[0196] y0'=Dx0+Ey0+F

[0197] This formula shows that the new y-coordinate y0' is calculated by a linear combination of the original x-coordinate and y-coordinate plus a translation term F.

[0198] The optimization and improvement of various deep learning models (specifically DeepLabV3+ model, YOLOv8 model, MMPose model and StyleGAN model) include the following aspects:

[0199] 1. Data enhancement: Expand the diversity of the data set and improve the generalization ability of the model so that it can better adapt to different scenarios and samples.

[0200] Specifically include:

[0201] Rotate: Randomly rotate the image to simulate different shooting angles and poses.

[0202] Zoom: Randomly zoom in and out of the image to simulate different distances and sizes.

[0203] Cropping: Randomly crop the image to simulate different fields of view.

[0204] Color transformation: Perform color transformation on images, such as adjusting brightness, contrast, saturation, etc., to simulate different lighting conditions and shooting equipment.

[0205] Noise addition: Add random noise to the image to simulate different shooting environments and equipment quality.

[0206] 2. Model lightweight:

[0207] Purpose: To reduce the model size and improve the model running efficiency so that it can be deployed on mobile devices or embedded devices.

[0208] The lightweight methods are:

[0209] Pruning: Removing redundant neurons and connections from a model, for example using structured pruning or unstructured pruning.

[0210] Quantization: Convert model parameters from floating point numbers to integers or fixed point numbers, such as using INT8 quantization or FP16 quantization.

[0211] Knowledge distillation: Use a large model to train a small model and transfer the knowledge of the large model to the small model.

[0212] 3. Multi-task learning: multiple tasks are integrated into a system framework, so that multiple tasks can be learned simultaneously on a shared feature representation, including nail segmentation, key point detection and nail map generation, which share some underlying feature extraction layers, improving the efficiency and performance of the entire system proposed in the present invention.

[0213] Specifically include:

[0214] Shared features: Use a shared feature extraction network to extract common features for multiple tasks.

[0215] Federated learning: Use a joint loss function to optimize the performance of multiple tasks simultaneously.

[0216] Attention Mechanism: Use the attention mechanism to focus the model’s attention on key areas of different tasks.

[0217] In the present invention, the DeepLabV3+ model can be used through ASPP and decoder network to effectively segment the nail area, and even if the nail shape is irregular or there is occlusion, accurate segmentation results can be obtained. The YOLOv8 model can be used to quickly and accurately detect the position of key points of the nail, such as the fingertips, the center of the nail, etc., to provide an accurate reference for subsequent measurement and beautification. The MMPose model can be used to identify the posture and orientation of the fingers, provide key information for nail rendering, and ensure that the nail map can be accurately attached to the hand image.

[0218] In the present invention, by segmenting the nail and the calibration object and calculating the minimum circumscribed rectangle, the actual length and width of the nail can be accurately measured without the need for manual measurement by the user. By detecting the key points of the nail and calculating the angle, the curvature of the nail can be accurately measured, providing a reference for nail art design. The StyleGAN model can generate a variety of nail art maps to meet the personalized needs of users, and can be generated and adjusted in real time according to the user's choice.

[0219] The present invention can generate a nail art rendering in real time, allowing users to preview the effects of different nail art schemes and select a satisfactory nail art scheme. Users can adjust the position, size and direction of the nail art map according to the nail art rendering to achieve personalized nail art design. Users can superimpose the nail art rendering with the real hand image to achieve a virtual try-on effect and feel the nail art effect more intuitively.

[0220] A nail measurement and beautification device based on deep learning, comprising: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the nail measurement and beautification method based on deep learning.

[0221] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0222] The above description enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A nail measurement and beautification method based on deep learning, characterized in that: The method comprises the following steps: S1: Collect and annotate nail segmentation datasets, including nail images of multiple fingers, different shapes and colors; Collect and annotate a dataset of nail key points, including the locations of fingertips and nail center key points; Collect a diverse dataset of nail art stickers, including nail art images of different colors, patterns, and styles; S2: A nail segmentation model is established and trained based on the DeepLabV3+ model for nail segmentation; S3: A finger nail side detection model is established and trained based on the YOLOv8 model to detect the key points of the nails; a finger joint point detection model is established and trained based on the MMPose model to detect the key points of the fingers; S4: A nail art sticker generation model is established based on the StyleGAN model and trained to generate nail art stickers; S5: Use the trained nail segmentation model and finger nail side detection model to perform nail measurement, including nail length, width and curvature measurement; S6: Rendering the nail according to the obtained length, width and curvature of the nail; S7: The user inputs the rendered nail photo into the trained nail map generation model, and performs nail segmentation, finger joint point detection, length and width measurement, arc measurement and nail map generation operations according to the information in the nail photo; S8: Feedback the nail art renderings obtained after processing the nail art mapping generation model to the user, including the nail art renderings and nail size data; the user selects different nail art plans based on the nail art renderings and makes adjustments; the user checks the nail size data and selects appropriate nail art products and services.

2. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S1, the nail segmentation dataset obtains hand images from the Internet or public datasets, including Finger dataset, Hand dataset; uses a mobile phone or camera to take real hand images, including nails of different shapes, colors and postures; uses 3D modeling software to generate a hand model and renders the nail image; Use the image segmentation tool LabelImg or MakeSense to label the nail area and generate a pixel-level mask image.

3. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S1, during the collection of the nail key point dataset, a hand image containing finger key points is obtained from the Internet or a public dataset, including the MPII Hand Pose dataset and the Hand keypoints dataset, and the positions of the finger key points are marked, including the fingertips, nail center points, and joint points.

4. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S2, the specific steps of training the nail segmentation model include: S2.1: preprocess the data in the nail segmentation dataset collected and annotated in step S1, scale and normalize the images, and use them as input for the nail segmentation model; The nail segmentation model includes: Encoder: Use ResNet, Xception or MobileNet backbone network to extract nail features; Decoder: Use Atrous Spatial Pyramid Pooling and the decoder network for feature fusion and upsampling to generate a pixel-level nail segmentation map; S2.2: Use the cross entropy loss function to evaluate the performance of the nail segmentation model and calculate the difference between the predicted nail segmentation map and the true segmentation map. The calculation formula of the cross entropy loss function is: L=-Σ(y0*log(p1)+(1-y0)*log(1-p1)); Among them, L is the loss function, y0 is the true label, and p1 is the predicted probability; S2.3: The training process of the nail segmentation model is as follows: Use the preprocessed nail segmentation dataset for training and set the training nail segmentation model parameters; Use the gradient descent algorithm to update the nail segmentation model parameters to minimize the cross entropy loss function in step S2.2; Use the validation set to evaluate the generalization ability of the nail segmentation model and obtain the nail segmentation model with the best performance; S2.4: Use evaluation indicators to evaluate the segmentation performance of the nail segmentation model. The evaluation indicators include accuracy, recall rate, and F1 value.

5. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S3, the training process of the fingernail side detection model includes: S3.1: pre-processing the data in the data set of nail key points collected and annotated in step S1, scaling and normalizing the images as input to the fingernail side detection model; The fingernail side detection model consists of the following parts: Backbone: Use CSPDarknet and EfficientNet backbone networks to extract features on the side of fingernails; Neck: Use PANet or YOLOv5Neck for feature fusion and upsampling; Head: used to predict the position, category and key point position of the nail target box; S3.2: Evaluate model performance using a multi-task loss function consisting of position loss, category loss, and keypoint loss. The calculation formula of the position loss function is: L_loc=Σ(x1-x1') 2 +(y1-y1') 2 Among them, L_loc is the position loss, (x1, y1) is the coordinate of the center point of the real target box, and (x1', y1') is the coordinate of the center point of the predicted target box; The calculation formula of the category loss function is: L_cls=-Σ(y2*log(p2)+(1-y2)*log(1-p2)) Among them, L_cls is the category loss, y2 is the true category label, and p2 is the predicted category probability; The calculation formula of the key point loss function is: L_kp=Σ(x3-x3') 2 +(y3-y3') 2 Among them, L_kp is the key point loss, (x3, y3) is the real key point coordinate, (x3', y3') is the predicted key point coordinate; S3.3: Use the preprocessed nail key point data set to train the finger nail side detection model and set the training parameters of the finger nail side detection model; Use the gradient descent algorithm to update the parameters of the fingernail side detection model to minimize the multi-task loss function in step S3.2; Use the validation set to evaluate the generalization ability of the fingernail side detection model and obtain the fingernail side detection model with the best performance; S3.4: Use the mAP metric to evaluate the object detection performance of the model.

6. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S4, the training process of the nail art image generation model includes: S4.1: In step S1, the data in the nail art image dataset is collected and preprocessed, and the images are scaled and normalized as inputs of the nail art image generation model; The nail art sticker generation model consists of a generator and a discriminator: the generator uses multi-layer convolution and deconvolution networks to convert random noise or latent vectors into nail art stickers; the discriminator is used to determine whether the nail art stickers generated by the generator are real; S4.2: Evaluate model performance using a multi-task loss function consisting of adversarial loss, content loss, and style loss. The calculation formula of the adversarial loss function is: L_adv=-E_{x~p_data}[log(D(x4))]-E_{z~p_z}[log(1-D(G(z1)))] Among them, L_adv is the adversarial loss, D is the discriminator, G is the generator, x4 is the real picture, z1 is the random noise; x~p_data means that x is a sample drawn from the real data distribution p_data, z~p_z means that z is a sample drawn from the prior noise distribution p_z, and E_{} represents the expected value, which is used to measure the average or expected value of a random variable; The calculation formula of the content loss function is: L_content=||F(x4)-F(G(z1))|| Among them, L_content is the content loss, F is the feature extraction network, x4 is the real picture, and z1 is random noise; The calculation formula of the style loss function is: L_style=Σ||G_style(z1)-F_style(x4)|| Among them, L_style is the style loss, G_style and F_style are the style layers of the generator and discriminator respectively; S4.3: Use the preprocessed nail art image dataset to train the nail art image generation model, and set the training parameters of the nail art image generation model, including learning rate and batch size; Use the gradient descent algorithm to update the parameters of the nail art image generation model to minimize the multi-task loss function in step S4.2; The validation set is used to evaluate the generalization ability of the nail art image generation model and obtain the nail art image generation model with the best performance.

7. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S5, the length and width measurement includes: ① Segmentation of nails and calibration objects: Use the trained nail segmentation model to segment nails and calibration objects; ② Get the minimum enclosing rectangle: Get the minimum enclosing rectangle of the nail and the calibration object; ③ Calculate length and width: Use a scale or reference object for calibration, and calculate the actual length and width of the nail based on the size and pixel density of the calibration object. The calculation formula is: actual nail size = nail pixel size / pixel density.

8. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S5, the arc measurement includes: ① Detect nail key points: Use the trained finger nail side detection model to detect the position of nail key points; ② Mark five points: Mark five points on the nail contour, excluding the two end points; ③Calculate radians: By calibrating five points on the nail contour, taking three adjacent points, fitting the equation of the circle through these three points, and then calculating the coordinates of the center of the circle, the calculation formula is: x5 2 +y5 2 +Dx5+Ey5+F=0; where D, E, F are the parameters of the circle, and (x5, y5) represents a point on the circle; The center angle of the circle is calculated using the coordinates of the center of the circle and the three points. The angle between the center of the circle and two adjacent points is converted into radians. The calculation formula is: radians (rad) = angle (deg) * π / 180. The average value of the radians is taken as the radian of the nail.

9. The method for nail measurement and beautification based on deep learning according to claim 1, characterized in that: In step S6, the nail piece rendering specifically includes: S6.1: Segment nail region: Use the trained nail segmentation model to segment the nail region; S6.2: Obtain bounding box direction: Use the trained finger nail side detection model to detect finger key points and obtain the nail bounding box direction; S6.3: Texture rendering: The nail art texture is attached to the hand image according to the direction of the bounding box to achieve the nail rendering effect; S6.4: Perform perspective transformation on the nail plate image, specifically including: S6.4.1: Obtain the width and height of the nail image and define four coordinate points of the nail image; S6.4.2: Calculate the perspective transformation matrix and perform perspective transformation on the nail plate image; S6.4.3: Fit the transformed nail plate image to the hand image and display the resulting image; The perspective transformation formula is: x0'=(a*x6+b*y6+c) / (g*x6+h*y6+i), Among them, a, b, c, g, h, i are the parameters of the perspective transformation matrix, which are calculated through the coordinates of the finger key points and the nail image. (x6, y6) represents the original coordinate point on the nail image, and x0' represents the new x coordinate of the corresponding point on the nail image after the perspective transformation.

10. A nail measurement and beautification device based on deep learning, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the nail measurement and beautification method based on deep learning as described in any one of claims 1 to 9.

Images