A method and system for generating plastic surgery scheme based on image recognition

Abstract

The present invention discloses a method and system for generating a plastic surgery plan based on image recognition, the method comprising: acquiring a user's facial image to construct a half-face model, generating a similar face set and recommending it to the user, generating a full-face comparison image set, screening the full-face comparison image set, determining a target case, predicting a facial image after plastic surgery, training a conversion model for converting a plane face image into a three-dimensional face model, converting the predicted facial image after plastic surgery into a three-dimensional face model after plastic surgery, and generating a plastic surgery plan for the user based on the user's facial three-dimensional data model and the target three-dimensional face model. The present invention realizes the objectification, personalization and high-precision generation of plastic surgery plans, the user can intuitively preview the plastic surgery effect, participate in the formulation of the plan, improve satisfaction and trust, and the plastic surgeon can formulate a more professional and feasible surgical plan based on detailed plans and data support, reduce surgical risks, and improve the success rate.

Description

[Technical field]

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for generating a plastic surgery plan based on image recognition. [Background technology]

[0002] With the improvement of people's living standards and the enhancement of aesthetic awareness, plastic surgery has become an important way for many people to improve their image. Traditional plastic surgery plans mainly rely on the professional experience and aesthetic judgment of plastic surgeons, and the surgery is designed according to the subjective needs of patients and the doctor's advice. However, this approach has the following problems:

[0003] It is highly subjective and lacks objective basis, which can easily lead to postoperative results not meeting the patient's expectations; it is not personalized enough and cannot fully consider the patient's facial features and individual differences; communication is insufficient and it is difficult for patients to accurately express their needs; the expected results are difficult to present, and traditional methods cannot intuitively demonstrate the expected results after plastic surgery before surgery. [Summary of the invention]

[0004] In view of this, an embodiment of the present invention provides a method and system for generating a plastic surgery plan based on image recognition.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a shaping scheme based on image recognition, the method comprising:

[0006] S1. Obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half face according to the evaluation results and build a half face model;

[0007] S2, matching similar faces from a preset face database based on the half-face model, generating a similar face set and recommending it to the user;

[0008] S3, receiving a set of desired face images selected by a user, and fusing the desired face with a half-face image of the user to generate a set of full-face comparison images;

[0009] S4, automatically screening the full-face comparison image set according to the first screening criterion, and then manually screening the full-face comparison image set according to the second screening criterion to determine the target case;

[0010] S5, using a generative adversarial network model to predict the target case and predict the facial image after plastic surgery;

[0011] S6, obtaining three-dimensional scanning data of the user's face, establishing a three-dimensional data model of the user's face, establishing and training a conversion model for converting a planar face image into a three-dimensional face model, converting the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjusting the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model;

[0012] S7. Generate a plastic surgery plan for the user based on the user's three-dimensional facial data model and the target three-dimensional face model.

[0013] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the evaluating the left and right halves of the user's face specifically includes:

[0014] Evaluate the left and right halves of the user's face and calculate their respective aesthetic evaluation scores: left half face S L and right half face S L , where the left half of the face S L and right half face S L The calculation formulas are as follows:

[0015] S=α·S s +β·S g , α+β=1,

[0016]

[0017] Among them, S represents the half-face aesthetic evaluation score, S s represents the symmetry score, S g represents the aesthetic ratio score, α and β represent weight coefficients, N represents the number of feature point pairs, P i represents the coordinates of the i-th feature point on the left half of the face, represents the coordinates of the i-th symmetrical feature point corresponding to the right half of the face, d max Indicates the maximum size of the face, used for normalization, M indicates the number of proportions involved in the calculation, D j represents the distance between the jth key facial parts, D j+1 represents the distance between the j+1th facial key parts, and θ represents the golden ratio;

[0018] Compare the left half of the face S L and right half face S L High or low, the half-face with a higher score is selected as the target half-face according to the evaluation results.

[0019] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein selecting a target half face and constructing a half face model according to the evaluation result specifically includes:

[0020] Define the model structure: organize the target half-face feature points according to the facial anatomical structure; the contour area: the feature points of the jawline and cheek edge; the eye area: the feature points of the eye corners and eyelids; the nose area: the feature points of the nose bridge and nose wings; the mouth area: the feature points of the mouth corners and lip line;

[0021] Assign a unique number to each feature point and establish a feature point index;

[0022] Create feature vector: Expand the feature point coordinates into a feature vector f = [x1, y1, x2, y2, ..., x n ,y n ], n represents the number of feature points of the target half face;

[0023] Model Standardization:

[0024] Translation processing: calculate the center point coordinates, x c ,y c Represents the average coordinates of the feature points, which is used to represent the centroid position of the model. n represents the number of feature points. x i ,y i represents the original coordinates of the i-th feature point; coordinate translation, x′ i =x i -x c , y′ i =y′ i -y c , x′, y′ i Represents the coordinates of the feature points after translation;

[0025] Scaling: Select reference length, d r represents the reference length, (x′ a , y′ a )、(x′ b , y′ b ) represents the coordinates of the two selected feature points after translation; calculate the scale factor, s represents the scale factor, which is used to scale the model to a standard size, and d s Indicates the preset standard length, which is a constant; coordinate scaling, x″ i =s·x′ i , y″ i =s·y′ i ,x″ i ,y″ i Represents the coordinates of the feature points after scaling;

[0026] Affine transformation: calculate the rotation angle, θ represents the angle that the model needs to be rotated and corrected, (x″ e , y″e ) represents the scaled coordinates of the first facial feature point, (x″ m , y″ m ) represents the scaled coordinates of the second facial feature point; construct a rotation matrix, R represents a two-dimensional rotation matrix; the rotation matrix is ​​applied to rotate the coordinates. x″′, y″′ represent the coordinates of the feature points after rotation correction;

[0027] Formation of the normalized eigenvector: f s =[x″′1,y″′1,x″′2,y″′2,...,x″′ n , y″′ n ], f s represents the feature vector after translation, scaling and rotation correction;

[0028] The feature point coordinates are translated and scaled to make the model scale and position invariant, and the facial tilt is corrected through affine transformation;

[0029] Data storage: Store the standardized feature vectors and related parameters.

[0030] According to the above aspects and any possible implementation, an implementation is further provided, wherein the method for matching similar faces is as follows:

[0031] Obtain each sample face image from a preset face database, segment it into half-face images corresponding to the target half-face, extract half-face features, and calculate the sample feature vector;

[0032] Compare the number of feature points of the sample half face with the number of feature points of the target half face, and take the samples whose difference is lower than the preset threshold as the samples to be matched;

[0033] Preprocess the standardized feature vector of the target half face and the feature vector in the sample to be matched, calculate the mean and standard deviation to eliminate the scale, and use the illumination compensation algorithm Retinex to eliminate the illumination difference;

[0034] Calculate the comprehensive distance D between the standardized feature vector of the target half face and the feature vector of each sample in the sample to be matched d , where the comprehensive distance D d The calculation formula is as follows:

[0035] D d =w1·D e +w2·D m , w1+w2=1,

[0036]

[0037] Among them, Dd Represents the comprehensive distance, which is used to calculate the comprehensive similarity between the target half-face feature and the sample to be matched, D e represents the Euclidean distance, D m represents the Manhattan distance, x i Represents the value of the first eigenvector in the i-th dimension, y i represents the value of the second eigenvector in the i-th dimension, and n represents the number of feature dimensions;

[0038] The integrated distance D d Compare with the preset similarity threshold and filter out samples whose similarity meets the requirements.

[0039] According to the above aspects and any possible implementation, an implementation is further provided, wherein the step of fusing the desired face with the half-face image of the user in S3 specifically includes:

[0040] Receive a desired face image set selected by a user, perform image segmentation, and then generate a desired half-face image set;

[0041] Align facial feature points of the target half-face image and the desired half-face image;

[0042] Construct the affine transformation matrix:

[0043]

[0044] Where A represents the affine transformation matrix, (x, y) represents the original point coordinates, (x′, y′) represents the transformed point coordinates, a and d represent the parameters for controlling scaling and rotation, b and c represent the parameters for controlling the skew of the image, and tx and ty represent the translation on the x-axis and y-axis respectively;

[0045] Using the least squares method, the parameters of the affine transformation matrix are calculated based on the corresponding feature point pairs established;

[0046] The desired half-face image is transformed using the calculated affine transformation matrix to obtain the desired half-face image aligned with the target half-face image;

[0047] The target half-face image and the transformed desired half-face image are fused into the full-face image using a progressive fusion algorithm, and the fusion weight of the target half-face and the desired half-face is set to 0.5:0.5;

[0048] Extract key feature points of the mouth and nose areas, and make the brightness and color of the mouth and nose areas consistent with the full face image through color histogram matching;

[0049] Smooth the midline of the entire face.

[0050] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the automatic screening of the full-face comparison image set according to the first screening criterion comprises:

[0051] The full face comparison image in the full face comparison image set is divided into a left half face and a right half face along the midline, one half face is mirror-flipped, and overlapped and compared with the other half face;

[0052] Calculate the symmetry score S k , where the symmetry score S k The calculation formula is as follows:

[0053]

[0054] Among them, S k It represents the symmetry score, ranging from 0 to 1. The closer the value is to 1, the better the symmetry is. N represents the number of feature points used for calculation. i left represents the coordinates of the i-th feature point on the left half of the face, P i right represents the coordinates of the i-th feature point on the right half of the face, M(P i right ) represents the mirror mapping point of the i-th feature point on the right half of the face about the facial midline, ||·|| represents the Euclidean distance, that is, the straight-line distance between two points, d f represents the normalized distance of facial features;

[0055] The symmetry score S k Compared with the preset symmetry threshold, the symmetry score S k The full-face comparison images that do not meet the requirements are deleted, otherwise they are retained;

[0056] Manually screening the full-face comparison image set according to the second screening criterion includes:

[0057] Obtaining a success rate score provided by a medical staff, wherein the success rate score ranges from 0 to 1;

[0058] Obtain the aesthetic score provided by the medical staff, ranging from 0 to 1;

[0059] The success rate score and the aesthetic score were weighted to obtain a comprehensive score;

[0060] Compare the comprehensive score with the preset score threshold, delete the full-face comparison images whose comprehensive scores do not meet the requirements, and retain the others;

[0061] Determining the target case includes: comparing the retained full-face comparison images and taking the comprehensive score with the highest comprehensive score as the target case.

[0062] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein S5 specifically includes:

[0063] Collecting and processing to generate a facial image dataset, wherein the dataset includes full-face comparison images and reshaped face images;

[0064] Clean the data to remove blurry, occluded or poor quality images;

[0065] Label the data, mark the shaping area and feature points;

[0066] Construct a conditional generative adversarial network cGAN combined with a U-Net structure, including a generator G and a discriminator D. The generator G uses a U-Net structure, takes the input full-face comparison image before plastic surgery and feature annotation as conditional input, and uses jump connections to combine low-level features with high-level features. The discriminator D uses a PatchGAN discriminator and combines it with a VGG network to extract features.

[0067] Define the loss function:

[0068] L t (G, D) = L g (G, D)+λL r (G),

[0069]

[0070] Among them, L t (G, D) represents the total loss, L g (G, D) represents the adversarial loss, L r (G) represents the reconstruction loss, λ represents the balance parameter, x represents the input full face comparison image before plastic surgery, and y true represents the real face image after plastic surgery, G(x) represents the face image after plastic surgery generated by the generator, D(x, y) represents the discrimination result of the discriminator on the input image pair, and ||·||1 represents the L1 norm, which is used to calculate the pixel difference between the generated image and the real image;

[0071] Select Adam optimizer, set learning rate α = 0.0002, momentum parameter β1 = 0.5, batch size 1, and train for 200 rounds;

[0072] Alternately train the generator and discriminator: fix the generator, train the discriminator D, and use real images and generated images for discrimination; fix the discriminator, train the generator G, and optimize the similarity between the generated plastic images and the real plastic images; use progressive training, starting from low-resolution images and gradually increasing the resolution;

[0073] The generated images were evaluated using the structural similarity index SSIM and peak signal-to-noise ratio PSNR;

[0074] Input the target case image and feature annotation, generate the reshaped facial image through the trained generator G, and the output image is processed by denoising and smoothing;

[0075] Among them, the structures of generator G and discriminator D are as follows:

[0076] Generator G:

[0077] Encoder: 8 convolutional layers, the number of convolution kernels in each layer is 64, 128, 256, 512, 512, 512, 512, 512, the convolution kernel size is 4×4, the step size is 2, each layer is followed by Batch Normalization and Leaky ReLU activation function, and the negative slope of Leaky ReLU is set to 0.2; Decoder: 8 deconvolutional layers, the number of convolution kernels in each layer is 512, 512, 512, 512, 256, 128, 64, the number of output channels, deconvolution is used in each layer, the step size is 2, the convolution kernel size is 4×4, each layer is followed by Batch Normalization and ReLU activation function, jump connection: the same layer features of the corresponding encoder and decoder are connected, Dropout is used with a probability of 0.5 to prevent overfitting, output layer: Tanh activation function is used to map the output to the [-1, 1] interval;

[0078] Discriminator D:

[0079] There are 5 convolutional layers, the number of convolution kernels are 64, 128, 256, 512, and 1 respectively, the convolution kernel size is 4×4, the step size is 2, and each layer is followed by Batch Normalization and Leaky ReLU activation function. PatchGAN: The output is an N×N matrix, each element represents the authenticity probability of the corresponding image area, and the output layer: uses Sigmoid activation function, and the output range is [0, 1].

[0080] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein S6 specifically includes:

[0081] Use a 3D facial scanner to obtain the user's 3D facial data, perform data preprocessing, and use a triangulation algorithm to convert the point cloud data into a mesh model to generate a high-quality 3D facial model;

[0082] Collect 2D face images and corresponding 3D face models to build a training data set, and rotate, flip, and crop the 2D images to increase the data volume;

[0083] The conversion model is constructed and trained by combining the convolutional neural network (CNN) and the 3D Morphable Model.

[0084] The predicted face image after plastic surgery is converted into a three-dimensional face model after plastic surgery using the trained conversion model;

[0085] The reshaped 3D face model is adjusted according to the user's 3D face model, and the target 3D face model is output.

[0086] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein S7 specifically includes:

[0087] Mark the specific parts that need surgery based on the 3D scanning data and the target 3D face model, calculate the size change of the plastic surgery area and generate the surgery parameters;

[0088] Generate a plastic surgery report containing surgical parameters;

[0089] A three-dimensional plastic surgery effect display unit for generating a comparison between before and after plastic surgery includes a user interface and a functional module. The functional module includes a model display area, a control panel and a comparison mode. The model display area is used to display the three-dimensional model before and after plastic surgery. The control panel is used to provide control buttons for model rotation, scaling and switching. The comparison mode is used to simultaneously display the models before and after plastic surgery, use a sliding bar to compare the effects before and after plastic surgery at the same position, and superimpose two models semi-transparently to intuitively display the differences.

[0090] In a second aspect, an embodiment of the present invention provides a plastic surgery scheme generation system based on image recognition, the system comprising:

[0091] A half-face model generation module is used to obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half-face according to the evaluation results, and build a half-face model;

[0092] A matching module, used for matching similar faces from a preset face database based on the half-face model, and generating a similar face set for recommendation to the user;

[0093] A fusion module is used to receive a set of desired face images selected by a user, and fuse the desired face with a half-face image of the user to generate a set of full-face comparison images;

[0094] A screening module, configured to automatically screen the full-face comparison image set according to a first screening criterion, and then manually screen the full-face comparison image set according to a second screening criterion to determine a target case;

[0095] A prediction module, used to predict the target case using a generative adversarial network model to predict the facial image after plastic surgery;

[0096] A three-dimensional face module is used to obtain three-dimensional scanning data of the user's face, establish a three-dimensional data model of the user's face, establish and train a conversion model for converting a flat face image into a three-dimensional face model, convert the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjust the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model;

[0097] The solution generation module is used to generate a plastic surgery solution for the user based on the user's facial 3D data model and the target 3D face model.

[0098] One of the above technical solutions has the following beneficial effects:

[0099] The method of the embodiment of the present invention proposes a method and system for generating a plastic surgery plan based on image recognition. The method overcomes the shortcomings of traditional plastic surgery plan generation methods and realizes the objectivity, personalization and high-precision generation of plastic surgery plans. Users can intuitively preview the plastic surgery effects and participate in the formulation of plans, thereby improving satisfaction and trust. Plastic surgeons can formulate more professional and feasible surgical plans based on detailed plans and data support, thereby reducing surgical risks and improving success rates.

Brief Description of the Drawings

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0101] Figure 1 It is a flow chart of the shaping scheme generation method S1-S7 based on image recognition provided by an embodiment of the present invention;

[0102] Figure 2 This is a functional block diagram of a plastic surgery solution system based on image recognition provided by an embodiment of the present invention. [Specific implementation method]

[0103] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0104] Please refer to Figure 1 , which is a flow chart of a shaping scheme generation method S1-S7 based on image recognition provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0105] S1. Obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half face according to the evaluation results and build a half face model;

[0106] S2, matching similar faces from a preset face database based on the half-face model, generating a similar face set and recommending it to the user;

[0107] S3, receiving a set of desired face images selected by a user, and fusing the desired face with a half-face image of the user to generate a set of full-face comparison images;

[0108] S4, automatically screening the full-face comparison image set according to the first screening criterion, and then manually screening the full-face comparison image set according to the second screening criterion to determine the target case;

[0109] S5, using a generative adversarial network model to predict the target case and predict the facial image after plastic surgery;

[0110] S6, obtaining three-dimensional scanning data of the user's face, establishing a three-dimensional data model of the user's face, establishing and training a conversion model for converting a planar face image into a three-dimensional face model, converting the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjusting the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model;

[0111] S7. Generate a plastic surgery plan for the user based on the user's three-dimensional facial data model and the target three-dimensional face model.

[0112] The embodiment of the present invention can identify the differences between the left and right sides of a human face through accurate facial feature point extraction and symmetrical aesthetic score calculation, and select a half face that is more in line with aesthetic standards. Selecting a better half face as a benchmark can lay a foundation for later generation of more beautiful plastic surgery images that better meet user expectations. In addition, processing a half face can reduce the amount of data, reduce computational complexity, improve algorithm efficiency, and more accurately analyze and process facial features. By screening out facial images similar to the user's half face from a database, on the one hand, the user can select his or her favorite sample from a group of faces with high similarity, and on the other hand, improve the matching accuracy and success rate of plastic surgery, helping users to more easily find references to plastic surgery plans that meet personal needs. By fusing the feature images of the user's half face with the desired face to generate a full face image, the user can select the desired face image from a group of faces with high similarity. The face comparison image ensures the accuracy of subsequent screening of target cases; automatic screening combines factors such as symmetry scoring and aesthetic scoring to ensure that the screened plans meet basic aesthetic standards, while manual screening adds professional success rate and aesthetic opinions to improve the professionalism of the screening results and the feasibility of the operation; the generative adversarial network model can accurately predict the image after plastic surgery, so that the effect after plastic surgery can be accurately presented; the three-dimensional scanning and conversion model establishment will make the predicted plastic surgery image three-dimensional and accurate, so that users and doctors can carefully observe the details of the plastic surgery plan, ensure the consistency of the plastic surgery plan in multiple dimensions, and improve the preparation before surgery and the practicability of the plan; the final generated plastic surgery plan combined with the three-dimensional model provides users with detailed visual effects and specific surgical guidance, helping doctors to make accurate surgical preparations.

[0113] The embodiments of the present invention not only provide accurate and personalized display of plastic surgery effects through step-by-step image processing and three-dimensional modeling, but also achieve high-quality generation of plans through step-by-step screening and evaluation, helping users to establish clear effect expectations before surgery, while helping doctors to develop scientific surgical plans, reduce risks, and improve the predictability and consistency of effects.

[0114] In a preferred embodiment of the present invention, the evaluating of the left and right halves of the user's face specifically includes:

[0115] Evaluate the left and right halves of the user's face and calculate their respective aesthetic evaluation scores. L and right half face S L , where the left half of the face S L and right half face S L The calculation formulas are as follows:

[0116] S=α·S s +β·S g , α+β=1,

[0117]

[0118] Among them, S represents the half-face aesthetic evaluation score, Ss represents the symmetry score, S g represents the aesthetic ratio score, α and β represent weight coefficients, N represents the number of feature point pairs, P i represents the coordinates of the i-th feature point on the left half of the face, represents the coordinates of the i-th symmetrical feature point corresponding to the right half of the face, d max Indicates the maximum size of the face, used for normalization, M indicates the number of proportions involved in the calculation, D j represents the distance between the jth key facial parts, D j+1 represents the distance between the j+1th facial key parts, and θ represents the golden ratio;

[0119] Compare the left half of the face S L and right half face S L High or low, the half-face with a higher score is selected as the target half-face according to the evaluation results.

[0120] The embodiment of the present invention objectifies the difference between the left and right halves of the face through numerical analysis to select the half-face with a higher score, so that the generated plastic surgery plan conforms to the user's natural symmetrical aesthetics. The symmetry and proportion scores obtained through quantification can more accurately reflect the user's individual facial features, making the constructed half-face model more accurate and more customized to the user's own facial features, thereby improving the scientificity and accuracy of subsequent plastic surgery plans.

[0121] In a preferred embodiment of the present invention, selecting a target half face and constructing a half face model according to the evaluation result specifically includes:

[0122] Define the model structure: organize the target half-face feature points according to the facial anatomical structure; the contour area: the feature points of the jawline and cheek edge; the eye area: the feature points of the eye corners and eyelids; the nose area: the feature points of the nose bridge and nose wings; the mouth area: the feature points of the mouth corners and lip line;

[0123] Assign a unique number to each feature point and establish a feature point index;

[0124] Create feature vector: Expand the feature point coordinates into a feature vector f = [x1, y1, x2, y2, ..., x n ,y n ], n represents the number of feature points of the target half face;

[0125] Model Standardization:

[0126] Translation processing: calculate the center point coordinates, x c ,y c Represents the average coordinates of the feature points, which is used to represent the centroid position of the model. n represents the number of feature points. x i ,y irepresents the original coordinates of the i-th feature point; coordinate translation, x′ i =x i -x c , y′ i =y′ i -y c , x′, y′ i Represents the coordinates of the feature points after translation;

[0127] Scaling: Select reference length, d r represents the reference length, (x′ a , y′ a )、(x′ b , y′ b ) represents the coordinates of the two selected feature points after translation; calculate the scale factor, s represents the scale factor, which is used to scale the model to a standard size, and d s Indicates the preset standard length, which is a constant; coordinate scaling, x″ i =s·x′ i , y″ i =s·y′ i ,x″ i ,y″ i Represents the coordinates of the feature points after scaling;

[0128] Affine transformation: calculate the rotation angle, θ represents the angle that the model needs to be rotated and corrected, (x″ e , y″ e ) represents the scaled coordinates of the first facial feature point, (x″ m , y″ m ) represents the scaled coordinates of the second facial feature point; construct a rotation matrix, R represents a two-dimensional rotation matrix; the rotation matrix is ​​applied to rotate the coordinates. x″′, y″′ represent the coordinates of the feature points after rotation correction;

[0129] Formation of the normalized eigenvector: f s =[x″′1,y″′1,x″′2,y″′2,...,x″′ n , y″′ n ], f s represents the feature vector after translation, scaling and rotation correction;

[0130] The feature point coordinates are translated and scaled to make the model scale and position invariant, and the facial tilt is corrected through affine transformation;

[0131] Data storage: Store the standardized feature vectors and related parameters.

[0132] The embodiment of the present invention ensures the uniformity, scale and position consistency of the facial model through precise geometric correction, and the standardized processing steps make the half-face model more stable and adaptable in facial data processing, reduce the error in the comparison and processing of different facial features, and improve the accuracy of model matching and the scientific nature of plastic surgery plan generation.

[0133] In a preferred embodiment of the present invention, the method for matching similar faces is as follows:

[0134] Obtain each sample face image from a preset face database, segment it into half-face images corresponding to the target half-face, extract half-face features, and calculate the sample feature vector;

[0135] Compare the number of feature points of the sample half face with the number of feature points of the target half face, and take the samples whose difference is lower than the preset threshold as the samples to be matched;

[0136] Preprocess the standardized feature vector of the target half face and the feature vector in the sample to be matched, calculate the mean and standard deviation to eliminate the scale, and use the illumination compensation algorithm Retinex to eliminate the illumination difference;

[0137] Calculate the comprehensive distance D between the standardized feature vector of the target half face and the feature vector of each sample in the sample to be matched d , where the comprehensive distance D d The calculation formula is as follows:

[0138] D d =w1·D e +w2·D m , w1+w2=1,

[0139]

[0140] Among them, D d Represents the comprehensive distance, which is used to calculate the comprehensive similarity between the target half-face feature and the sample to be matched, D e represents the Euclidean distance, D m represents the Manhattan distance, x i Represents the value of the first eigenvector in the i-th dimension, y i represents the value of the second eigenvector in the i-th dimension, and n represents the number of feature dimensions;

[0141] The integrated distance D d Compare with the preset similarity threshold and filter out samples whose similarity meets the requirements.

[0142] The method of using the illumination compensation algorithm Retinex to eliminate illumination differences is as follows: calculate the illumination component of the input image, separate it into illumination and reflection components using a filtering method in logarithmic space, remap the reflection image after illumination compensation to the normal brightness range, and generate an output image with uniform brightness. The illumination-compensated image is used to match a set of similar faces to ensure higher matching accuracy when illumination conditions are inconsistent.

[0143] The embodiment of the present invention constructs an accurate and robust similar face screening method through the steps of feature point extraction, illumination compensation and comprehensive distance calculation. The uniformity of feature points and the quantification of similarity in the matching process are ensured through formula calculation. The advantages of Euclidean distance and Manhattan distance are combined to make the matching effect have higher accuracy and detail discrimination.

[0144] In a preferred embodiment of the present invention, the step of fusing the desired face with the half-face image of the user in S3 specifically includes:

[0145] Receive a desired face image set selected by a user, perform image segmentation, and then generate a desired half-face image set;

[0146] Align facial feature points of the target half-face image and the desired half-face image;

[0147] Construct the affine transformation matrix:

[0148]

[0149] Where A represents the affine transformation matrix, (x, y) represents the original point coordinates, (x′, y′) represents the transformed point coordinates, a and d represent the parameters for controlling scaling and rotation, b and c represent the parameters for controlling the skew of the image, and tx and ty represent the translation on the x-axis and y-axis respectively;

[0150] Using the least squares method, the parameters of the affine transformation matrix are calculated based on the corresponding feature point pairs established;

[0151] The desired half-face image is transformed using the calculated affine transformation matrix to obtain the desired half-face image aligned with the target half-face image;

[0152] The target half-face image and the transformed desired half-face image are fused into the full-face image using a progressive fusion algorithm, and the fusion weight of the target half-face and the desired half-face is set to 0.5:0.5;

[0153] Extract key feature points of the mouth and nose areas, and make the brightness and color of the mouth and nose areas consistent with the full face image through color histogram matching;

[0154] Smooth the midline of the entire face.

[0155] The embodiment of the present invention ensures the basic accuracy of fusion through image segmentation and feature point alignment, so that the desired half-face and the user's half-face image are consistent in position and proportion. Through affine transformation, the system can naturally fuse images of different angles and proportions. Progressive fusion ensures a smooth visual transition between the target half-face and the desired half-face, making the fusion area natural and without obvious boundaries, thereby improving the realism and aesthetics of the plastic surgery preview. Color matching ensures the consistency of the fused image in brightness and color. The midline smoothing further enhances the symmetry of the image, making the transition between the left and right halves of the face natural, without any abruptness caused by splicing, and the smoothed image is more visually harmonious.

[0156] Among them, the progressive fusion algorithm is used as follows: before fusion, the feature points of the user's half-face and the target half-face images are aligned, and the affine transformation matrix can be used to align the two images to a unified reference; according to the divided areas, weight matrices are set for the eyes, nose, mouth, etc. For example, the initial weight in the eye area is set to 0.3, and it increases by 0.1 each round, and the initial weight in the contour area is set to 0.1, and it increases by 0.05; for each round of image fusion, the weight matrix is ​​used to superimpose the two half-face images according to the weights to obtain a new fused image; after each round of fusion, Gaussian smoothing is performed on the fused area to eliminate the abrupt fusion boundary; through multiple rounds of iteration, the fusion weight is gradually increased until the predetermined weight threshold is reached, and finally a smooth and natural fused image is obtained.

[0157] In a preferred embodiment of the present invention, the automatic screening of the full face comparison image set according to the first screening criterion includes:

[0158] The full face comparison image in the full face comparison image set is divided into a left half face and a right half face along the midline, one half face is mirror-flipped, and overlapped and compared with the other half face;

[0159] Calculate the symmetry score S k , where the symmetry score S k The calculation formula is as follows:

[0160]

[0161] Among them, S k It represents the symmetry score, ranging from 0 to 1. The closer the value is to 1, the better the symmetry is. N represents the number of feature points used for calculation. i left represents the coordinates of the i-th feature point on the left half of the face, P i right represents the coordinates of the i-th feature point on the right half of the face, M(P i right) represents the mirror mapping point of the i-th feature point on the right half of the face about the facial midline, ||·|| represents the Euclidean distance, that is, the straight-line distance between two points, d f represents the normalized distance of facial features;

[0162] The symmetry score S k Compared with the preset symmetry threshold, the symmetry score S k The full-face comparison images that do not meet the requirements are deleted, otherwise they are retained;

[0163] Manually screening the full-face comparison image set according to the second screening criterion includes:

[0164] Obtaining a success rate score provided by a medical staff, wherein the success rate score ranges from 0 to 1;

[0165] Obtain the aesthetic score provided by the medical staff, ranging from 0 to 1;

[0166] The success rate score and the aesthetic score were weighted to obtain a comprehensive score;

[0167] Compare the comprehensive score with the preset score threshold, delete the full-face comparison images whose comprehensive scores do not meet the requirements, and retain the others;

[0168] Determining the target case includes: comparing the retained full-face comparison images and taking the comprehensive score with the highest comprehensive score as the target case.

[0169] The embodiment of the present invention realizes a scientific screening process for plastic surgery cases through automatic screening of symmetry scores and manual screening of success rates and aesthetic scores. The symmetry scores ensure the visual symmetry of the screening results, while the manual scores inject the professional judgment of experts. Ultimately, the optimal target case is determined through the comprehensive score to ensure that the recommended plastic surgery plan is both beautiful and feasible.

[0170] In a preferred embodiment of the present invention, S5 specifically includes:

[0171] Collect and process to generate a facial image dataset, which includes a full-face comparison image and a face image after plastic surgery; the full-face comparison image is a fusion comparison image of the user's half-face image before plastic surgery and the desired target half-face image, and the face image after plastic surgery is the user's actual full-face image;

[0172] Clean the data to remove blurry, occluded or poor quality images;

[0173] Label the data, mark the shaping area and feature points;

[0174] Construct a conditional generative adversarial network cGAN combined with a U-Net structure, including a generator G and a discriminator D. The generator G uses a U-Net structure, takes the input full-face comparison image before plastic surgery and feature annotation as conditional input, and uses jump connections to combine low-level features with high-level features. The discriminator D uses a PatchGAN discriminator and combines it with a VGG network to extract features.

[0175] Define the loss function:

[0176] L t (G, D) = L g (G, D)+λL r (G),

[0177]

[0178] Among them, L t (G, D) represents the total loss, L g (G, D) represents the adversarial loss, L r (G) represents the reconstruction loss, λ represents the balance parameter, x represents the input full face comparison image before plastic surgery, and y true represents the real face image after plastic surgery, G(x) represents the face image after plastic surgery generated by the generator, D(x, y) represents the discrimination result of the discriminator on the input image pair, and ||·||1 represents the L1 norm, which is used to calculate the pixel difference between the generated image and the real image;

[0179] Select Adam optimizer, set learning rate α = 0.0002, momentum parameter β1 = 0.5, batch size 1, and train for 200 rounds;

[0180] Alternately train the generator and discriminator: fix the generator, train the discriminator D, and use real images and generated images for discrimination; fix the discriminator, train the generator G, and optimize the similarity between the generated plastic images and the real plastic images; use progressive training, starting from low-resolution images and gradually increasing the resolution;

[0181] The generated images were evaluated using the structural similarity index SSIM and peak signal-to-noise ratio PSNR;

[0182] Input the target case image and feature annotation, generate the reshaped facial image through the trained generator G, and the output image is processed by denoising and smoothing;

[0183] Among them, the structures of generator G and discriminator D are as follows:

[0184] Generator G:

[0185] Encoder: 8 convolutional layers, the number of convolution kernels in each layer is 64, 128, 256, 512, 512, 512, 512, 512, the convolution kernel size is 4×4, the step size is 2, each layer is followed by Batch Normalization and Leaky ReLU activation function, and the negative slope of Leaky ReLU is set to 0.2; Decoder: 8 deconvolutional layers, the number of convolution kernels in each layer is 512, 512, 512, 512, 256, 128, 64, the number of output channels, deconvolution is used in each layer, the step size is 2, the convolution kernel size is 4×4, each layer is followed by Batch Normalization and ReLU activation function, jump connection: the same layer features of the corresponding encoder and decoder are connected, Dropout is used with a probability of 0.5 to prevent overfitting, output layer: Tanh activation function is used to map the output to the [-1, 1] interval;

[0186] Discriminator D:

[0187] There are 5 convolutional layers, the number of convolution kernels are 64, 128, 256, 512, and 1 respectively, the convolution kernel size is 4×4, the step size is 2, and each layer is followed by Batch Normalization and Leaky ReLU activation function. PatchGAN: The output is an N×N matrix, each element represents the authenticity probability of the corresponding image area, and the output layer: uses Sigmoid activation function, and the output range is [0, 1].

[0188] The cleaning and labeling of the data set in the embodiment of the present invention ensure the training quality of the GAN model, label the feature points of the reshaped area, and ensure that the generated reshaped image is more in line with the specific parts of the user's needs; the conditional input method of the cGAN model allows the generator to generate more targeted reshaped images according to the input conditions, and at the same time, the U-Net structure ensures the retention of low-level features and the detailed generation of high-level features through jump connections, so that the output image has high precision. PatchGAN and VGG networks focus on the authenticity of local areas during the discrimination process, making the image details generated by the model more realistic and natural; the adversarial loss enables the generator to generate The image is as realistic as possible to deceive the discriminator; the reconstruction loss constrains the pixel difference between the generated plastic surgery image and the target image to ensure the similarity between the generated image and the actual plastic surgery effect. The combination of the two enables the model to take into account both visual authenticity and the fit with the actual effect, thereby generating plastic surgery images that are more in line with user expectations; the progressive training strategy improves the stability and training efficiency of the model, enabling it to generate high-resolution images and adapt to complex facial structures; denoising and smoothing processing make the final generated plastic surgery image clearer and more natural in details, eliminate possible texture or noise defects, and enhance the visualization experience of plastic surgery effects.

[0189] In a preferred embodiment of the present invention, S6 specifically includes:

[0190] Use a 3D facial scanner to obtain the user's 3D facial data, perform data preprocessing, and use a triangulation algorithm to convert the point cloud data into a mesh model to generate a high-quality 3D facial model;

[0191] Collect 2D face images and corresponding 3D face models to build a training data set, and rotate, flip, and crop the 2D images to increase the data volume;

[0192] The conversion model is constructed and trained by combining the convolutional neural network (CNN) and the 3D Morphable Model.

[0193] The predicted face image after plastic surgery is converted into a three-dimensional face model after plastic surgery using the trained conversion model;

[0194] The reshaped 3D face model is adjusted according to the user's 3D face model, and the target 3D face model is output.

[0195] The embodiment of the present invention realizes the generation of personalized three-dimensional plastic surgery effects from two-dimensional images through three-dimensional facial scanning, three-dimensional mesh model construction, two-dimensional to three-dimensional conversion model training and three-dimensional face model adjustment after plastic surgery, ensuring that the generated three-dimensional model is authentic and reliable in details and overall structure, meeting the user's needs for previewing plastic surgery effects from multiple angles, and providing doctors with accurate surgical references.

[0196] Among them, the conversion model is constructed by combining the convolutional neural network CNN and the 3D Morphable Model. The two-dimensional face image is used as input and the corresponding three-dimensional face model parameters are used as output. CNN is used to extract the features of the two-dimensional image and regress the parameters of the 3DMM. The specific structure is as follows:

[0197] Input layer: input preprocessed 2D face image with a uniform size of 224×224×3;

[0198] Use the pre-trained ResNet-50 network to extract the image feature F, and map the feature F to the parameter space of 3DMM through the fully connected layer, and regress the shape parameter α, expression parameter β, texture parameter γ and posture parameter p;

[0199] 3DMM reconstruction module: The shape model is defined as: The texture model is defined as: Among them, S represents the reconstructed 3D face shape, represents the average shape model, B shape represents the shape basis matrix, α represents the shape parameter vector, B exprepresents the expression basis matrix, β represents the expression parameter vector, T represents the reconstructed texture, represents the average texture model, B tex represents the texture basis matrix, γ represents the texture parameter vector;

[0200] Projection and rendering: The three-dimensional shape S is transformed and projected to obtain the coordinates on the two-dimensional image plane, u = fRS + t, where u represents the two-dimensional vertex coordinates after projection, f represents the focal length of the camera, R represents the rotation matrix, which is determined by the posture parameter p, and t represents the translation vector; the reconstructed face is rendered using the texture T and the projection coordinates u to obtain the synthesized two-dimensional image I render ;

[0201] Define the loss function:

[0202] Pixel loss: I input Represents the input two-dimensional face image; key point loss: u i represents the coordinates of the i-th key point after projection, k i Represents the coordinates of the corresponding i-th key point in the input image; regularization loss: Total loss function: L total =λ1L pixel +λ2L kp +λ3L reg , λ1, λ2 and λ3 represent the weight parameters of the loss function;

[0203] Input data for training, use Adam optimizer, set learning rate to α = 0.0001, momentum parameters β1 = 0.9, β2 = 0.999; initialize with pre-trained ResNet-50 weights, minimize the total loss function through back propagation, monitor the loss on the validation set to prevent overfitting;

[0204] Convert the predicted facial image after plastic surgery into a 3D face model after plastic surgery. Input: predicted facial image after plastic surgery. Output: parameters of the 3D face model after plastic surgery (α pred , β pred , γ pred , p pred ), and the reconstructed 3D face model S pred ;

[0205] Adjust the reconstructed 3D face model according to the user's 3D face model:

[0206] Shape Fusion: S final =S user +w·(S pred -S user ), where Suser represents the user's 3D facial model, w represents the weight coefficient, and only the parts that need plastic surgery are fused;

[0207] Output: target 3D face model S final .

[0208] In a preferred embodiment of the present invention, the S7 specifically includes:

[0209] Mark the specific parts that need surgery based on the 3D scanning data and the target 3D face model, calculate the size change of the plastic surgery area and generate the surgery parameters;

[0210] Generate a plastic surgery report containing surgical parameters;

[0211] A three-dimensional plastic surgery effect display unit for generating a comparison between before and after plastic surgery includes a user interface and a functional module. The functional module includes a model display area, a control panel and a comparison mode. The model display area is used to display the three-dimensional model before and after plastic surgery. The control panel is used to provide control buttons for model rotation, scaling and switching. The comparison mode is used to simultaneously display the models before and after plastic surgery, use a sliding bar to compare the effects before and after plastic surgery at the same position, and superimpose two models semi-transparently to intuitively display the differences.

[0212] The embodiments of the present invention provide users with a full range of plastic surgery plan display and operation guidance through surgical site marking, surgical parameter generation, plastic surgery report preparation and three-dimensional display functions, which significantly improves the scientificity, transparency and interactivity of the plastic surgery plan. It not only provides detailed operation references for plastic surgeons, but also allows users to confirm the plastic surgery effect through a multi-angle and detailed three-dimensional comparison experience.

[0213] The embodiments of the present invention further provide device embodiments for implementing the steps and methods in the above method embodiments.

[0214] Please refer to Figure 2 , which is a functional block diagram of a shaping scheme generation system based on image recognition provided by an embodiment of the present invention. Figure 2 As shown, the system includes:

[0215] A half-face model generation module is used to obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half-face according to the evaluation results, and build a half-face model;

[0216] A matching module, used for matching similar faces from a preset face database based on the half-face model, and generating a similar face set for recommendation to the user;

[0217] A fusion module is used to receive a set of desired face images selected by a user, and fuse the desired face with a half-face image of the user to generate a set of full-face comparison images;

[0218] A screening module, configured to automatically screen the full-face comparison image set according to a first screening criterion, and then manually screen the full-face comparison image set according to a second screening criterion to determine a target case;

[0219] A prediction module, used to predict the target case using a generative adversarial network model to predict the facial image after plastic surgery;

[0220] A three-dimensional face module is used to obtain three-dimensional scanning data of the user's face, establish a three-dimensional data model of the user's face, establish and train a conversion model for converting a flat face image into a three-dimensional face model, convert the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjust the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model;

[0221] The solution generation module is used to generate a plastic surgery solution for the user based on the user's facial 3D data model and the target 3D face model.

[0222] The present invention forms a set of efficient and intelligent plastic surgery plan generation process by clearly dividing the work of each module and cooperating with each other. The system not only realizes the full process automation of plastic surgery plan from data acquisition, matching, fusion, screening to three-dimensional modeling and report generation, but also improves the visualization of plastic surgery effects and the user's decision-making participation. The cooperation of each module enables the system to provide users with accurate and personalized plastic surgery plans, increase user satisfaction and predictability of plastic surgery effects, and at the same time provide scientific operational support for plastic surgeons, thereby improving the success rate and consistency of plastic surgery.

[0223] Since each unit module in this embodiment can execute Figure 1 For the method shown in the embodiment, the part not described in detail in this embodiment can be referred to Figure 1 Related instructions.

[0224] At the hardware level, the device may include a processor, and optionally an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. Of course, the device may also include hardware required for other services.

[0225] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0226] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0227] The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0228] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0229] For the convenience of description, the above device is described as various units or modules according to their functions. Of course, when implementing the present invention, the functions of each unit or module can be implemented in the same or multiple software and / or hardware.

[0230] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0231] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0233] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0234] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0235] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0236] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0237] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0238] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0239] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0240] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0241] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A shaping scheme generation method based on image recognition, characterized in that: The method comprises: S1. Obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half face according to the evaluation results and build a half face model; S2, matching similar faces from a preset face database based on the half-face model, generating a similar face set and recommending it to the user; S3, receiving a set of desired face images selected by a user, and fusing the desired face with a half-face image of the user to generate a set of full-face comparison images; S4, automatically screening the full-face comparison image set according to the first screening criterion, and then manually screening the full-face comparison image set according to the second screening criterion to determine the target case; S5, using a generative adversarial network model to predict the target case and predict the facial image after plastic surgery; S6, obtaining three-dimensional scanning data of the user's face, establishing a three-dimensional data model of the user's face, establishing and training a conversion model for converting a planar face image into a three-dimensional face model, converting the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjusting the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model; S7, generating a plastic surgery plan for the user based on the user's three-dimensional facial data model and the target three-dimensional face model; In S3, the desired face is merged with the half-face image of the user, specifically including: Receive a desired face image set selected by a user, perform image segmentation, and then generate a desired half-face image set; Align facial feature points of the target half-face image and the desired half-face image; Construct the affine transformation matrix: Where A represents the affine transformation matrix, (x, y) represents the original point coordinates, (x′, y′) represents the transformed point coordinates, a and d represent the parameters for controlling scaling and rotation, b and c represent the parameters for controlling the skew of the image, and tx and ty represent the translation on the x-axis and y-axis respectively; Using the least squares method, the parameters of the affine transformation matrix are calculated based on the corresponding feature point pairs established; The desired half-face image is transformed using the calculated affine transformation matrix to obtain the desired half-face image aligned with the target half-face image; The target half-face image and the transformed desired half-face image are fused into the full-face image using a progressive fusion algorithm, and the fusion weight of the target half-face and the desired half-face is set to 0.5:0.5; Extract key feature points of the mouth and nose areas, and make the brightness and color of the mouth and nose areas consistent with the full face image through color histogram matching; Smooth the midline of the entire face.

2. The method according to claim 1, characterized in that The evaluation of the left and right halves of the user's face specifically includes: Evaluate the left and right halves of the user's face and calculate their respective aesthetic evaluation scores. L and right half face S L , where the left half of the face S L and right half face S L The calculation formulas are as follows: S=α·S s +β·S g ,α+β=1, Among them, S represents the half-face aesthetic evaluation score, S s represents the symmetry score, S g represents the aesthetic ratio score, α and β represent weight coefficients, N represents the number of feature point pairs, P i represents the coordinates of the i-th feature point on the left half of the face, P i represents the coordinates of the i-th symmetrical feature point corresponding to the right half of the face, d max Indicates the maximum size of the face, used for normalization, M indicates the number of proportions involved in the calculation, D j represents the distance between the jth key facial parts, D j+1 represents the distance between the j+1th facial key parts, and θ represents the golden ratio; Compare the left half of the face S L and right half face S L High or low, the half-face with a higher score is selected as the target half-face according to the evaluation results.

3. The method according to claim 2, characterized in that The step of selecting a target half face and constructing a half face model according to the evaluation result specifically includes: Define the model structure: organize the target half-face feature points according to the facial anatomical structure; the contour area: the feature points of the jawline and cheek edge; the eye area: the feature points of the eye corners and eyelids; the nose area: the feature points of the nose bridge and nose wings; the mouth area: the feature points of the mouth corners and lip line; Assign a unique number to each feature point and establish a feature point index; Create feature vector: Expand the feature point coordinates into a feature vector f = [x1, y1, x2, y2, ..., x n ,y n ], n represents the number of feature points of the target half face; Model Standardization: Translation processing: calculate the center point coordinates, x c ,y c Represents the average coordinates of the feature points, which is used to represent the centroid position of the model. n represents the number of feature points. x i ,y i represents the original coordinates of the i-th feature point; coordinate translation, x′ i =x i -x c , y′ i =y′ i -y c , x′ i , y′ i Represents the coordinates of the feature points after translation; Scaling: Select reference length, d r represents the reference length, (x′ a , y′ a )、(x′ b , y′ b ) represents the coordinates of the two selected feature points after translation; calculate the scale factor, s represents the scale factor, which is used to scale the model to a standard size, and d s Indicates the preset standard length, which is a constant; coordinate scaling, x″ i =s·x′ i , y″ i =s·y′ i ,x″ i ,y″ i Represents the coordinates of the feature points after scaling; Affine transformation: calculate the rotation angle, θ represents the angle that the model needs to be rotated and corrected, (x″ e , y″ e ) represents the scaled coordinates of the first facial feature point, (x″ m , y″ m ) represents the scaled coordinates of the second facial feature point; construct a rotation matrix, R represents a two-dimensional rotation matrix; the rotation matrix is ​​applied to rotate the coordinates. x″′ i ,y″′ i Represents the coordinates of the feature points after rotation correction; Formation of the normalized eigenvector: f s =[x″′1,y″′1,x″′2,y″′2,. . . ,x″′ n ,y″′ n ], f s represents the feature vector after translation, scaling and rotation correction; The feature point coordinates are translated and scaled to make the model scale and position invariant, and the facial tilt is corrected through affine transformation; Data storage: Store the standardized feature vectors and related parameters.

4. The method according to claim 3, characterized in that: The method for matching similar faces is as follows: Obtain each sample face image from a preset face database, segment it into half-face images corresponding to the target half-face, extract half-face features, and calculate the sample feature vector; Compare the number of feature points of the sample half face with the number of feature points of the target half face, and take the samples whose difference is lower than the preset threshold as the samples to be matched; Preprocess the standardized feature vector of the target half face and the feature vector in the sample to be matched, calculate the mean and standard deviation to eliminate the scale, and use the illumination compensation algorithm Retinex to eliminate the illumination difference; Calculate the comprehensive distance D between the standardized feature vector of the target half face and the feature vector of each sample in the sample to be matched d , where the comprehensive distance D d The calculation formula is as follows: D d =w1·D e +w2·D m ,w1+w2=1, Among them, D d Represents the comprehensive distance, which is used to calculate the comprehensive similarity between the target half-face feature and the sample to be matched, D e represents the Euclidean distance, D m represents the Manhattan distance, x i Represents the value of the first eigenvector in the i-th dimension, y i represents the value of the second eigenvector in the i-th dimension, and n represents the number of feature dimensions; The integrated distance D d Compare with the preset similarity threshold and filter out samples whose similarity meets the requirements.

5. The method according to claim 1, characterized in that The automatic screening of the full-face comparison image set according to the first screening criterion comprises: The full face comparison image in the full face comparison image set is divided into a left half face and a right half face along the midline, one half face is mirror-flipped, and overlapped and compared with the other half face; Calculate the symmetry score S k , where the symmetry score S k The calculation formula is as follows: Among them, S k It represents the symmetry score, ranging from 0 to 1. The closer the value is to 1, the better the symmetry is. N represents the number of feature points used for calculation. i left represents the coordinates of the i-th feature point on the left half of the face, P i right represents the coordinates of the i-th feature point on the right half of the face, M(P i right ) represents the mirror mapping point of the i-th feature point on the right half of the face about the facial midline, ||·|| represents the Euclidean distance, that is, the straight-line distance between two points, d f represents the normalized distance of facial features; The symmetry score S k Compared with the preset symmetry threshold, the symmetry score S k The full-face comparison images that do not meet the requirements are deleted, otherwise they are retained; Manually screening the full-face comparison image set according to the second screening criterion includes: Obtaining a success rate score provided by a medical staff, wherein the success rate score ranges from 0 to 1; Obtain the aesthetic score provided by the medical staff, ranging from 0 to 1; The success rate score and the aesthetic score were weighted to obtain a comprehensive score; Compare the comprehensive score with the preset score threshold, delete the full-face comparison images whose comprehensive scores do not meet the requirements, and retain the others; Determining the target case includes: comparing the retained full-face comparison images and taking the comprehensive score with the highest comprehensive score as the target case.

6. The method according to claim 1, characterized in that The S5 specifically includes: Collecting and processing to generate a facial image dataset, wherein the dataset includes full-face comparison images and reshaped face images; Clean the data to remove blurry, occluded or poor quality images; Label the data, mark the shaping area and feature points; Construct a conditional generative adversarial network cGAN combined with a U-Net structure, including a generator G and a discriminator D. The generator G uses a U-Net structure, takes the input full-face comparison image before plastic surgery and feature annotation as conditional input, and uses jump connections to combine low-level features with high-level features. The discriminator D uses a PatchGAN discriminator and combines it with a VGG network to extract features. Define the loss function: L t (G,D)=L g (G,D)+λL r (G), Among them, L t (G, D) represents the total loss, L g (G, D) represents the adversarial loss, L r (G) represents the reconstruction loss, λ represents the balance parameter, x represents the input full face comparison image before plastic surgery, and y true represents the real face image after plastic surgery, G(x) represents the face image after plastic surgery generated by the generator, and D(x, y true ) represents the discriminant result of the input image pair, ||·||1 represents the L1 norm, which is used to calculate the pixel difference between the generated image and the real image; Select Adam optimizer, set learning rate α = 0.0002, momentum parameter β1 = 0.5, batch size 1, and train for 200 rounds; Alternately train the generator and discriminator: fix the generator, train the discriminator D, and use real images and generated images for discrimination; fix the discriminator, train the generator G, and optimize the similarity between the generated plastic images and the real plastic images; use progressive training, starting from low-resolution images and gradually increasing the resolution; The generated images were evaluated using the structural similarity index SSIM and peak signal-to-noise ratio PSNR; Input the target case image and feature annotation, generate the reshaped facial image through the trained generator G, and the output image is processed by denoising and smoothing; Among them, the structures of generator G and discriminator D are as follows: Generator G: Encoder: 8 convolutional layers, the number of convolution kernels in each layer is 64, 128, 256, 512, 512, 512, 512, 512, the convolution kernel size is 4×4, the step size is 2, each layer is followed by Batch Normalization and Leaky ReLU activation function, and the negative slope of LeakyReLU is set to 0.2; Decoder: 8 deconvolutional layers, the number of convolution kernels in each layer is 512, 512, 512, 512, 256, 128, 64, the number of output channels, deconvolution is used in each layer, the step size is 2, the convolution kernel size is 4×4, each layer is followed by Batch Normalization and ReLU activation function, jump connection: the same layer features of the corresponding encoder and decoder are connected, Dropout is used with a probability of 0.5 to prevent overfitting, output layer: Tanh activation function is used to map the output to the [-1, 1] interval; Discriminator D: There are 5 convolutional layers, the number of convolution kernels are 64, 128, 256, 512, and 1 respectively, the convolution kernel size is 4×4, the step size is 2, and each layer is followed by Batch Normalization and Leaky ReLU activation function. PatchGAN: The output is an N×N matrix, each element represents the authenticity probability of the corresponding image area, and the output layer: uses Sigmoid activation function, and the output range is [0, 1].

7. The method according to claim 1, characterized in that The S6 specifically includes: Use a 3D facial scanner to obtain the user's 3D facial data, perform data preprocessing, and use a triangulation algorithm to convert the point cloud data into a mesh model to generate a high-quality 3D facial model; Collect 2D face images and corresponding 3D face models to build a training data set, and rotate, flip, and crop the 2D images to increase the data volume; The conversion model is constructed and trained by combining the convolutional neural network (CNN) and the 3D Morphable Model. The predicted face image after plastic surgery is converted into a three-dimensional face model after plastic surgery using the trained conversion model; The reshaped 3D face model is adjusted according to the user's 3D face model, and the target 3D face model is output.

8. The method according to claim 1, characterized in that The S7 specifically includes: Mark the specific parts that need surgery based on the 3D scanning data and the target 3D face model, calculate the size change of the plastic surgery area and generate surgical parameters; Generate a plastic surgery report containing surgical parameters; A three-dimensional plastic surgery effect display unit for generating a comparison between before and after plastic surgery includes a user interface and a functional module. The functional module includes a model display area, a control panel and a comparison mode. The model display area is used to display the three-dimensional model before and after plastic surgery. The control panel is used to provide control buttons for model rotation, scaling and switching. The comparison mode is used to simultaneously display the models before and after plastic surgery, use a sliding bar to compare the effects before and after plastic surgery at the same position, and superimpose two models semi-transparently to intuitively display the differences.

9. A plastic surgery plan generation system based on image recognition using the method of claim 1, characterized in that: include: A half-face model generation module is used to obtain the user's facial image, extract the facial feature points of the left and right halves of the face after preprocessing, evaluate the left and right halves of the user's face, select the target half-face according to the evaluation results, and build a half-face model; A matching module, used for matching similar faces from a preset face database based on the half-face model, and generating a similar face set for recommendation to the user; A fusion module is used to receive a set of desired face images selected by a user, and fuse the desired face with a half-face image of the user to generate a set of full-face comparison images; A screening module, configured to automatically screen the full-face comparison image set according to a first screening criterion, and then manually screen the full-face comparison image set according to a second screening criterion to determine a target case; A prediction module, used to predict the target case using a generative adversarial network model to predict the facial image after plastic surgery; A three-dimensional face module is used to obtain three-dimensional scanning data of the user's face, establish a three-dimensional data model of the user's face, establish and train a conversion model for converting a flat face image into a three-dimensional face model, convert the predicted face image after plastic surgery into a three-dimensional face model after plastic surgery, and adjust the three-dimensional face model after plastic surgery according to the three-dimensional face data model of the user to generate a target three-dimensional face model; The solution generation module is used to generate a plastic surgery solution for the user based on the user's facial 3D data model and the target 3D face model.

Images