A face aesthetic analysis method based on feature fusion, a terminal and a storage medium

Abstract

This invention discloses a facial aesthetic analysis method, terminal, and storage medium based on feature fusion. The method includes: acquiring a facial image to be analyzed; detecting two-dimensional facial landmarks in the facial image for aesthetic analysis; inputting the facial image into a three-dimensional reconstruction network to obtain a three-dimensional reconstructed facial model, and detecting three-dimensional facial landmarks in the facial image for aesthetic analysis; extracting two-dimensional geometric features from the two-dimensional facial landmarks, and extracting three-dimensional geometric features from the three-dimensional facial landmarks; performing feature fusion based on the extracted two-dimensional and three-dimensional geometric features, and performing aesthetic task analysis based on the fused features, outputting the analysis results. This invention, through a thorough analysis and comparison of the influence of 2D and 3D geometric features on facial beauty, proposes a 2D and 3D feature fusion method suitable for facial aesthetic index analysis, improving the accuracy of facial aesthetic index analysis.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition applications, and in particular to a facial aesthetic analysis method, terminal, and storage medium based on feature fusion. Background Technology

[0002] The desire for beauty is universal, and "beauty" is highly valued in daily life. From photo enhancement on social media and daily makeup assessments to guidance for cosmetic surgery, the establishment of an objective facial aesthetic evaluation system is indispensable. Therefore, discovering and defining a scientific and comprehensive aesthetic evaluation system is crucial. Psychological research shows that different people share similar views on beauty. Furthermore, research indicates that the concept of facial beauty is unified and can be learned by computers. This has led to the development of facial aesthetics research, which combines image processing, biometric recognition technology, machine learning methods, and deep learning methods to objectively study and define the essence of "beauty." Facial aesthetics primarily studies the geometric structure, texture, and skin color characteristics of the face. Among these, the geometric structure of the face, because it remains largely unchanged regardless of lighting, makeup, or age, can be considered a fundamental factor in analyzing its impact on facial beauty.

[0003] Current research on facial aesthetics can be divided into two categories based on data type: research based on 2D facial images and research based on 3D facial scanning models.

[0004] I. Research Based on 2D Face Images

[0005] With the rapid development of computer vision, it is easy to collect 2D face images and a large number of face image databases are now publicly available. Therefore, most existing aesthetic research is based on 2D images.

[0006] Since ancient times, people have been exploring the definition of "beauty." Early aesthetic research sought a fixed standard for facial proportions to measure beauty, proposing hypotheses such as the "golden ratio," the "three courts and five eyes" theory, the "average face" hypothesis, and the "symmetrical face" hypothesis. These methods aimed to find a definite standard to measure facial aesthetics; that is, faces whose geometric structures conformed to certain rules were considered harmonious and beautiful (e.g., faces with facial proportions conforming to the golden ratio are more beautiful, or faces with facial features arranged according to the "three courts and five eyes" principle are more beautiful). However, research indicates that there is almost no correlation between these specific proportions and facial beauty; many beautiful faces do not actually conform to these proportions. Similarly, some studies have demonstrated that the assumptions of symmetry theory are not applicable to aesthetic analysis; perfectly symmetrical faces can actually appear rigid and lacking in beauty. Therefore, the results of early aesthetic research are no longer applicable to current facial aesthetic analysis.

[0007] In recent years, inspired by pattern recognition and biometrics, several machine learning-based aesthetic analysis methods have been developed. These methods focus on analyzing the relationship between facial geometric features, texture features, and skin color features and facial aesthetics. These methods first manually extract features and then train various learning models for aesthetic analysis. Currently commonly used texture feature extraction methods include Gabor filter responses, Local Binary Patterns (LBP), and skin smoothness indices. Color features include hue, saturation, and lightness (HSV) coordinates, as well as color distribution and color symmetry. For example, color information can be extracted using R, G, and B three-channel images, and LBP can be used to extract facial texture information for aesthetic analysis. Brightness information is also introduced during the feature extraction stage to assist aesthetic analysis. However, because texture and color features change with facial makeup, ambient light, facial expressions, and natural aging, the features extracted from the same individual under different conditions often vary significantly, thus affecting the accuracy of aesthetic evaluation results.

[0008] Geometric features reflect information such as the distance between facial features, the angles between them, and the shape of the face. They are more robust to changes in environment, makeup, and age. Existing methods characterize facial features, shapes, and proportions by extracting a certain number of key landmarks and calculating geometric features such as distances and angles between these landmarks. These features have good interpretability, facilitating aesthetic analysis. For example, some studies use facial geometric features and semi-supervised learning for aesthetic analysis. This method uses 2D key landmarks to represent geometric information and trains it with a Hessian semi-supervised learning model. However, they simply consider 2D geometric features and ignore the contribution of 3D features to facial aesthetic analysis. In fact, existing methods are poor at extracting geometric features from facial regions with significant depth information, such as the eyes, nose, lips, and jaw. This makes features extracted from these regions from 2D images lack discriminative beauty and difficult to use for subsequent facial aesthetic analysis. This deficiency greatly limits the performance of aesthetic evaluation models.

[0009] In recent years, convolutional neural networks (CNNs) have been widely used in facial aesthetic analysis due to their superior ability to extract high-dimensional features. Training with deep learning methods requires a large number of training samples, but current aesthetic databases are relatively small. Simply applying deep learning methods for aesthetic analysis cannot achieve ideal results; it is usually necessary to combine them with other tasks to improve the network's generalization ability and robustness. For example, after extracting high-dimensional feature representations of facial images using feature extraction networks, combining them with age recognition and expression recognition tasks can further improve the network's prediction accuracy. Another example is using CNNs to extract features and then training the model to predict beauty scores, while simultaneously incorporating brightness to assist in refining the beauty score prediction. Furthermore, the features extracted by deep learning methods are local features and cannot represent the geometric structure of the face, lacking interpretability because deep learning models only extract features within the receptive field, ignoring global features such as facial contours and proportions. Additionally, deep networks have long training times and high hardware performance requirements, making it difficult to deploy aesthetic analysis models.

[0010] II. Research Based on 3D Models

[0011] In the medical field, cosmetic analysis typically relies on high-precision 3D facial scans. This involves creating a 3D model after a comprehensive facial scan using medical imaging equipment, and then extracting geometric features from the 3D model for aesthetic analysis. These methods aim to establish new biomorphological standards for facial attractiveness and develop appropriate treatment plans for patients requiring cosmetic surgery. For example, some studies have selected attractive volunteers, manually annotated key facial landmarks after obtaining 3D facial models, and extracted 80 geometric features based on these landmarks. Subsequently, Student's t-tests were used to select features with key aesthetic impacts, and their means, variances, medians, maximums, and minimums were statistically analyzed to establish standards for facial attractiveness. However, 3D facial scanning and model creation are costly, requiring multiple specialized devices to scan volunteers, consuming significant time and money, and making it difficult to build large-scale 3D facial databases. Therefore, researchers currently lack large-scale 3D beautiful face datasets (existing publicly available 3D face datasets, such as 3D Facial Norm (3DFN) and Bosphorus, only contain scanned face models from approximately 100 volunteers). The size of these datasets only allows for simple statistical analysis methods, making it difficult to guarantee the accuracy of conclusions, and they are insufficient for facial aesthetic analysis based on deep learning and machine learning methods. The research and development of such methods is greatly hindered by the size of existing 3D face databases, the difficulty and cost of acquiring 3D models.

[0012] In the field of computer vision, with the development of 3D face reconstruction technology, some studies are now considering using face reconstruction techniques to obtain 3D models, and then selecting key facial landmarks and extracting geometric features for facial aesthetic analysis. However, these methods simply use 3D features for aesthetic analysis, ignoring the irreplaceable and crucial role of 2D features in aesthetic analysis. This deficiency affects the results of facial aesthetic analysis and urgently needs improvement.

[0013] Therefore, existing technologies still need improvement. Summary of the Invention

[0014] The technical problem to be solved by the present invention is to provide a face aesthetic analysis method, terminal and storage medium based on feature fusion, in order to solve the technical problem of low accuracy of face aesthetic index analysis based on existing 2D images or 3D models.

[0015] The technical solution adopted by this invention to solve the technical problem is as follows:

[0016] In a first aspect, the present invention provides a facial aesthetic analysis method based on feature fusion, which includes the following steps:

[0017] Obtain the face image to be analyzed;

[0018] Detect two-dimensional facial landmarks in the facial image for aesthetic analysis;

[0019] The face image is input into a 3D reconstruction network to obtain a 3D reconstructed face model, and 3D face landmarks for aesthetic analysis are detected in the face image.

[0020] Two-dimensional geometric features are extracted from the two-dimensional face landmarks, and three-dimensional geometric features are extracted from the three-dimensional face landmarks;

[0021] Feature fusion is performed based on the extracted two-dimensional and three-dimensional geometric features, and aesthetic task analysis is conducted based on the fused features, outputting the analysis results.

[0022] In one implementation, the extraction of two-dimensional geometric features from two-dimensional facial landmarks includes:

[0023] Based on the two-dimensional face landmarks, the key vector between any two two-dimensional face landmarks is calculated by iterating through them.

[0024] Iterate through and calculate the distance features between any two 2D face landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors.

[0025] In one implementation, the traversal calculation to obtain the key vector between any two two-dimensional face landmarks includes:

[0026] The key vector is obtained by traversing and calculating the vector formed by the line connecting any two two-dimensional face landmarks.

[0027] In one implementation, the extraction of two-dimensional geometric features from two-dimensional face landmarks further includes:

[0028] Based on the publicly available aesthetic dataset, the Pearson correlation coefficient between each distance feature and the facial beauty score in the two-dimensional facial landmarks is calculated, and features with an absolute value of the correlation coefficient greater than a first threshold are retained as extracted two-dimensional geometric features.

[0029] In one implementation, extracting three-dimensional geometric features from three-dimensional facial landmarks includes:

[0030] Iterate and calculate the distance features between any two 3D face landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors;

[0031] The plane formed by the endpoints of the eyebrows and the endpoint of the chin is selected as the jaw plane;

[0032] By calculating the distance between each three-dimensional facial landmark and the jaw plane, depth features representing facial depth are obtained.

[0033] In one implementation, the extraction of three-dimensional geometric features from three-dimensional face landmarks further includes:

[0034] Calculate the Pearson correlation coefficients of distance features, angle features, depth features, and facial beauty scores in the three-dimensional face landmarks, and retain features whose absolute values ​​of the correlation coefficients are greater than the second threshold as extracted three-dimensional geometric features.

[0035] In one implementation, the step of performing feature fusion based on the extracted two-dimensional and three-dimensional geometric features, and then performing aesthetic task analysis based on the fused features, includes:

[0036] The retained two-dimensional and three-dimensional geometric features are respectively fused and drawn into the face image;

[0037] Based on the facial distribution of aesthetically strongly correlated features, identify the areas on the face where the preserved two-dimensional and three-dimensional geometric features are emphasized.

[0038] Aesthetic task analysis is performed based on the geometric features of the identified key representation regions.

[0039] In one implementation, the two-dimensional geometric features of the emphasized representation region are: the distance features of the eyebrows, eyes, and nose;

[0040] The three-dimensional geometric features of the emphasized region are: the angular features of the eyes, nose, lips and the entire jawline region, and the depth features of the eyes, eyebrows, lips, nose and jawline region.

[0041] In a second aspect, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a facial aesthetic analysis program based on feature fusion, and the facial aesthetic analysis program based on feature fusion, when executed by the processor, is used to implement the facial aesthetic analysis method based on feature fusion as described in the first aspect.

[0042] Thirdly, the present invention provides a storage medium storing a face aesthetic analysis program based on feature fusion, which, when executed by a processor, is used to implement the face aesthetic analysis method based on feature fusion as described in the first aspect.

[0043] The present invention, by employing the above technical solution, has the following effects:

[0044] This invention reconstructs a 3D face model from a face image using 3D reconstruction technology. Then, 2D and 3D geometric features are extracted from different key regions of both the face image and the face model for aesthetic analysis, providing a good complementary effect. Furthermore, by traversing both 2D and 3D facial landmarks to obtain 2D and 3D geometric features, the correlation between each feature and the degree of facial beauty is calculated. Strongly correlated features are then retained, and the key regions represented by 2D and 3D features are marked in the facial landmarks. Through a thorough analysis and comparison of the influence of 2D and 3D geometric features on facial beauty, this invention proposes a 2D and 3D feature fusion method suitable for facial aesthetic index analysis, improving the accuracy of facial aesthetic index analysis. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the facial aesthetic analysis method based on feature fusion of the present invention.

[0047] Figure 2This is a block diagram of the 2D and 3D feature fusion method strategy applicable to facial aesthetic analysis according to the present invention.

[0048] Figure 3 This is a flowchart of the 2D and 3D feature emphasis area selection process for aesthetic purposes in this invention.

[0049] Figure 4 This is a schematic diagram of the facial depth information extraction method for aesthetic purposes according to the present invention.

[0050] Figure 5 This is a schematic diagram of the region extraction of 2D and 3D distance features in this invention; (left: 2D distance features; right: 3D distance features).

[0051] Figure 6 This is a schematic diagram of the regional extraction of 2D and 3D angle features in this invention (left: 2D angle features; right: 3D angle features).

[0052] Figure 7 This is a schematic diagram of the 68 facial landmarks selected in this invention.

[0053] Figure 8 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] Exemplary methods

[0057] Currently, most facial aesthetic analysis is based solely on 2D images or 3D models. These methods fail to fully consider whether there are differences in the specific impact of the two on facial aesthetics and whether they can be integrated for analysis. Therefore, there is still a technical problem of low accuracy in facial aesthetic index analysis.

[0058] To address the aforementioned technical problems, this invention provides a facial aesthetic analysis method based on feature fusion. Through thorough analysis and comparison of the influence of 2D and 3D geometric features on facial beauty, this invention proposes a 2D and 3D feature fusion method suitable for facial aesthetic analysis, thereby improving the accuracy of facial aesthetic index analysis.

[0059] like Figure 1As shown, this embodiment of the invention provides a facial aesthetic analysis method based on feature fusion, which includes the following steps:

[0060] Step S100: Obtain the face image to be analyzed.

[0061] In this embodiment, the facial aesthetic analysis method based on feature fusion is applied to a terminal, which includes, but is not limited to, devices such as computers and mobile terminals.

[0062] The facial aesthetic analysis method provided in this invention is based on facial geometric features. By acquiring the facial image to be analyzed, 2D key landmarks in the image and 3D key landmarks in the 3D facial model can be extracted respectively. Then, the distance features and angle features between the key landmarks are extracted for aesthetic analysis, and the differences between 2D features and 3D features used for facial aesthetic analysis are compared respectively.

[0063] like Figure 2 As shown, in one implementation of this embodiment, the 2D and 3D feature fusion method strategy suitable for facial aesthetic analysis includes the following steps:

[0064] Step S11: Input a face image;

[0065] Step S12: Detect 2D face landmarks for aesthetic analysis;

[0066] Step S13: Input the face image into the reconstruction network to obtain a 3D reconstructed face model;

[0067] Step S14: Detect 3D face landmarks for aesthetic analysis;

[0068] Step S15: Extract 2D geometric features and 3D geometric features from 2D landmarks and 3D landmarks respectively;

[0069] Step S16: The extracted 2D and 3D geometric features are fused for aesthetic analysis tasks.

[0070] The method of this invention indicates that the geometric features suitable for acquisition from 2D faces are mainly distance features, which are applicable to characterizing information such as the length and width of the eyebrows, eyes, and chin regions in a frontal view of a face. The geometric features suitable for acquisition from 3D face models include distance features, angle features, and depth features. These features are suitable for characterizing regions containing depth information that are difficult to obtain aesthetic information from images, such as the nose, eyes, lips, and the entire jawline.

[0071] This invention extracts 2D and 3D features from different regions to achieve mutual complementarity between features during aesthetic analysis. Tests on a publicly available standard face dataset show that the fused aesthetic features obtained by region-based feature extraction according to this invention yield slightly better aesthetic analysis results than using only 3D features, and significantly better than using only 2D features. In other words, the 2D and 3D feature fusion method proposed in this invention achieves the best aesthetic analysis results.

[0072] like Figure 1 As shown, in one implementation of the present invention, the facial aesthetic analysis method based on feature fusion further includes the following steps:

[0073] Step S200: Detect two-dimensional facial landmarks in the facial image for aesthetic analysis;

[0074] Step S300: Input the face image into a 3D reconstruction network to obtain a 3D reconstructed face model, and detect 3D face landmarks in the face image for aesthetic analysis.

[0075] In this embodiment, for the acquired face image to be analyzed, two-dimensional face landmarks for aesthetic analysis are detected in the face image using pre-calibrated coordinate points; at the same time, the face image is input into a three-dimensional reconstruction network to obtain a three-dimensional reconstructed face model, and three-dimensional face landmarks for aesthetic analysis are detected in the face image.

[0076] In this embodiment, the selection of 2D and 3D feature points is exactly the same, and a total of 68 facial key landmarks were extracted, such as... Figure 7 As shown; of course, in other embodiments, the key facial landmarks can also be adaptively set as needed; by extracting two-dimensional and three-dimensional facial landmarks, corresponding geometric features can be obtained based on these landmarks, and then aesthetic task analysis can be performed based on these geometric features to output analysis results.

[0077] like Figure 1 As shown, in one implementation of the present invention, the facial aesthetic analysis method based on feature fusion further includes the following steps:

[0078] Step S400: Extract two-dimensional geometric features from the two-dimensional face landmarks, and extract three-dimensional geometric features from the three-dimensional face landmarks.

[0079] In this embodiment, a convolutional neural network is first used to extract 2D facial landmarks from the image (the selection of 2D and 3D feature points is exactly the same, and a total of 68 facial landmarks are extracted).

[0080] In one implementation of this embodiment, step S400 specifically includes the following steps:

[0081] Step S410: Based on the two-dimensional face landmarks, calculate the key vector between any two two-dimensional face landmarks.

[0082] Specifically, the key vector between any two two-dimensional face landmarks is obtained by traversing and calculating, including: step S411, traversing and calculating the vector formed by the line connecting any two two-dimensional face landmarks to obtain the key vector.

[0083] Step S420: Iterate through and calculate the distance features between any two two-dimensional face landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors.

[0084] In this embodiment, key vectors are calculated from each pair of key landmarks (wherein, a key vector is a vector formed by connecting two facial landmarks, and subsequent distance and angle features can be calculated from this vector). Then, the distance features between facial key landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors are calculated.

[0085] Step S430: Based on the publicly available aesthetic dataset, calculate the Pearson correlation coefficient between each distance feature and the face beauty score in the two-dimensional face landmarks, and retain the features whose absolute value of the correlation coefficient is greater than the first threshold as the extracted two-dimensional geometric features.

[0086] Unlike conventional methods that subjectively select features, this method enhances objectivity by traversing all feature combinations, avoiding the problems of missing key aesthetic features or mistakenly selecting aesthetically irrelevant features due to subjective feature selection. After extracting all 2D features, Pearson correlation coefficients between each distance feature and the face beauty score are calculated on the publicly available aesthetic dataset SCUT-FBP5500. Pearson correlation coefficients between angle features and the face beauty score are also calculated. Features with an absolute correlation coefficient greater than 0.7 are then retained as extracted 2D features for subsequent aesthetic analysis.

[0087] Steps S410 to S430 above describe the method for extracting two-dimensional geometric features from two-dimensional face landmarks; while the method for extracting three-dimensional geometric features from three-dimensional face landmarks is as follows.

[0088] In one implementation of this embodiment, step S400 further includes the following steps:

[0089] Step S440: Iterate through and calculate the distance features between any two 3D face landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors;

[0090] Step S450: Select the plane formed by the endpoints of the eyebrows and the endpoint of the chin as the jaw plane;

[0091] Step S460: By calculating the distance between each three-dimensional facial landmark and the jaw plane, a depth feature representing the facial depth is obtained;

[0092] Step S470: Calculate the Pearson correlation coefficients of each distance feature, angle feature, depth feature, and face beauty score in the three-dimensional face landmarks, and retain the features whose absolute value of the correlation coefficient is greater than the second threshold as the extracted three-dimensional geometric features.

[0093] In this embodiment, the method uses face 3D reconstruction technology to reconstruct a 3D face model from the image, and extracts 3D features based on this model for aesthetic analysis. The method extracts three geometric features, including distance features, angle features, and depth features.

[0094] Specifically, this method considers that facial key points correspond one-to-one before and after the 3DMM model is mapped from 2D to 3D space. Therefore, it extracts 3D facial key points corresponding to 2D facial key landmarks to extract 3D geometric features. Similar to the 2D feature extraction method, it iterates and calculates the distance features between facial key landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features.

[0095] In addition, such as Figure 4 As shown, this method selects the plane formed by the endpoints of the eyebrows and the endpoint of the chin as the jaw plane (i.e., Figure 4 The plane containing the central triangle remains parallel to the facial plane under different poses and individuals. Therefore, facial depth information can be represented by calculating the distances between various landmarks and this plane (e.g., Figure 4 The black line in the middle represents the depth of the nose tip. Similarly, the depth information of landmarks in areas such as the eyes, eyebrows, lips, and jawline can be calculated. Likewise, the Pearson correlation coefficients of various distance features, angle features, depth features, and facial beauty scores are calculated. Features with an absolute correlation coefficient greater than 0.7 are retained as extracted 3D features for subsequent aesthetic analysis.

[0096] like Figure 1 As shown, in one implementation of the present invention, the facial aesthetic analysis method based on feature fusion further includes the following steps:

[0097] Step S500: Perform feature fusion based on the extracted two-dimensional and three-dimensional geometric features, and perform aesthetic task analysis based on the fused features, outputting the analysis results.

[0098] In this embodiment, the method proposes a regional feature extraction and fusion strategy. This scheme can selectively select key 2D and 3D geometric features from different facial regions for aesthetic analysis. Experiments have shown that the scheme outperforms aesthetic analysis using only 2D or only 3D features.

[0099] Step S510: The retained two-dimensional geometric features and three-dimensional geometric features are respectively fused and drawn in the face image;

[0100] Step S520: Based on the facial distribution of aesthetically strongly correlated features, find the areas on the face where the retained two-dimensional and three-dimensional geometric features are emphasized.

[0101] Step S530: Perform aesthetic task analysis based on the geometric features of the identified key representation region.

[0102] In this embodiment, the region selection strategy is specifically as follows:

[0103] 1. Select and retain 2D and 3D features that are significantly related to facial aesthetics.

[0104] 2. Plot the features (angle features, distance features, and depth features) that are significantly related to facial aesthetics from the 2D and 3D features onto the face image respectively.

[0105] 3. Based on the facial distribution of aesthetically strong features, identify the areas on the face where 2D and 3D features are emphasized, and use these areas to extract 2D and 3D geometric features in a targeted manner.

[0106] In this embodiment, the two-dimensional geometric features of the emphasized characterization area are: the distance features of the eyebrows, eyes, and nose; the three-dimensional geometric features of the emphasized characterization area are: the angular features of the eyes, nose, lips, and the entire jawline area, and the depth features of the eyes, eyebrows, lips, nose, and jawline area.

[0107] like Figures 5-6 As shown, the strategy for selecting regions in this method is as follows:

[0108] (1) 2D images are suitable for extracting distance features, but not for extracting angle features. The key 2D feature regions selected in this method primarily extract distance features between key landmarks from the eyebrows, eyes, and nose of the face, i.e., concentrated in the central facial region; angle features with a significant aesthetic impact are similar to distance features (such as...). Figure 6 The angle features marked by the light-colored straight line on the left actually represent the eye distance features. The other angle features have large errors in obtaining them from the image and are difficult to distinguish when used for aesthetic analysis. Therefore, angle features can be omitted from the image.

[0109] (2) Distance, angle, and depth features are suitable for extraction from 3D face models. This method selects key regions for 3D features. Specifically, the key extraction regions for distance features in 3D features are similar to those for 2D features but also have differences. They mainly include the eyes, nose, lips, chin, and cheeks of the face. 3D models are also suitable for extracting angle features for aesthetic analysis. The main extraction regions for angle features include the eyes, nose, lips, and the entire jawline. Depth features have a significant impact on aesthetic analysis. The eyes, eyebrows, lips, nose, and jawline are all suitable for extracting depth information. The 3D geometric information of these regions can effectively measure facial aesthetics.

[0110] This region-based feature extraction strategy guides the extraction of both 2D and 3D features, which complement each other to represent facial aesthetic information. Experiments on the SCUT-FBP5500 dataset using the fused features extracted through this method demonstrate that this invention enables 2D and 3D features to complement each other, achieving performance exceeding that of models using single features across different models. This invention designs a method for aesthetic analysis that simultaneously uses region-based feature extraction from two modalities, thereby achieving 2D and 3D feature fusion to improve the performance of aesthetic analysis tasks.

[0111] like Figure 3 As shown, in one implementation of this embodiment, the process for selecting 2D and 3D feature-focused regions for aesthetic purposes includes the following steps:

[0112] Step S21, input a face image;

[0113] Step S22a, 2D feature extraction; Step S22b, 3D reconstruction module reconstruction;

[0114] Step S23, 3D feature extraction;

[0115] Step S24, 2D / 3D facial feature analysis of key areas of interest;

[0116] Step S25a yields the 2D feature region; step S25b yields the 3D feature region.

[0117] It is worth mentioning that this method has been implemented using PyTorch and Matlab and tested on the publicly available face aesthetic image set SCUT-FBP5500. The experimental results show that, compared with existing face aesthetic analysis methods (using only 2D features or only 3D features), the fusion features obtained by the proposed regional feature extraction method can achieve the highest beauty score prediction accuracy under various common regression models and the best classification accuracy under various common classification models.

[0118] This solution is feasible and performs well in tests on different models. It enables the extraction of key aesthetic features from two-dimensional images and three-dimensional face models for aesthetic analysis, providing a new approach for aesthetic research and effectively promoting the further development of face aesthetics research.

[0119] The above-described technical solution in this embodiment has the following effects:

[0120] 1. This embodiment considers using two modalities, 2D face images and 3D face models, for aesthetic analysis. The method reconstructs face images using 3D reconstruction technology to obtain 3D face models. Then, 2D and 3D geometric features are extracted from different key regions in the face images and face models for aesthetic analysis, which plays a good complementary role.

[0121] 2. In this embodiment, 2D geometric features and 3D geometric features are obtained by traversing 2D and 3D facial key landmarks respectively. Then, the correlation between each feature and the degree of facial beauty is calculated. Subsequently, strongly correlated features are retained and the key regions of 2D features and key regions of 3D features are marked in the facial key landmarks respectively, so as to design an aesthetic analysis method for extracting features by region suitable for aesthetics.

[0122] 3. Distance features are suitable for extraction from 2D face images, but not for angular features. This method selects key 2D feature regions. The extraction regions for distance features between key landmarks are mainly the eyebrows, eyes, and nose, concentrated in the central area of ​​the face. Strongly correlated angular features mainly reflect facial distance information. Other angular features have large errors in image acquisition, and the obtained features have low correlation with facial aesthetics. Therefore, there are no suitable regions for extracting 2D angular features.

[0123] 4. Distance, angle, and depth features are suitable for extraction from 3D face models. This method selects key 3D feature regions. The extraction regions for distance features between key landmarks are mainly the eyes, nose, lips, chin, and cheeks. 3D models are also suitable for extracting angle features for aesthetic analysis; the main extraction regions for angle features include the eyes, nose, lips, and the entire jawline. The key regions for depth features are mainly the eyes, eyebrows, nose, lips, and jawline; depth features have a significant impact on aesthetics.

[0124] In summary, through a thorough analysis and comparison of the influence of 2D and 3D geometric features on the beauty of faces, this invention proposes a 2D and 3D feature fusion method suitable for facial aesthetic index analysis, thereby improving the accuracy of facial aesthetic index analysis.

[0125] Exemplary device

[0126] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 8 As shown.

[0127] The terminal includes: a processor, memory, interface, display screen, and communication module connected via a system bus; wherein, the processor provides computing and control capabilities; the memory includes a storage medium and internal memory; the storage medium stores the operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external terminal devices, such as mobile terminals and computers; the display screen is used to display corresponding facial aesthetic analysis information based on feature fusion; and the communication module is used to communicate with a cloud server or mobile terminal.

[0128] When executed by a processor, this computer program is used to implement a facial aesthetic analysis method based on feature fusion.

[0129] It will be understood by those skilled in the art that Figure 8 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a feature fusion-based facial aesthetic analysis program, which, when executed by the processor, is used to implement the feature fusion-based facial aesthetic analysis method as described above.

[0131] In one embodiment, a storage medium is provided, wherein the storage medium stores a feature fusion-based facial aesthetic analysis program, which, when executed by a processor, is used to implement the feature fusion-based facial aesthetic analysis method described above.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0133] In summary, this invention provides a facial aesthetic analysis method, terminal, and storage medium based on feature fusion. The method includes: acquiring a facial image to be analyzed; detecting two-dimensional facial landmarks in the facial image for aesthetic analysis; inputting the facial image into a three-dimensional reconstruction network to obtain a three-dimensional reconstructed facial model, and detecting three-dimensional facial landmarks in the facial image for aesthetic analysis; extracting two-dimensional geometric features from the two-dimensional facial landmarks, and extracting three-dimensional geometric features from the three-dimensional facial landmarks; performing feature fusion based on the extracted two-dimensional and three-dimensional geometric features, and performing aesthetic task analysis based on the fused features, outputting the analysis results. This invention, through a thorough analysis and comparison of the influence of 2D and 3D geometric features on facial beauty, proposes a 2D and 3D feature fusion method suitable for facial aesthetic index analysis, improving the accuracy of facial aesthetic index analysis.

[0134] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A facial aesthetic analysis method based on feature fusion, characterized in that, The facial aesthetic analysis method based on feature fusion includes the following steps: Obtain the face image to be analyzed; Detect two-dimensional facial landmarks in the facial image for aesthetic analysis; The face image is input into a 3D reconstruction network to obtain a 3D reconstructed face model, and 3D face landmarks for aesthetic analysis are detected in the face image. Extracting two-dimensional geometric features from the two-dimensional face landmarks and extracting three-dimensional geometric features from the three-dimensional face landmarks includes: calculating key vectors between any two two-dimensional face landmarks based on the two-dimensional face landmarks; calculating distance features between any two two-dimensional face landmarks, distance features between key landmarks and key vectors, distance features between two key vectors, and angle features between two key vectors. Feature fusion is performed based on the extracted two-dimensional and three-dimensional geometric features, and aesthetic task analysis is conducted based on the fused features, outputting the analysis results.

2. The facial aesthetic analysis method based on feature fusion according to claim 1, characterized in that, The traversal calculation yields the key vector between any two two-dimensional face landmarks, including: The key vector is obtained by traversing and calculating the vector formed by the line connecting any two two-dimensional face landmarks.

3. The facial aesthetic analysis method based on feature fusion according to claim 1, characterized in that, The extraction of two-dimensional geometric features from two-dimensional facial landmarks also includes: Based on the publicly available aesthetic dataset, the Pearson correlation coefficient between each distance feature and the facial beauty score in the two-dimensional facial landmarks is calculated, and features with an absolute value of the correlation coefficient greater than a first threshold are retained as extracted two-dimensional geometric features.

4. The facial aesthetic analysis method based on feature fusion according to claim 1, characterized in that, The extraction of 3D geometric features from 3D facial landmarks includes: Iterate and calculate the distance features between any two 3D face landmarks, the distance features between key landmarks and key vectors, the distance features between two key vectors, and the angle features between two key vectors; The plane formed by the endpoints of the eyebrows and the endpoint of the chin is selected as the jaw plane; By calculating the distance between each three-dimensional facial landmark and the jaw plane, depth features representing facial depth are obtained.

5. The facial aesthetic analysis method based on feature fusion according to claim 4, characterized in that, The extraction of three-dimensional geometric features from three-dimensional facial landmarks also includes: Calculate the Pearson correlation coefficients of distance features, angle features, depth features, and facial beauty scores in the three-dimensional face landmarks, and retain features whose absolute values ​​of the correlation coefficients are greater than the second threshold as extracted three-dimensional geometric features.

6. The facial aesthetic analysis method based on feature fusion according to claim 1, characterized in that, The process of feature fusion based on extracted two-dimensional and three-dimensional geometric features, followed by aesthetic task analysis based on the fused features, includes: The retained two-dimensional and three-dimensional geometric features are respectively fused and drawn into the face image; Based on the facial distribution of aesthetically strongly correlated features, identify the areas on the face where the preserved two-dimensional and three-dimensional geometric features are emphasized. Aesthetic task analysis is performed based on the geometric features of the identified key representation regions.

7. The facial aesthetic analysis method based on feature fusion according to claim 6, characterized in that, The two-dimensional geometric features of the emphasized region are: the distance features of the eyebrows, eyes, and nose; The three-dimensional geometric features of the emphasized region are: the angular features of the eyes, nose, lips and the entire jawline region, and the depth features of the eyes, eyebrows, lips, nose and jawline region.

8. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a facial aesthetic analysis program based on feature fusion, which, when executed by the processor, is used to implement the facial aesthetic analysis method based on feature fusion as described in any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores a face aesthetic analysis program based on feature fusion, which, when executed by a processor, is used to implement the face aesthetic analysis method based on feature fusion as described in any one of claims 1-7.

Images