Method and apparatus for determining local areas that influence facial aging

Abstract

The present application relates to facial aging prediction technology, and discloses a method and device for determining a local area that affects the degree of facial aging, which can accurately evaluate the degree of influence of a local area of ​​the face on facial aging. The method includes: acquiring a first facial image of an object; inputting the first facial image into an apparent age prediction model of a human face to obtain a first apparent age; performing image processing on the first facial image, changing a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image; and determining the degree of influence of the changed pixels and / or area in the second facial image on the first apparent age based on the first apparent age, the apparent age prediction model and the second facial image.

Description

Technical Field

[0001] The present application relates to facial aging prediction technology, and in particular to a technology for determining local areas that affect the degree of facial aging. Background Art

[0002] Facial aging refers to a complex biological process in which facial morphology and structure change over time. With the improvement of quality of life, people are paying more and more attention to facial aging. Facial aging is not only a criterion for judging health status in the biomedical field, but also a problem of general concern in society. Therefore, accurately predicting facial aging areas and helping individuals to delay or improve facial aging in a more targeted manner has extremely important research and application value. Summary of the invention

[0003] The purpose of the present application is to provide a method and device for determining a local area that affects the degree of facial aging, which can accurately evaluate the degree of influence of a local area of ​​the face on facial aging.

[0004] The present application discloses a method for determining a local area that affects the degree of facial aging, comprising:

[0005] Acquiring a first facial image of a subject;

[0006] Inputting the first facial image into an apparent age prediction model for a human face to obtain a first apparent age;

[0007] Performing image processing on the first facial image to change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image;

[0008] The degree of influence of the changed pixels and / or regions in the second facial image on the first apparent age is determined according to the first apparent age, the apparent age prediction model and the second facial image.

[0009] In a preferred embodiment, the image processing adopts a method selected from the following group:

[0010] Pixel derivative method, area masking method, or a combination thereof.

[0011] In a preferred example, the predetermined number of pixels are all pixels of the first facial image.

[0012] In a preferred embodiment, the predetermined area is selected from one or more of the following groups:

[0013] Eye area, cheek area, mouth area, forehead area.

[0014] In a preferred embodiment, the image processing adopts pixel derivation method;

[0015] The performing image processing on the first facial image to change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image further includes:

[0016] Performing image processing on the first facial image by using a pixel derivative method, and adding Gaussian noise to the predetermined number of pixel points to obtain a second facial image;

[0017] Determining the influence of the changed pixels and / or regions in the second facial image on the first apparent age according to the first apparent age, the apparent age prediction model and the second facial image further includes:

[0018] The second facial image is derived using the apparent age prediction model to obtain a derivative value corresponding to each pixel on the second facial image, and the influence of the changed pixel on the first apparent age is calculated based on the derivative value of each pixel.

[0019] In a preferred embodiment, the method further comprises: using the apparent age prediction model to derive the second facial image to obtain a derivative value corresponding to each pixel on the second facial image, and calculating the influence of the changed pixel on the first apparent age based on the derivative value of each pixel.

[0020] Dividing the first facial image into a plurality of local regions, and calculating the sum of derivative values ​​of all pixel points in each local region as a weight coefficient of influence of the local region on the overall facial aging degree;

[0021] Based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

[0022] In a preferred embodiment, the image processing adopts a regional masking method;

[0023] The performing image processing on the first facial image to change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image further comprises:

[0024] Performing image processing on the first facial image by using a region masking method, masking the predetermined region with a pixel mean value of the first facial image, so as to obtain a second facial image;

[0025] The determining, based on the first apparent age, the apparent age prediction model and the second facial image, the degree of influence of the changed pixels and / or regions in the second facial image on the first apparent age further includes:

[0026] inputting the second facial image into the apparent age prediction model to obtain a second apparent age;

[0027] The second apparent age is compared with the first apparent age, and the influence of the preset area on the apparent age of the human face is calculated based on the comparison result.

[0028] In a preferred embodiment, the step of performing image processing on the first facial image by using a region masking method and masking the predetermined region with the pixel mean of the first facial image to obtain the second facial image further comprises:

[0029] Dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with a pixel mean of the first facial image to obtain a corresponding second facial image covering each local region;

[0030] After dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with the pixel mean of the first facial image to obtain a corresponding second facial image covering each local region, the method further includes:

[0031] Counting the difference between the second apparent age and the first apparent age corresponding to each local area as a weight coefficient of influence of the local area on the overall facial aging degree;

[0032] Based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

[0033] In a preferred embodiment, the step of dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with a pixel mean of the first facial image to obtain a corresponding second facial image masking each local region further comprises:

[0034] The first facial image is divided into four local areas, namely, an eye area, a cheek area, a mouth area, and a forehead area. The first facial image is processed by using a regional masking method. Each local area is masked with a pixel mean of the first facial image in turn to obtain a corresponding second facial image covering each local area.

[0035] In a preferred embodiment, the apparent age prediction model is obtained by a method comprising the following steps:

[0036] Using a perceptual experiment to quantify the age distribution, age mean or age median of facial sample images as deep learning training labels to obtain a training sample set; and

[0037] The convolutional neural network model is trained using the training sample set to obtain the apparent age prediction model.

[0038] In a preferred example, the convolutional neural network model is a ResNet18 model.

[0039] The present application also discloses a device for determining a local area affecting the degree of facial aging, comprising:

[0040] An image acquisition module, used to acquire a first facial image of an object;

[0041] An image processing module, configured to perform image processing on the first facial image, and change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image;

[0042] an age prediction module, configured to input the first facial image into an apparent age prediction model of a human face to obtain a first apparent age;

[0043] An influence degree determination module is used to determine the influence degree of the changed pixels and / or regions in the second facial image on the first apparent age based on the first apparent age, the apparent age prediction model and the second facial image.

[0044] The present application also discloses a device for determining a local area affecting the degree of facial aging, comprising:

[0045] a memory for storing computer executable instructions; and,

[0046] A processor is used to implement the steps in the method described above when executing the computer executable instructions.

[0047] The present application also discloses a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the steps in the method described above are implemented.

[0048] In the implementation manner of the present application, at least the following advantages and beneficial effects are included:

[0049] The facial perception experimental method is used to quantify the facial aging phenotype as a whole, and combined with deep learning and visualization methods, the main influencing areas of overall facial aging are located, which can objectively and effectively evaluate the influence of local facial areas on facial aging.

[0050] Furthermore, through visualization methods such as pixel derivation and / or area masking, facial areas that affect facial aging can be located more accurately and objectively, providing a more scientific decision-making basis for medical and beauty fields.

[0051] Furthermore, deep learning models such as ResNet18 are used, and age distribution is used as training labels. The human perception age experiment is accurately simulated to quantify the overall facial aging phenotype, so as to more quickly and effectively evaluate the influence of local facial areas on facial aging.

[0052] A large number of technical features are recorded in the specification of this application, which are distributed in various technical solutions. If all possible combinations of technical features of this application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of this application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions are all deemed to have been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be deemed to have been recorded because it is technically infeasible, and the solution of A+B+C+E should be deemed to have been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a method for determining a local area that affects the degree of facial aging according to the first embodiment of the present application.

[0054] Figure 2 The figure shows a schematic diagram of the facial image data preprocessing process in an embodiment of the present application. Among them, Figure A is a schematic diagram of the recognition of 106 facial key points in the facial area; Figure B is a schematic diagram of the position of the facial central axis calculated using a regression model; Figure C is a schematic diagram of rotating the facial image according to the tilt angle to align the face vertically; Figures D and E are schematic diagrams of the effect of intercepting two images according to the mandibular point, left and right cheek points, and upper forehead point.

[0055] Figure 3 A schematic diagram of facial area division according to an embodiment of the present application is shown.

[0056] Figure 4 A schematic diagram of a curve showing the variation of sampling variance with the number of evaluators using 1000 samples in one embodiment of the present application is shown.

[0057] Figure 5 A schematic diagram of the deep learning, visualization and verification process in one embodiment of the present application is shown.

[0058] Figure 6 The schematic diagram of the ResNet18 network structure in one embodiment of the present application is shown. Figure a is a basic module of the residual network, which establishes a shortcut link from input to output; Figure b is the network structure of ResNet18, where the dotted line indicates feature doubling.

[0059] Figure 7 A schematic diagram comparing the training effects of three different deep learning models and three different training labels in one embodiment of the present application is shown.

[0060] Figure 8 A schematic diagram of facial region division and visualization processing in one embodiment of the present application is shown. FIG. a is a schematic diagram of facial region division, showing four facial regions; FIG. b is a schematic diagram of a second facial image obtained based on the pixel derivative method; FIG. d is a heat map of the aging degree of four parts obtained by statistically calculating the pixel derivative result of FIG. b; and FIG. c is a heat map of the aging degree of four parts obtained by using the result of the regional masking method.

[0061] Fig. 9 A schematic diagram of a process of using a pixel derivation method in one embodiment of the present application is shown.

[0062] Fig.10 A schematic diagram of a process flow of using a regional masking method in one embodiment of the present application is shown.

[0063] Fig.11 A schematic diagram of consistency check in one embodiment of the present application is shown. Figure A is an example of deep learning sorting results, and the numbers represent the order of importance of different regions; Figures BD show the sorting results of eye movement experiments (or manual evaluation), and the bold dashed boxes are the main comparison areas, where Figure B indicates that the order of the four regions is completely consistent, Figure C indicates that only the most important region is consistent, and Figure D indicates that the most important region is consistent with the second most important region.

[0064] Fig.12 A schematic diagram of the structure of a device for determining a local area that affects the degree of facial aging according to a second embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0066] Terminology explanation:

[0067] Visualization: Display the basis for the neural network's decision-making in the form of images or pictures.

[0068] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0069] The first embodiment of the present application relates to a method for determining a local area that affects the degree of facial aging, and the process is as follows: Figure 1 As shown, the method comprises the following steps:

[0070] In step 101, a first facial image of a subject is obtained;

[0071] In step 102, the first facial image is input into a human facial apparent age prediction model to obtain a first apparent age;

[0072] In step 103, image processing is performed on the first facial image to change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image;

[0073] In step 104, the influence of the changed pixels and / or regions in the second facial image on the first apparent age is determined based on the first apparent age, the apparent age prediction model and the second facial image.

[0074] Optionally, the first facial image in step 101 may be a whole facial image or a partial facial image.

[0075] Optionally, before step 102, the apparent age prediction model is obtained in advance through the following steps ① and ②: in step ①, the age distribution, age mean or age median of the facial sample images quantified by the perceptual experiment is used as a deep learning training label to obtain a training sample set; then step ② is executed to train a convolutional neural network model with the training sample set to obtain the apparent age prediction model.

[0076] Preferably, in step ①, age distribution is used as training label.

[0077] Optionally, the convolutional neural network model may be, but is not limited to, a VGG 16 model, a ResNet 18 model, or a ResNet 50. Preferably, the convolutional neural network model is a ResNet 18 model.

[0078] Optionally, in step 103, the image processing may adopt a pixel derivative method, an area masking method, or a combination thereof.

[0079] Optionally, the predetermined number of pixels are all pixels of the first facial image.

[0080] Optionally, the predetermined area is a sub-area of ​​m1 pixels×m2 pixels, and m1 and m2 are each independently a positive integer of 1-1000, preferably 2-500, more preferably 3-250, and most preferably 5-100. Optionally, the predetermined area is 0.01%-25% of the entire facial area, preferably 0.1-10%, and more preferably 1-5%.

[0081] Preferably, the predetermined area is selected from one or more of the following group: eye area, cheek area, mouth area, forehead area.

[0082] In one embodiment, step 103 can be further implemented as the following steps: using pixel derivation method to process the first facial image, adding Gaussian noise to the predetermined number of pixels to obtain the second facial image, for example, random Gaussian noise can be added to all pixels of the first facial image, but not limited thereto. Further, step 104 can be further implemented as the following steps: using the apparent age prediction model to derive the second facial image to obtain a derivative value corresponding to each pixel on the second facial image, and calculating the influence of the changed pixel on the first apparent age based on the derivative value of each pixel.

[0083] Optionally, after the above “using the apparent age prediction model to derive the second facial image to obtain a derivative value corresponding to each pixel on the second facial image, and calculating the degree of influence of the changed pixel on the first apparent age based on the derivative value of each pixel”, the following steps ① and ② are also included: in step ①, the first facial image is divided into multiple local areas, and the sum of the derivative values ​​of all pixels in each local area is counted as the influence weight coefficient of the local area on the overall facial aging degree; and in step ②, based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

[0084] In another embodiment, step 103 can be further implemented as the following steps: performing image processing on the first facial image using a regional masking method, masking the predetermined area with the pixel mean of the first facial image to obtain a second facial image. Further, step 104 can be further implemented as the following steps: inputting the second facial image into the apparent age prediction model of the human face to obtain a second apparent age, comparing the second apparent age with the first apparent age, and calculating the influence of the preset area on the apparent age of the human face based on the comparison result.

[0085] Optionally, the above-mentioned “using a regional masking method to perform image processing on the first facial image, and masking the predetermined area with the pixel mean of the first facial image to obtain a second facial image” is further implemented as follows: dividing the first facial image into a plurality of local areas, using a regional masking method to perform image processing on the first facial image, and sequentially masking each local area with the pixel mean of the first facial image to obtain a corresponding second facial image that masks each local area. For example, the first facial image can be divided into four local areas, namely, an eye area, a cheek area, a mouth area, and a forehead area, and using a regional masking method to perform image processing on the first facial image, and sequentially masking each local area with the pixel mean of the first facial image to obtain a corresponding second facial image that masks each local area.

[0086] Optionally, after the above “dividing the first facial image into multiple local areas, performing image processing on the first facial image using a regional masking method, and sequentially masking each local area with the pixel mean of the first facial image to obtain a corresponding second facial image covering each local area”, the following steps ① and ② are also included: in step ①, the difference between the second apparent age and the first apparent age corresponding to each local area is respectively counted as the influence weight coefficient of the local area on the overall facial aging degree; and in step ②, based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

[0087] The present invention will be further described below in conjunction with specific examples. It should be understood that the examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0088] In this embodiment, a deep learning network such as ResNet 18 is used to build a facial aging evaluation system, and deep learning visualization methods such as pixel derivative or facial masking are used to locate the main areas of facial aging. The specific scheme is as follows:

[0089] 1. Data Collection

[0090] In order to conduct the facial perception experiment, this example needs to remove some information such as the subject's hairstyle and clothing from the image. According to the experimental requirements, an automated process is designed to perform image preprocessing, such as Figure 2 shown.

[0091] First, face++ software was used to identify 106 facial key points in the facial area ( Figure 2 A, https: / / www.faceplusplus.com / ). Then, according to the position coordinates of the eyes, nose and mouth in the key points, the position of the facial central axis is calculated using the regression model ( Figure 2B, red solid line), and at the same time calculate the vertical line between the central axis and the numerical direction ( Figure 2 B, red dotted line). Based on this, the facial image is rotated according to the tilt angle to align the face vertically ( Figure 2 C). Finally, we cut the image based on the jaw point, left and right cheek points, and forehead point. The final effect is as follows: Figure 2 D and Figure 2 E. The captured images were used for subsequent experiments and analysis.

[0092] 2. Human Evaluation of Perceptual Age

[0093] We recruited 22 evaluators to conduct perceived age evaluation on the samples, including 10 male evaluators and 12 female evaluators. In order to minimize the experimental error, the evaluators used a unified display device for perceived age evaluation. Before the experiment, the evaluators did not know the age of the samples and the age distribution of the data set. During the evaluation, the evaluators needed to observe all 5,768 sample images, and then predict the age of each sample and record it. In addition, we selected 1,014 sample photos (500 males and 514 females). For these 1,014 sample photos, the evaluators needed to predict the age of each sample based on the age of the sample. Figure 3 Evaluate the facial areas that are of interest when evaluating the sample age, and then check them in the evaluation table. An example of the evaluation table is shown in Table 1 below, where multiple selections are allowed for the area part.

[0094] Table 1

[0095] Image Number Perceived age Eye mouth Forehead cheek All regions 15HanTZ0005TB1_F 15HanTZ0010TB1_F 15HanTZ0014TB1_F 15HanTZ0022TB1_F 15HanTZ0023TB1_F

[0096] 3. Analysis of Reviewer Evaluation Quality

[0097] A perceptual experiment is used to quantify overall facial aging and obtain high-quality deep learning training labels.

[0098] In order to verify the reliability of deep learning training labels (perceived age), we first conducted a simulation analysis of the number of evaluators and the quality of evaluation. We generated simulation parameters based on the real data of 22 evaluators, constructed evaluation data containing systematic errors and random errors, and used the relationship between the mean and standard error statistics and the number of evaluators to learn, thereby reflecting the relationship between the number of evaluators and the quality of evaluation.

[0099] First, the arithmetic mean of the evaluation values ​​of each sample by the 22 evaluators is taken as the true value of the sample, and the evaluation value of each evaluator for each sample is subtracted from the true value of the sample, and the arithmetic mean of the difference is taken as the systematic error of the evaluator. Then, the data after subtracting the evaluation value of each sample by each evaluator from the systematic error is taken as the random error of the evaluator. We calculate the standard deviation σ of the random errors of different evaluators ij, assuming that the random error follows N(0, σ ij ). Then, we generated 10,000 simulated data of perceived age from 1,000 evaluators based on the obtained simulation parameters (sample true value, systematic error, random error, etc.). i =1-100) are sampled from the simulated data. And 1000 simulated values ​​of perceived age are repeatedly sampled for the i-th sampling, and the variance of each perceived age is calculated. The calculation formula is as follows (1), X i It represents the simulated value of each subject’s perceived age in the i-th sampling, and n represents the number of selected evaluators.

[0100]

[0101] Figure 4 The variance of 1000 random samples varies with the number of evaluators. As can be seen from the figure, as the number of evaluators increases, the trend of variance reduction gradually slows down. The curve is derived to find the inflection point (n = 12), which is the optimal selection method for the number of evaluators. That is, under the premise of ensuring data quality, the minimum number of evaluators is 12, and the more evaluators there are, the better the data quality. In our perception experiment, there are 22 evaluators, which is far more than the optimal solution, which fundamentally guarantees the quality of perception data.

[0102] 4. Deep learning to obtain a local aging model

[0103] like Figure 5 As shown, deep learning uses 5,768 sample photos as training data, and uses the perceived age quantitative facial aging evaluated by the perceptual experiment as the deep learning training label; then deep learning visualization is used to locate the main areas that affect facial aging.

[0104] (1) Training dataset enhancement

[0105] Since the training data set only has 5,768 samples, the small amount of training data will not only affect the learning of network parameters, but also easily lead to overfitting. Therefore, before the actual training process, we use the traditional deep learning data enhancement method to enhance the training data set by mirroring and cropping. Mirror enhancement generates an image by reflecting an image relative to the Y axis, that is, generating two mirror images from one original image; crop enhancement captures five times in the upper and lower left, upper and lower right, and center to generate five different area images. Based on this, the training data set can be expanded tenfold.

[0106] (2) Deep learning model selection

[0107] Convolutional neural network models VGG 16, ResNet 18 and ResNet 50 are selected for age prediction. Among them, ResNet18 is one of the five main models of ResNet network structure, which mainly includes three parts: input, output and intermediate convolution. One of the main difficulties in training deep neural networks is that as the number of network layers increases, the parameters of the neural network will become difficult to optimize, and the problem of network degradation will occur, that is, the ability of the trained model to fit the data is even lower than that of the model with fewer network layers. Compared with other commonly used deep learning models, ResNet establishes a shortcut link from shallow features to deep features by introducing a residual module, and uses a neural network to model the residual between deep features and shallow features instead of the deep features themselves, so that when the back propagation algorithm is used to optimize the network parameters, the gradient can be more effectively returned, so the degradation problem of the deep learning network can be better solved. In addition, in order to verify the robustness of ResNet18, this project uses VGG16 and ResNet50 to measure the training effect of the samples. When training the three neural networks, the network parameters were initialized using the model pre-trained on ImageNet, the batch size was set to 64, the learning rate was set to 0.001, and a total of 100 epochs were trained.

[0108] Figure 6 (a) is the basic module of the residual network, which establishes a shortcut link from input to output, where weightlayer refers to the weight layer and relu is an activation function. Figure 6 (b) is the network structure of ResNet18, where the dotted line indicates feature doubling. The solid line connection part: indicates the same channel, such as 3*3conv,64 means using 64 3*3 convolution kernels for convolution; the dotted line connection part: indicates different channels, and the features are doubled, such as 3*3conv,128, / 2 means using 128 3*3 convolution kernels for convolution, / 2 means the feature layer is doubled compared to the upper layer 64.

[0109] During model training, the evaluation result distribution, evaluation median and evaluation mean of 22 age evaluators were used as labels for deep learning model training. After model training, the error variance between the predicted results and the actual results was used as the effect evaluation standard to select the model.

[0110] In addition, the project used the 10-fold cross-validation method to obtain the prediction data of the training data set, that is, the sample data was divided into 10 parts, and 9 parts were used for training each time to obtain the model to predict the age of the remaining 1 part. The cycle was repeated 10 times to use the deep learning model to obtain the perceived age data predicted by all samples.

[0111] (3) Comparison of deep learning model training effects

[0112] The deep learning method is used to simulate humans to evaluate facial appearance age, with a correlation of more than 96%. The model evaluation uses three training labels: age distribution, mean age, and median age of 22 evaluators, as well as three deep learning models: VGG16, ResNet 18, and ResNet 50.

[0113] The evaluation results are shown in Table 2 below.

[0114] Table 2

[0115]

[0116]

[0117] Note: The correlation coefficient was calculated using the Pearson correlation coefficient, and the P values ​​in the table are all less than 0.001.

[0118] The results show that using the ResNet18 model and using age distribution as training labels presents the best results (mean difference 2.27 years, correlation coefficient 0.96, as shown in Figure 2). Figure 7 ). Preferably, a ResNet18 model trained with age distribution as training labels is used for subsequent processes.

[0119] 5. Deep learning visualization to assess local facial aging

[0120] The pixel derivative method and the regional masking method are selected to realize visualization, and the face is divided into four regions: forehead, eyes, mouth and cheeks according to the facial anatomical structure and 106 facial feature points automatically calibrated by face++ ( Figure 8 a).

[0121] (1) Pixel Derivative Method

[0122] The core idea of ​​the pixel derivative method (SmoothGrad) is to differentiate the predicted perceived age with respect to the pixel value, and use the size of the derivative to measure the importance of each pixel in the image to the perceived age. Because neural networks are highly nonlinear mappings, there are usually a small number of pixels with very large derivatives, which makes visualization difficult. Therefore, random noise is added to the image, and the results of multiple derivatives are averaged to obtain a smoother visualization result. In order to determine the features that the deep learning model relies on when making predictions, we can establish a feature importance mask with the same size as the original image, whose brightness value corresponds to the importance of each pixel. The entire image is called a sensitivity mask. The pixel derivative method adds noise to each pixel separately, and then calculates the derivative of the perceived age with respect to the pixel with noise added, and uses the derivative to evaluate the importance of each pixel. According to the following formula (2), where n refers to the number of calculations, refers to Gaussian noise with a standard error of σ, x refers to the original pixel value, M c Finger sensitivity:

[0123]

[0124] After obtaining the importance of each pixel, we counted the mean of the derivatives of each facial region in the training, and sorted them by size to quantify the influence of local facial regions on facial aging. When applying the pixel derivative method, the number of calculations n was set to 10, and the standard deviation σ of the Gaussian noise was set to 0.3.

[0125] Specific as Fig. 9 , in step 901, the first facial image is input into the trained ResNet18 model to output the corresponding first apparent age; in step 902, the first facial image is copied n times, and random Gaussian noise is added to each of them to obtain the corresponding n second facial images; in step 903, for each second facial image, the derivative of the trained ResNet18 model for each pixel in the image is calculated; in step 904, the derivative results of the n second facial images are averaged and visualized to obtain the second facial image (such as Figure 8 b, the brighter the color, the more important the point is); in step 905, the derivative sum of each local area of ​​the face (mouth, forehead, eyes, cheeks) is counted, and each part is sorted according to the corresponding sum value. The higher the sum, the higher the importance and aging degree of the area, and each local area is marked with a color depth to indicate the degree of aging (such as Figure 8 d, the redder the color, the older it looks).

[0126] (2) Area masking method

[0127] The region masking rule is to mask each region with the mean of all pixels in the dataset and evaluate the importance of each region by the difference in prediction before and after masking. First, the model for perceptual age estimation is trained with unmasked images, and then the mean of all pixels in the image is calculated. For each facial partition of the sample ( Figure 8 a) Use the pixel mean to cover the four areas respectively, and then use the network to predict the age of the covered image, and then get the age difference before and after the cover. If the age difference is negative, it means that the covered area will cause the overall aging of the face to increase; on the contrary, it means that the covered area will cause the overall aging of the face to decrease, and the larger the difference, the stronger the degree.

[0128] The specific flow chart is as follows Fig.10 As shown, in step 1001, the first facial image (unblocked) is input into the trained ResNet18 model to output the corresponding first apparent age; in step 1002, the four local areas (mouth, forehead, eyes, cheeks) of the first facial image are blocked respectively and filled with the image mean to obtain the corresponding second facial image; in step 1003, the four blocked images are input into the trained ResNet18 model to obtain the corresponding second apparent age; in step 1004, the difference between the apparent age prediction results before and after blocking is used as the aging degree of each area, and the areas are sorted (such as Figure 8 c, the value corresponding to the color is the predicted age difference between before and after occlusion, the stronger the blue, the older, and the stronger the red, the younger).

[0129] 6. Result Verification

[0130] In order to verify the effectiveness of deep learning visualization results, perception experiment data, eye movement experiment data and facial local area aging score data were collected.

[0131] (1) Perception Experimental Data (Manual Evaluation)

[0132] In the perception experiment, 22 evaluators were selected to observe the pictures, predict the age and record the facial regions based on the judgment of the 1014 selected photos. Based on this, we counted the number of times each region was selected, and the number of times each region of each sample was selected by the evaluators was recorded as a number, and this value was used to quantify the influence of facial regions on facial aging assessment.

[0133] (2) Eye movement data (machine measurement)

[0134] Another 20 evaluators were selected to conduct eye movement experiments on 18 subjects' 2D facial images. The experiment used Eyelink 1000 eye tracker to observe and record the evaluator's gaze point in each facial partition ( Fig.11) to the total gaze time, which is a more objective and quantitative way of evaluating the influence of local facial areas on the overall facial aging assessment.

[0135] (3)Consistency verification method

[0136] In order to evaluate the consistency between different methods (manual evaluation, eye movement experiment and deep learning visualization), this project designed three consistency test indicators based on the numerical characteristics of different methods. Using these indicators, combined with the results of manual evaluation and eye movement experiments, the reliability of the deep learning visualization of this application is verified.

[0137] The indicators are described as follows:

[0138] Complete match rate: The ratio of the number of samples with complete matching in regional importance ranking to the total number of samples in different methods ( Fig.11 B). This ratio indicates that there is complete agreement between the two methods.

[0139] Top1 matching rate: the number of samples whose most important regions match each other in different methods ( Fig.11 C) Ratio to the total number of samples. This ratio shows that the two methods are completely consistent when exploring the local area that has the most significant impact on the overall aging of the face.

[0140] Top2 matching rate: the number of samples whose most important regions match each other in different methods ( Fig.11 C) and the most important matching samples of the second most important region ( Fig.11 D) The ratio of the sum of the numbers to the total number of samples. The higher the ratio, the stronger the consistency between the two methods.

[0141] (4) We used deep learning visualization methods to locate key areas that affect facial aging and verified the effectiveness of the results.

[0142] Deep learning visualization uses two methods, pixel derivation and region masking, and evaluates the visualization effect based on three evaluation criteria: complete consistency rate, key matching rate, and secondary key matching rate.

[0143] The evaluation results are shown in Table 3 below.

[0144] Table 3

[0145]

[0146] The results show that pixel derivatives achieve better results than region masking. The deep learning-pixel derivative visualization results have the highest complete consistency rate with manual evaluation (0.18 vs 0.13), which is about 38% higher than the region masking method.

[0147] In addition, when using deep learning-pixel derivative visualization, the Top1 matching rate is 0.52, and the Top2 matching rate reaches 0.89; its Top1 matching rate with the eye movement experiment is 0.61, and the Top2 matching rate is 0.85.

[0148] The above results show that deep learning visualization is highly consistent with the two manual methods, which further verifies the reliability of the deep learning visualization method.

[0149] The second embodiment of the present application relates to a device for determining a local area that affects the degree of facial aging, and its structure is as follows: Fig.12 As shown, the device for determining the local area affecting the degree of facial aging includes:

[0150] An image acquisition module, used to acquire a first facial image of an object;

[0151] An image processing module, configured to perform image processing on the first facial image, and change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image;

[0152] an age prediction module, configured to input the first facial image into an apparent age prediction model of a human face to obtain a first apparent age;

[0153] An influence degree determination module is used to determine the influence degree of the changed pixels and / or regions in the second facial image on the first apparent age based on the first apparent age, the apparent age prediction model and the second facial image.

[0154] The first implementation manner is a method implementation manner corresponding to the present implementation manner. The technical details in the first implementation manner can be applied to the present implementation manner, and the technical details in the present implementation manner can also be applied to the first implementation manner.

[0155] It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the above-mentioned embodiment of the device for determining the local area affecting the degree of facial aging can be understood by referring to the relevant description of the method for determining the local area affecting the degree of facial aging. The functions of each module shown in the above-mentioned embodiment of the device for determining the local area affecting the degree of facial aging can be implemented by a program (executable instruction) running on a processor, or by a specific logic circuit. If the above-mentioned device for determining the local area affecting the degree of facial aging in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), disk or optical disk and other media that can store program code. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0156] Accordingly, the present application also provides a computer-readable storage medium in which computer executable instructions are stored, and when the computer executable instructions are executed by the processor, the various method embodiments of the present application are implemented. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media do not include temporary computer-readable media (transitory media), such as modulated data signals and carriers.

[0157] In addition, the embodiment of the present application also provides a device for determining the local area that affects the degree of facial aging, which includes a memory for storing computer executable instructions, and a processor; the processor is used to implement the steps in the above-mentioned method implementations when executing the computer executable instructions in the memory. Among them, the processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as "DSP"), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as "ASIC"), etc. The aforementioned memory can be a read-only memory (read-only memory, referred to as "ROM"), a random access memory (random access memory, referred to as "RAM"), a flash memory (Flash), a hard disk or a solid-state hard disk, etc. The steps of the method disclosed in each embodiment of the present invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0158] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the element. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0159] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as a basis for modification when necessary. In addition, it should be understood that the above are only preferred embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for determining a local area that affects the degree of facial aging, characterized in that: include: Acquiring a first facial image of a subject; The first facial image is input into an apparent age prediction model of a human face to obtain a first apparent age, wherein the apparent age prediction model is obtained by a method comprising the following steps: The age distribution, age mean or age median of the facial sample images are quantified by perceptual experiments as deep learning training labels to obtain a training sample set; and Using the training sample set to train a convolutional neural network model to obtain the apparent age prediction model; Performing image processing on the first facial image, changing a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image, wherein the image processing adopts a pixel derivative method and / or a region masking method, wherein the pixel derivative method further includes adding Gaussian noise to the predetermined number of pixels to obtain the second facial image, and the region masking method further includes masking the predetermined area with a pixel mean value of the first facial image to obtain the second facial image; Determining the influence of the changed pixels and / or regions in the second facial image on the first apparent age according to the first apparent age, the apparent age prediction model and the second facial image, further comprising: The second facial image is differentiated using the apparent age prediction model to obtain a derivative value corresponding to each pixel on the second facial image, and the influence of the changed pixel on the first apparent age is calculated based on the derivative value of each pixel, and / or the second facial image is input into the apparent age prediction model to obtain a second apparent age, the second apparent age is compared with the first apparent age, and the influence of a preset area on the apparent age of the human face is calculated based on the comparison result.

2. The method for determining a local area affecting the degree of facial aging according to claim 1, characterized in that: The predetermined number of pixels are all the pixels of the first facial image.

3. The method for determining a local area affecting the degree of facial aging according to claim 1, characterized in that: The predetermined area is selected from one or more of the following groups: Eye area, cheek area, mouth area, forehead area.

4. The method for determining a local area affecting the degree of facial aging according to claim 1, characterized in that: After using the apparent age prediction model to derive the second facial image to obtain a derivative value corresponding to each pixel point on the second facial image, and calculating the influence of the changed pixel point on the first apparent age based on the derivative value of each pixel point, the method further includes: Dividing the first facial image into a plurality of local regions, and calculating the sum of derivative values ​​of all pixel points in each local region as a weight coefficient of influence of the local region on the overall facial aging degree; Based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

5. The method for determining a local area affecting the degree of facial aging according to claim 1, characterized in that: The step of performing image processing on the first facial image by using a region masking method and masking the predetermined region with the pixel mean of the first facial image to obtain a second facial image further comprises: Dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with a pixel mean of the first facial image to obtain a corresponding second facial image covering each local region; After dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with the pixel mean of the first facial image to obtain a corresponding second facial image covering each local region, the method further includes: Counting the difference between the second apparent age and the first apparent age corresponding to each local area as a weight coefficient of influence of the local area on the overall facial aging degree; Based on the influence weight coefficient of each local area, each local area is marked on the first facial image to obtain a third facial image.

6. The method for determining a local area affecting the degree of facial aging according to claim 5, characterized in that: The step of dividing the first facial image into a plurality of local regions, performing image processing on the first facial image using a regional masking method, and sequentially masking each local region with the pixel mean of the first facial image to obtain a corresponding second facial image covering each local region further comprises: The first facial image is divided into four local areas, namely, an eye area, a cheek area, a mouth area, and a forehead area. The first facial image is processed by using a regional masking method. Each local area is masked with a pixel mean of the first facial image in turn to obtain a corresponding second facial image covering each local area.

7. The method for determining a local area affecting the degree of facial aging according to claim 1, characterized in that: The convolutional neural network model is a ResNet18 model.

8. A device for determining local areas that affect the degree of facial aging, characterized in that: include: An image acquisition module, used to acquire a first facial image of an object; an image processing module, configured to perform image processing on the first facial image, change a predetermined number of pixels and / or a predetermined area in the first facial image to obtain a second facial image, wherein the image processing adopts a pixel derivative method and / or a region masking method, wherein the pixel derivative method further includes adding Gaussian noise to the predetermined number of pixels to obtain the second facial image, and the region masking method further includes masking the predetermined area with a pixel mean of the first facial image to obtain the second facial image; An age prediction module is used to input the first facial image into an apparent age prediction model of a human face to obtain a first apparent age, wherein the apparent age prediction model is obtained by a method comprising the following steps: The age distribution, age mean or age median of the facial sample images are quantified by perceptual experiments as deep learning training labels to obtain a training sample set; and Using the training sample set to train a convolutional neural network model to obtain the apparent age prediction model; an influence degree determination module, configured to determine the influence degree of the changed pixels and / or regions in the second facial image on the first apparent age according to the first apparent age, the apparent age prediction model and the second facial image, further comprising: The second facial image is differentiated using the apparent age prediction model to obtain a derivative value corresponding to each pixel on the second facial image, and the influence of the changed pixel on the first apparent age is calculated based on the derivative value of each pixel, and / or the second facial image is input into the apparent age prediction model to obtain a second apparent age, the second apparent age is compared with the first apparent age, and the influence of a preset area on the apparent age of the human face is calculated based on the comparison result.

9. A device for determining a local area affecting the degree of facial aging, characterized in that: include: A memory for storing computer executable instructions; as well as, A processor, configured to implement the steps in the method according to any one of claims 1 to 7 when executing the computer executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images