Appearance evaluation model training method and components, appearance evaluation method and components

Abstract

The present application discloses a method and component for training a face value evaluation model, a face value evaluation method and component, in which samples with face value labels, age labels and gender labels are used as model training samples, and facial feature information of the training samples is extracted at the same time, and the age evaluation sub-model, gender evaluation sub-model and face value scoring sub-model in the initial model are trained accordingly, and the face value evaluation model can be obtained after the training is completed. The model update process comprehensively considers factors such as gender, age, and facial features. Therefore, the face value evaluation model composed of the age evaluation sub-model, gender evaluation sub-model and face value scoring sub-model can comprehensively consider factors such as gender, age, and facial features, and output a relatively comprehensive face value score, thereby improving the model scoring accuracy and performance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a face value assessment model training method and component, and a face value assessment method and component. Background Art

[0002] At present, the facial appearance scoring model does not consider objective factors such as gender and age, and only scores appearance based on facial images. Since appearance itself is a relatively comprehensive subjective impression, the factors referenced by the existing facial appearance scoring model are relatively limited and cannot output a relatively comprehensive appearance score, that is, the model scoring accuracy is not accurate enough. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a face value evaluation model training method and component, a face value evaluation method and component, so as to improve the scoring accuracy of the face value scoring model. The specific scheme is as follows:

[0004] To achieve the above objectives, on the one hand, the present application provides a method for training an appearance evaluation model, comprising:

[0005] Obtaining labeled training samples from a face image sample set, wherein the labels include appearance labels, age labels, and gender labels;

[0006] Extracting facial feature information of the training sample;

[0007] The labels and facial feature information of the training samples are input into the initial model, and the age assessment sub-model of the initial model obtains an age weight prediction value and an age prediction value based on the facial feature information; the gender assessment sub-model of the initial model obtains a gender weight prediction value and a gender prediction result based on the facial feature information; the appearance scoring sub-model of the initial model obtains an appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value;

[0008] Determine a comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label;

[0009] Adjusting the parameters of the initial model based on the comprehensive loss to obtain an updated model;

[0010] When the model convergence conditions are met, the updated model is used as the appearance evaluation model.

[0011] On the other hand, the present application also provides a method for evaluating appearance, characterized in that it includes:

[0012] Extract facial feature information from face images;

[0013] The facial feature information is input into any one of the above-mentioned appearance assessment models, and the age assessment sub-model of the appearance assessment model obtains the age weight based on the facial feature information; the gender assessment sub-model of the appearance assessment model obtains the gender weight based on the facial feature information; the appearance scoring sub-model of the appearance assessment model obtains the appearance assessment information based on the facial feature information, the age weight and the gender weight.

[0014] On the other hand, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned method.

[0015] On the other hand, the present application further provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the aforementioned method is implemented.

[0016] This application uses samples with appearance labels, age labels and gender labels as model training samples, and extracts facial feature information of the training samples at the same time, uses the age assessment submodel of the initial model to process the facial feature information to obtain the age weight prediction value and the age prediction value, uses the gender assessment submodel of the initial model to process the facial feature information to obtain the gender weight prediction value and the gender prediction result, uses the appearance scoring submodel of the initial model to process the age weight prediction value, the gender weight prediction value and the facial feature information to obtain the appearance prediction value, and determines the comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label; if the parameters of the initial model are adjusted based on the comprehensive loss, and when the model convergence conditions are met, the updated initial model is used as the appearance evaluation model. It can be seen that the model update process comprehensively considers factors such as gender, age, and facial features. Therefore, the appearance evaluation model composed of the age assessment submodel, the gender assessment submodel and the appearance scoring submodel can comprehensively consider factors such as gender, age, and facial features, and output a relatively comprehensive appearance score, thereby improving the model scoring accuracy and performance.

[0017] Correspondingly, the present application provides a training device, equipment and storage medium for an appearance assessment model, and an appearance assessment device, equipment and storage medium, which also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a physical architecture applicable to the present application provided for the present application;

[0020] Figure 2 A flow chart of a method for training a face value assessment model provided in this application;

[0021] Figure 3 A schematic diagram of a face frame provided for this application;

[0022] Figure 4 A comparison diagram of the face frame before and after correction provided for this application;

[0023] Figure 5 A flow chart of a method for evaluating appearance provided in this application;

[0024] Figure 6 A flow chart of another appearance evaluation method provided for this application;

[0025] Figure 7 A schematic diagram of the steps of processing an image by a face value evaluation model provided in this application;

[0026] Figure 8 A schematic diagram of the structure and processing steps of a CNN branch 1 provided in this application;

[0027] Fig. 9 A schematic diagram of the structure and processing steps of a CNN branch2 provided in this application;

[0028] Fig.10 A schematic diagram of the structure and processing steps of a CNN branch 3 provided in this application;

[0029] Fig.11 A schematic diagram of a face value assessment model training device provided in this application;

[0030] Fig.12 A server structure diagram provided for this application;

[0031] Fig.13 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0032] The existing facial appearance scoring model does not consider objective factors such as gender and age, and only scores appearance based on facial images. Since appearance itself is a relatively comprehensive subjective impression, the factors referenced by the existing facial appearance scoring model are relatively limited and cannot output a relatively comprehensive appearance score, that is, the model scoring accuracy is not accurate enough.

[0033] In view of the above-mentioned problems that currently exist, this application proposes an appearance evaluation scheme, which comprehensively considers factors such as gender, age, and facial features in the model updating process. Therefore, the appearance evaluation model composed of the age assessment sub-model, the gender assessment sub-model and the appearance scoring sub-model can comprehensively consider factors such as gender, age, and facial features, and output a relatively comprehensive appearance score, thereby improving the model scoring accuracy and performance.

[0034] Technical terms involved in this application:

[0035] Deep learning algorithms (Deeplearning): Part of a broader family of machine learning methods, generally artificial neural networks with representation learning.

[0036] Convolution layer: A mathematical operator that generates a third function through two functions f and g, representing the integral of the overlapping length of the product of the overlapping function values ​​of the functions f and g after flipping and translation.

[0037] Pooling: It is an important concept in convolutional neural networks. It is actually a form of downsampling, and generally there are many different forms of nonlinear pooling functions.

[0038] Fully connected layer: Each node in the fully connected layer is connected to all nodes in the previous layer to combine the features extracted previously. Due to its fully connected nature, the fully connected layer generally has the most parameters.

[0039] Face detection technology: Detecting whether there is a face in a picture. The difficulty lies in the uncertain position, size, clarity, etc. of the face in the image.

[0040] For ease of understanding, the physical framework to which this application applies is first introduced.

[0041] It should be understood that the appearance evaluation model training method provided in this application can be applied to a system or program with a model training function. Of course, the appearance evaluation method provided in this application can also be applied to a system or program with appearance evaluation. Specifically, a system or program with a model training function, and a system or program with appearance evaluation can be run in a server, a personal computer, and other devices.

[0042] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture applicable to this application. Figure 1 In the invention, a system or program with a model training function can be run on a server, which obtains labeled training samples from a face picture sample set from other terminal devices through a network, wherein the labels include appearance labels, age labels and gender labels; extracts facial feature information of the training samples; inputs the labels and facial feature information of the training samples into an initial model, and the age assessment submodel of the initial model obtains an age weight prediction value and an age prediction value based on the facial feature information; the gender assessment submodel of the initial model obtains a gender weight prediction value and a gender prediction result based on the facial feature information; the appearance scoring submodel of the initial model obtains an appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value; determines a comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label; adjusts the parameters of the initial model based on the comprehensive loss to obtain an updated model; and uses the updated model as an appearance assessment model when the model convergence conditions are met.

[0043] exist Figure 1 In the present invention, the system or program with appearance evaluation can also run on a server, which obtains face pictures from other terminal devices through the network; extracts facial feature information of the face pictures; inputs the facial feature information into any of the above-mentioned appearance evaluation models, and the age evaluation sub-model of the appearance evaluation model obtains the age weight based on the facial feature information; the gender evaluation sub-model of the appearance evaluation model obtains the gender weight based on the facial feature information; the appearance scoring sub-model of the appearance evaluation model obtains the appearance evaluation information based on the facial feature information, the age weight and the gender weight.

[0044] As shown in the figure, the server can establish communication connections with multiple devices, and the server obtains facial images that meet the training conditions or need to be evaluated for appearance from these devices. The server collects facial images uploaded by these devices and trains the initial model including the age assessment sub-model, gender assessment sub-model and appearance scoring sub-model to obtain an appearance assessment model. Of course, the server also uses the appearance assessment model obtained through training to evaluate the appearance of faces in facial images.

[0045] Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the model training or appearance evaluation process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0046] It should be noted that the face evaluation model training method provided in this embodiment can be performed offline, that is, the server locally stores a set of face pictures, which can directly use the solution provided in this application to train the desired model. Of course, the face evaluation method provided in this embodiment can be performed offline, that is, the server locally stores the face pictures to be evaluated, which can directly use the solution provided in this application to evaluate the face pictures.

[0047] It is understandable that the above-mentioned systems and programs with model training functions, or systems and programs with appearance evaluation functions, can also be run on personal mobile terminals or as a type of cloud service program. The specific operating mode depends on the actual scenario and is not limited here.

[0048] Combined with the above commonalities, please see Figure 2 , Figure 2 This is a flow chart of a method for training a face value assessment model provided in an embodiment of the present application. Figure 2 As shown, the appearance evaluation model training method may include the following steps:

[0049] S201. Obtain labeled training samples from a face image sample set, where the labels include appearance labels, age labels, and gender labels.

[0050] The face image sample set includes multiple face images (i.e., multiple training samples). These face images can be original face images or partial images covered by face frames obtained by detecting the original face images using face detection technology. By detecting the original face images using face detection technology, the face frames corresponding to the original face images and the corresponding face key point information can be obtained. Figure 3 .

[0051] S202: Extract facial feature information of training samples.

[0052] In this embodiment, the training samples are processed using the face recognition model to obtain the corresponding face feature information, which can be represented by a face feature map.

[0053] In this embodiment, the facial feature information obtained by processing the facial image with the facial recognition model is used as training information. The facial recognition model can directly use an existing facial recognition model, which is used to extract facial feature information including common facial features (such as facial shape features, facial features, etc.) from the facial image.

[0054] S203. Input the labels and facial feature information of the training samples into the initial model. The age assessment submodel of the initial model obtains an age weight prediction value and an age prediction value based on the facial feature information. The gender assessment submodel of the initial model obtains a gender weight prediction value and a gender prediction result based on the facial feature information. The appearance scoring submodel of the initial model obtains an appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value.

[0055] S204: Determine the comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label.

[0056] The calculation of the above three losses may use the same or different loss functions.

[0057] S205. Adjust the parameters of the initial model based on the comprehensive loss to obtain an updated model.

[0058] S206, judging whether the comprehensive loss meets the model convergence condition; if so, executing S207; if not, executing S201 to re-obtain labeled training samples from the face image sample set and iteratively train the model.

[0059] S207: Use the updated model as a beauty evaluation model.

[0060] Here, executing S201 again can reacquire training samples, thereby iteratively training the updated model until the model convergence condition is reached.

[0061] This embodiment uses samples with appearance labels, age labels and gender labels as model training samples, and extracts facial feature information of the training samples at the same time, uses the age assessment submodel of the initial model to process the facial feature information to obtain the age weight prediction value and the age prediction value, uses the gender assessment submodel of the initial model to process the facial feature information to obtain the gender weight prediction value and the gender prediction result, uses the appearance scoring submodel of the initial model to process the age weight prediction value, the gender weight prediction value and the facial feature information to obtain the appearance prediction value, and determines the comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label; if the parameters of the initial model are adjusted based on the comprehensive loss and the model convergence condition is met, the updated initial model is used as the appearance assessment model. Otherwise, the training samples are re-acquired to iteratively train the updated model until the model convergence condition is met.

[0062] Among them, the parameters of the initial model are adjusted based on the comprehensive loss to obtain an updated model, including: updating the model parameters of the age assessment sub-model, the gender assessment sub-model and the appearance scoring sub-model based on the comprehensive loss, and combining the updated age assessment sub-model, the updated gender assessment sub-model and the updated appearance scoring sub-model into an updated model.

[0063] The initial model includes three parts: age assessment sub-model, gender assessment sub-model and appearance scoring sub-model. The age assessment sub-model, gender assessment sub-model and appearance scoring sub-model can be models of any structure based on deep learning algorithms.

[0064] It can be seen that this embodiment comprehensively considers factors such as gender, age, and facial features during the model updating process. Therefore, the appearance evaluation model composed of the age evaluation sub-model, the gender evaluation sub-model, and the appearance scoring sub-model can comprehensively consider factors such as gender, age, and facial features, and output a relatively comprehensive appearance score, thereby improving the model scoring accuracy and performance.

[0065] Based on the above embodiments, the detailed processing steps corresponding to the age assessment sub-model, the gender assessment sub-model and the appearance scoring sub-model can be referred to as follows.

[0066] In a specific implementation, the age assessment submodel of the initial model obtains an age weight prediction value and an age prediction value based on facial feature information, including: the age assessment submodel determines an age weighting weight based on facial feature information, uses the product of the age weighting weight and the facial feature information as an age feature, extracts an age depth feature from the age feature, and determines an age weight prediction value and an age prediction value based on the age depth feature. Wherein, the age assessment submodel determines an age weighting weight based on facial feature information, including: the age assessment submodel converts facial feature information into vector data, samples the vector data to obtain an age information sampling result, performs a linear transformation on the age information sampling result, and normalizes the linearly transformed age information sampling result to obtain an age weighting weight.

[0067] In a specific embodiment, the gender assessment submodel of the initial model obtains a gender weight prediction value and a gender prediction result based on facial feature information, including: the gender assessment submodel determines a gender weighting weight based on facial feature information, uses the product of the gender weighting weight and the facial feature information as a gender feature, extracts a gender depth feature from the gender feature, and determines a gender weight prediction value and a gender prediction result based on the gender depth feature. Wherein, the gender assessment submodel determines a gender weighting weight based on facial feature information, including: the gender assessment submodel converts facial feature information into vector data, samples the vector data to obtain a gender information sampling result, performs a linear transformation on the gender information sampling result, and normalizes the gender information sampling result after the linear transformation to obtain a gender weighting weight.

[0068] In a specific implementation, the appearance scoring submodel of the initial model obtains an appearance prediction value based on facial feature information, an age weight prediction value, and a gender weight prediction value, including: the appearance scoring submodel adds the age weight prediction value and the gender weight prediction value to obtain a comprehensive weight, normalizes the comprehensive weight, and uses the product of the normalized comprehensive weight and the facial feature information as an appearance feature, extracts an appearance depth feature from the appearance feature, and determines an appearance prediction value based on the appearance depth feature.

[0069] Based on the above embodiments, when the training sample is: a partial picture covered by each face frame obtained by detecting the original face picture using face detection technology, in a specific implementation, the generation process of the face picture sample set includes: obtaining a data set composed of original face pictures; detecting the face frame corresponding to each original face picture in the data set, and the face key point information corresponding to each face frame; for each face frame, using the face key point information corresponding to the face frame to perform position correction on the face frame; and forming the face picture sample set with the partial pictures covered by each face frame after the position correction.

[0070] Among them, to detect the face frame corresponding to each original face image in the data set, and the face key point information corresponding to each face frame, face detection technology can be used, and specifically a face detection network based on face detection technology can be used.

[0071] Among them, the position correction of the face frame is to rotate, translate, scale, etc. the face frame so that the face image covered by the face frame is displayed upright. For details, see Figure 4 .

[0072] The step of obtaining a data set composed of original face images includes: extracting frames from a video according to a preset frame rate to obtain a plurality of original face images, and forming a data set from the plurality of original face images. It can be seen that the original face images can be obtained by extracting frames from a video.

[0073] In order to make the model suitable for the facial features of the yellow race, the original face image can be a face image of the yellow race, or of other races.

[0074] In a specific implementation, detecting a face frame corresponding to each original face image in a data set, and face key point information corresponding to each face frame, includes: inputting each original face image in the data set into a face detection network combination, and the face detection network combination outputs a face frame corresponding to each original face image in the data set, and face key point information corresponding to each face frame.

[0075] The face detection network combination includes at least three face detection networks. The structures of the face detection networks in the face detection network combination are generally different.

[0076] If the face detection network combination includes three face detection networks, the structure of the first face detection network is: 3 convolutional layers; the structure of the second face detection network is: 3 convolutional layers and 1 fully connected layer; the structure of the third face detection network is: 4 convolutional layers and 1 fully connected layer. Any original face image in the data set is sequentially input into the first face detection network (used to output a rough face frame and corresponding face key point information), the second face detection network (used to remove false positive face frames, that is, non-face frames), and the third face detection network (used to make the position coordinates of the face frame more precise), and the face frame corresponding to the original face image and the corresponding face key point information can be obtained. Generally, an original face image can correspond to at least one face frame, and each face frame corresponds to a face key point information.

[0077] For example, the first face detection network is P-Net (Proposal Network), which has a structure of 3 convolutional layers. P-Net is a candidate network for face regions. The input of the network is set to a 12x12x3 image. After passing through 3 convolutional layers, it determines whether there is a face in the 12x12 image and gives the face frame and face key point information. Specifically, the first part of the network output is used to determine whether there is a face in the image, and the output vector size is 1x1x2, that is, two values. The second part of the network gives the exact position of the frame, which is generally called frame regression. Because the 12×12 image input to P-Net may not have a perfect face frame position, such as sometimes the face is not exactly square, and the face in the image may be biased to the left or right, it is necessary to output the offset of the current frame position relative to the perfect face frame position. The offset size is 1×1×4, which means the relative offset of the horizontal coordinate of the upper left corner of the frame, the relative offset of the vertical coordinate of the upper left corner of the frame, the error of the width of the frame, and the error of the height of the frame. The third part of the network gives the positions of the 5 key points of the face. The five key points correspond to the position of the left eye, the position of the right eye, the position of the nose, the position of the left mouth, and the position of the right mouth. Each key point requires two dimensions to represent, so the output is a vector of size 1×1×10.

[0078] The second face detection network is R-Net (Refine Network). The structure of this network is different from that of P-Net. It has an additional fully connected layer, so it can better suppress false positives. Before the output of the first network is input into R-Net, it needs to be scaled to 24x24x3. The output of R-Net is the same as that of P-Net. The purpose of R-Net is to remove a large number of non-face frames.

[0079] The third face detection network is O-Net (Output Network). The structure of this network has one more convolutional layer than R-Net, so the processing result will be more refined. The image size of the input to O-Net is 48x48x3, and the output includes the coordinate information of N face frames, the confidence of the face frames, and the location of key points.

[0080] Among them, the number of face detection networks in the face detection network combination, the number of convolutional layers and fully connected layers in each face detection network can be increased or decreased. Reducing the number will make the post-processing faster, but the accuracy will be reduced; increasing the number will make the post-processing slower, but the accuracy will be improved. Of course, the input size supported by the face detection network in the face detection network combination can also be flexibly adjusted.

[0081] In a specific implementation, if the face detection network combination also outputs the confidence of each face frame, when the confidence of any face frame is higher than the confidence threshold, it can be considered that the face frame is not a false positive face frame, and therefore the step of correcting the position of the face frame using the facial key point information corresponding to the face frame can be performed; otherwise, it can be considered that the face frame is a false positive face frame, that is, the local image covered by the face frame is not a face, and therefore the local image covered by the face frame and the facial key point information corresponding to the face frame are discarded.

[0082] In a specific implementation, each original face image in a data set is input into a face detection network combination, and the face detection network combination outputs a face frame corresponding to each original face image in the data set, and face key point information corresponding to each face frame, including: for each original face image in the data set, scaling the original face image according to different proportions to obtain images to be processed of different sizes; using the face detection network combination to detect each image to be processed, respectively, to obtain a face frame corresponding to each image to be processed and corresponding face key point information, as well as a confidence of each face frame; based on the confidence of the face frame and repeated filtering technology, the face frames and corresponding face key point information corresponding to all images to be processed are screened to obtain the face frame and corresponding face key point information corresponding to the original face image.

[0083] It can be seen that for any original face image in the data set, the size of the original face image can be changed, and the images of these sizes can be processed separately, so as to obtain more face frames and more training samples. At the same time, the appearance evaluation model can also learn the recognition ability of images of different sizes, thereby improving the model performance.

[0084] See also Figure 5 , Figure 5 This is a flow chart of a method for evaluating appearance provided in an embodiment of the present application. Figure 5 As shown, the method may include the following steps:

[0085] S501, obtaining a face image to be evaluated;

[0086] S502, extracting facial feature information of the face image;

[0087] S503. Input the facial feature information into the appearance evaluation model in any of the above embodiments. The age evaluation sub-model of the appearance evaluation model obtains the age weight based on the facial feature information. The gender evaluation sub-model of the appearance evaluation model obtains the gender weight based on the facial feature information. The appearance scoring sub-model of the appearance evaluation model obtains the appearance evaluation information based on the facial feature information, the age weight and the gender weight.

[0088] Of course, the age assessment sub-model can also output age, and the gender assessment sub-model can also output gender.

[0089] In this embodiment, for any face image to be evaluated, the age evaluation sub-model, gender evaluation sub-model and appearance scoring sub-model in the appearance evaluation model are used to process it, and the appearance evaluation information, age and gender of the image can be obtained. Of course, if there is no face in the face image to be evaluated, that is, the image without a face is mistakenly identified as a face image, the image can be directly discarded and an error is reported.

[0090] It can be seen that this embodiment uses the appearance evaluation model to evaluate the appearance of the face in the picture, and can obtain a relatively comprehensive appearance score, and can also output age and gender, so the result is more accurate.

[0091] The following is an introduction to the appearance evaluation solution provided by this application through a specific application scenario example description.

[0092] Set up a face recognition model and a face value evaluation model in the live broadcast platform. The face recognition model is used to extract facial feature information from images, and the face value evaluation model can evaluate the face value score, age, and gender of the face in the live broadcast video in real time based on the facial feature information. Figure 6 The specific process includes:

[0093] S601, the live broadcast terminal requests the live broadcast platform to run the live broadcast service;

[0094] S602, the live broadcast platform feeds back a response message to the live broadcast terminal;

[0095] S603, the live broadcast terminal starts live broadcast based on the response message;

[0096] S604, the live broadcast platform obtains the live broadcast video from the live broadcast terminal in real time;

[0097] S605, the live broadcast platform inputs the live broadcast video into the face recognition model in real time to obtain face feature information; inputs the face feature information into the appearance evaluation model to obtain the appearance score, age and gender of the face in the live broadcast video;

[0098] S606, the live broadcast platform determines a recommendation strategy based on the appearance score, age and gender;

[0099] S607: The live broadcast platform sets the corresponding position of the live broadcast video on the platform homepage according to the recommended strategy.

[0100] Among them, the live broadcast platform can be deployed in the server, and the live broadcast terminal can be a terminal capable of live broadcast, such as a smart phone, tablet computer, laptop computer or desktop computer.

[0101] exist Figure 6The steps for processing faces in live video by face recognition model and appearance evaluation model can be referred to in Figure 7 .exist Figure 7 In the figure, base CNN is the face recognition model, and the appearance assessment model is composed of CNN branch1 (i.e., age assessment sub-model), CNN branch2 (i.e., gender assessment sub-model), and CNN branch3 (i.e., appearance scoring sub-model).

[0102] Specifically, the base CNN can be obtained by training based on resnet50 and a yellow-skinned face dataset. The base CNN can convert a face image into a corresponding base feature map (i.e., a face feature map).

[0103] The above base CNN is used as a pre-trained model for the appearance evaluation model. The output of the pre-trained model is the base feature map. The base feature map is followed by three branch processes, such as Figure 8 , Fig. 9 , Fig.10 .in, Figure 8 The structure and detailed processing steps of CNN branch 1 are shown, and the final output is the age value and weight 1; Fig. 9 The structure and detailed processing steps of CNN branch 2 are shown, and the gender class and weight 2 are finally output; Fig.10 The structure and detailed processing steps of CNN branch 3 are shown, and finally the appearance information (i.e., appearance score) is output.

[0104] exist Figure 8 and Fig. 9 In the figure, the structures of CNN branch1 and CNN branch2 are the same, but in fact, the structures of CNN branch1 and CNN branch2 can be different.

[0105] in, Figure 8The detailed processing steps of the illustrated CNN branch1 include: CNN branch1 receives the feature map output by the base CNN. The feature map size is recorded as CHW, where C is the number of feature map channels, and HW is the height and width of the feature map, respectively. One path passes through the self-attention layer (Scale); the other path passes through a global pooling layer to convert the feature map matrix into a vector (length C), and then passes through two fully connected layers (FC) for linear transformation (the number of fully connected layers here can be increased or decreased, and its function is to make the age weighting more reasonable), and then passes through a sigmoid to map all values ​​of the vector to between 0 and 1 to obtain the age weighting weight, which is multiplied with the feature map output by the previous base CNN to obtain the weighted feature map; the weighted feature map passes through a CNN block (including convolution, batch norm, relu and other structures to extract deeper and more comprehensive features), global After pooling, it passes through two branches. One branch gives the age information after passing through a fully connected layer (the number of fully connected layers here can be increased or decreased), and the other branch passes through two fully connected layers (the number of fully connected layers here can be increased or decreased). After a sigmoid, all values ​​of the vector are mapped to between 0 and 1, and the output weight is 1 (i.e., the age weight).

[0106] Fig. 9 The detailed processing steps of the illustrated CNN branch2 are similar to those of CNN branch1 and will not be repeated here. CNN branch2 outputs gender information and weight 2 (i.e., gender weight).

[0107] Fig.10 The detailed processing steps of the illustrated CNN branch 3 include: CNN branch 3 receives the base feature map; Figure 8 , Fig. 9 The output weights 1 and 2 are first added in the same vector, and then a sigmoid is used to map all the values ​​of the vector to between 0 and 1. The weighted feature map is multiplied with the feature map output by the previous base CNN to obtain the weighted feature map. After passing through a CNN block (the structure and function are the same as above), golf pooling, and a fully connected layer (the number of fully connected layers can be increased or decreased here), the final appearance score is output.

[0108] It can be seen that this embodiment can score the appearance of the face pictures in the video. Specifically, it includes: directly extracting frames from the video to obtain pictures, detecting the faces in the pictures, correcting the faces, and sending them to the face recognition model and the appearance evaluation model to obtain the age, gender, and appearance score. The scoring accuracy and stability of the appearance evaluation model have been improved, and the gender and age are also given. The face recognition model and the appearance evaluation model can also be used in scenarios such as making friends in the same city and drifting bottles to achieve accurate recommendations and improve user stickiness.

[0109] See also Fig.11 , Fig.11 A schematic diagram of a face value assessment model training device provided in an embodiment of the present application includes:

[0110] The sample acquisition module 1101 is used to acquire labeled training samples from the face image sample set, where the labels include appearance labels, age labels, and gender labels;

[0111] A training feature extraction module 1102 is used to extract facial feature information of training samples;

[0112] Processing module 1103, used for inputting the label and facial feature information of the training sample into the initial model, the age assessment sub-model of the initial model obtains the age weight prediction value and the age prediction value based on the facial feature information; the gender assessment sub-model of the initial model obtains the gender weight prediction value and the gender prediction result based on the facial feature information; the appearance scoring sub-model of the initial model obtains the appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value;

[0113] A loss determination module 1104, configured to determine a comprehensive loss based on a loss between an age prediction value and an age label, a loss between a gender prediction result and a gender label, and a loss between an appearance prediction value and an appearance label;

[0114] An updating module 1105 is used to adjust the parameters of the initial model based on the comprehensive loss to obtain an updated model;

[0115] The output module 1106 is used to use the updated model as the appearance evaluation model when the model convergence conditions are met.

[0116] In a specific embodiment, it also includes:

[0117] The iteration module is used to re-acquire training samples from the face image sample set when the model convergence conditions are not met, so as to iteratively train the updated model until the model convergence conditions are met, and the updated model is used as the appearance evaluation model.

[0118] In a specific implementation, the update module is specifically used to:

[0119] Based on the comprehensive loss, the model parameters of the age assessment sub-model, the gender assessment sub-model and the appearance scoring sub-model are updated respectively, and the updated age assessment sub-model, the updated gender assessment sub-model and the updated appearance scoring sub-model are combined into an updated model.

[0120] In a specific implementation, the age assessment sub-model determines an age weighting weight based on facial feature information, takes the product of the age weighting weight and the facial feature information as the age feature, extracts the age depth feature from the age feature, and determines the age weight prediction value and the age prediction value based on the age depth feature.

[0121] In a specific implementation, the age assessment sub-model converts facial feature information into vector data, samples the vector data to obtain age information sampling results, performs linear transformation on the age information sampling results, normalizes the linearly transformed age information sampling results, and obtains an age weighted weight.

[0122] In a specific embodiment, the gender assessment sub-model determines the gender weighting weight based on facial feature information, takes the product of the gender weighting weight and the facial feature information as the gender feature, extracts the gender deep feature from the gender feature, and determines the gender weight prediction value and the gender prediction result based on the gender deep feature.

[0123] In a specific implementation, the gender assessment sub-model converts facial feature information into vector data, samples the vector data to obtain a gender information sampling result, performs a linear transformation on the gender information sampling result, and normalizes the gender information sampling result after the linear transformation to obtain a gender weighted weight.

[0124] In a specific implementation, the appearance scoring submodel adds the age weight prediction value and the gender weight prediction value to obtain a comprehensive weight, normalizes the comprehensive weight, and uses the product of the normalized comprehensive weight and facial feature information as the appearance feature, extracts the appearance depth feature from the appearance feature, and determines the appearance prediction value based on the appearance depth feature.

[0125] In a specific implementation, a generation module is also included for generating a face image sample set, and the generation module includes:

[0126] An acquisition unit, used for acquiring a data set consisting of original face images;

[0127] A detection unit, used to detect the face frame corresponding to each original face image in the data set, and the face key point information corresponding to each face frame;

[0128] A correction unit, for correcting the position of each face frame by using the face key point information corresponding to the face frame;

[0129] The composition unit is used to compose a face picture set from the partial pictures covered by each face frame after position correction.

[0130] In a specific implementation, the acquisition unit is specifically used to:

[0131] The video is frame-sampled according to a preset frame rate to obtain multiple original face images, and the multiple original face images are combined into a data set.

[0132] In a specific embodiment, the detection unit is specifically used for:

[0133] Input each original face image in the data set into the face detection network combination, and the face detection network combination outputs the face frame corresponding to each original face image in the data set, as well as the face key point information corresponding to each face frame;

[0134] Wherein, the face detection network combination includes at least three face detection networks.

[0135] In a specific embodiment, the detection unit is further used for:

[0136] If the face detection network combination also outputs the confidence of each face frame, when the confidence of any face frame is higher than the confidence threshold, the step of correcting the position of the face frame using the facial key point information corresponding to the face frame is executed; otherwise, the local image covered by the face frame and the facial key point information corresponding to the face frame are discarded.

[0137] In a specific embodiment, the detection unit comprises:

[0138] The scaling subunit is used to scale each original face image in the data set according to different proportions to obtain to-be-processed images of different sizes;

[0139] The detection subunit is used to detect each image to be processed by using the face detection network combination, and obtain the face frame and corresponding face key point information corresponding to each image to be processed, as well as the confidence of each face frame;

[0140] The screening subunit is used to screen the face frames and the corresponding face key point information corresponding to all the images to be processed based on the confidence of the face frames and the repeated filtering technology, and obtain the face frames and the corresponding face key point information corresponding to the original face image.

[0141] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0142] It can be seen that this embodiment provides a device for training an appearance assessment model, which comprehensively considers factors such as gender, age, and facial features during the model updating process. Therefore, the appearance assessment model composed of an age assessment sub-model, a gender assessment sub-model, and an appearance scoring sub-model can comprehensively consider factors such as gender, age, and facial features, and output a relatively comprehensive appearance score, thereby improving the model scoring accuracy and performance.

[0143] Furthermore, an embodiment of the present application provides a device for evaluating appearance, the device comprising:

[0144] Evaluation feature extraction module, used to extract facial feature information of face images;

[0145] An evaluation module is used to input facial feature information into an appearance evaluation model in any of the above embodiments; the age evaluation sub-model of the appearance evaluation model obtains an age weight based on the facial feature information; the gender evaluation sub-model of the appearance evaluation model obtains a gender weight based on the facial feature information; and the appearance scoring sub-model of the appearance evaluation model obtains appearance evaluation information based on the facial feature information, the age weight and the gender weight.

[0146] Furthermore, the present application also provides an electronic device. The electronic device can be Fig.12 The server 50 shown may also be Fig.13 Terminal 60 is shown. Fig.12 and Fig.13 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0147] Fig.12 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the corresponding method disclosed in any of the aforementioned embodiments.

[0148] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0149] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0150] The operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50, so as to realize the operation and processing of the data 523 in the memory 52 by the processor 51, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the corresponding methods disclosed in any of the aforementioned embodiments, the computer program 522 can also further include computer programs that can be used to complete other specific tasks. In addition to including application update information, pictures to be processed, videos and other data, the data 523 can also include application developer information and other data.

[0151] Fig.13 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0152] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0153] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0154] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the appearance assessment model training method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include, but is not limited to, update information of the application.

[0155] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0156] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0157] Furthermore, the embodiment of the present application also discloses a storage medium, in which computer executable instructions are stored, and when the computer executable instructions are loaded and executed by the processor, the corresponding method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0158] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0159] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0160] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for training a face value assessment model, characterized in that: include: Obtaining labeled training samples from a face image sample set, wherein the labels include appearance labels, age labels, and gender labels; Extracting facial feature information of the training sample; Inputting the label and facial feature information of the training sample into an initial model, wherein the age assessment submodel of the initial model obtains an age weight prediction value and an age prediction value based on the facial feature information; The gender assessment sub-model of the initial model obtains a gender weight prediction value and a gender prediction result based on the facial feature information; The appearance scoring sub-model of the initial model obtains an appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value; Determine a comprehensive loss based on the loss of the age prediction value and the age label, the loss of the gender prediction result and the gender label, and the loss of the appearance prediction value and the appearance label; Adjusting the parameters of the initial model based on the comprehensive loss to obtain an updated model; When the model convergence conditions are met, the updated model is used as the appearance evaluation model; The appearance scoring submodel of the initial model obtains the appearance prediction value based on the facial feature information, the age weight prediction value and the gender weight prediction value, including: The appearance scoring submodel adds the age weight prediction value and the gender weight prediction value to obtain a comprehensive weight, normalizes the comprehensive weight, and takes the product of the normalized comprehensive weight and the facial feature information as the appearance feature, extracts the appearance depth feature from the appearance feature, and determines the appearance prediction value based on the appearance depth feature.

2. The method according to claim 1, characterized in that: Also includes: When the model convergence condition is not met, training samples are re-obtained from the face image sample set to iteratively train the updated model until the model convergence condition is met, and the updated model is used as the appearance evaluation model.

3. The method according to claim 1, characterized in that The adjusting the parameters of the initial model based on the comprehensive loss to obtain an updated model includes: Based on the comprehensive loss, the model parameters of the age assessment sub-model, the gender assessment sub-model and the appearance scoring sub-model are updated respectively, and the updated age assessment sub-model, the updated gender assessment sub-model and the updated appearance scoring sub-model are combined into the updated model.

4. The method according to any one of claims 1 to 3, characterized in that: The age assessment sub-model of the initial model obtains an age weight prediction value and an age prediction value based on the facial feature information, including: The age assessment sub-model determines an age weighting weight based on the facial feature information, takes the product of the age weighting weight and the facial feature information as an age feature, extracts an age depth feature from the age feature, and determines the age weight prediction value and the age prediction value based on the age depth feature.

5. The method according to claim 4, characterized in that The age assessment sub-model determines the age weighting weight based on the facial feature information, including: The age assessment sub-model converts the facial feature information into vector data, samples the vector data to obtain an age information sampling result, performs a linear transformation on the age information sampling result, and normalizes the linearly transformed age information sampling result to obtain the age weighting weight.

6. The method according to any one of claims 1 to 3, characterized in that: The gender assessment sub-model of the initial model obtains a gender weight prediction value and a gender prediction result based on the facial feature information, including: The gender assessment sub-model determines a gender weighting based on the facial feature information, takes the product of the gender weighting and the facial feature information as the gender feature, extracts a gender depth feature from the gender feature, and determines the gender weight prediction value and the gender prediction result based on the gender depth feature.

7. The method according to claim 6, characterized in that The gender assessment sub-model determines the gender weighting based on the facial feature information, including: The gender assessment sub-model converts the facial feature information into vector data, samples the vector data to obtain a gender information sampling result, performs a linear transformation on the gender information sampling result, and normalizes the gender information sampling result after the linear transformation to obtain the gender weighted weight.

8. The method according to any one of claims 1 to 3, characterized in that: The generation process of the face image sample set includes: Get a dataset consisting of original face images; Detecting the face frame corresponding to each original face image in the data set, and the face key point information corresponding to each face frame; For each face frame, the position of the face frame is corrected using the face key point information corresponding to the face frame; The face picture sample set is composed of the local pictures covered by each face frame after position correction.

9. The method according to claim 8, characterized in that The step of obtaining a data set consisting of original face images includes: The video is frame-sampled according to a preset frame rate to obtain a plurality of original face images, and the plurality of original face images are combined into the data set.

10. The method according to claim 8, characterized in that The detecting of the face frame corresponding to each original face image in the data set, and the face key point information corresponding to each face frame, includes: Input each original face image in the data set into a face detection network combination, and the face detection network combination outputs a face frame corresponding to each original face image in the data set, and face key point information corresponding to each face frame; Wherein, the face detection network combination includes at least three face detection networks.

11. The method according to claim 10, characterized in that Also includes: If the face detection network combination also outputs the confidence of each face frame, when the confidence of any face frame is higher than the confidence threshold, the step of correcting the position of the face frame using the face key point information corresponding to the face frame is performed; Otherwise, the partial image covered by the face frame and the facial key point information corresponding to the face frame are discarded.

12. The method according to claim 10, characterized in that The step of inputting each original face image in the data set into a face detection network combination, and outputting a face frame corresponding to each original face image in the data set and face key point information corresponding to each face frame by the face detection network combination, comprises: For each original face image in the data set, scaling the original face image according to different proportions to obtain to-be-processed images of different sizes; Utilizing the face detection network combination to detect each to-be-processed picture respectively, obtaining a face frame and corresponding face key point information corresponding to each to-be-processed picture, as well as the confidence of each face frame; Based on the confidence of the face frame and repeated filtering technology, the face frames and corresponding face key point information corresponding to all the images to be processed are screened to obtain the face frames and corresponding face key point information corresponding to the original face image.

13. A method for evaluating appearance, characterized in that: include: Extract facial feature information from face images; The facial feature information is input into the appearance evaluation model as described in any one of claims 1 to 12, the age evaluation sub-model of the appearance evaluation model obtains an age weight based on the facial feature information; the gender evaluation sub-model of the appearance evaluation model obtains a gender weight based on the facial feature information; the appearance scoring sub-model of the appearance evaluation model obtains appearance evaluation information based on the facial feature information, the age weight and the gender weight.

14. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 13.

15. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images