A data processing method, information recommendation method and related device

Abstract

The present invention discloses a data processing method, an information recommendation method and related devices, and relates to the field of computer technology. A specific implementation of the method includes: locating the feature point position information in the face area according to the input face image data; calculating the distance feature vector of the face using the feature point position information, the distance feature vector is a vector composed of the distance features between the feature points; determining the label value of each organ through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value; using the distance feature vector and the black label value, according to the preset rules, the evaluation value corresponding to the face image data is calculated, and the evaluation value is used to evaluate the difference between the face image data and the standard face image data of the corresponding standard score. This implementation can score faces based on the knowledge related to face aesthetics and aesthetic standards, and can make targeted information recommendations based on face features, which is interesting and has a good user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device and an information recommendation method and device. Background Art

[0002] At present, various mobile applications based on face recognition technology are popular, and the main solutions are divided into two categories: one is to input video streams, recognize faces based on dynamics, locate facial key points, and apply dynamic stickers; the other is for users to upload photos or take photos, and perform face recognition and facial key point positioning based on static image data streams. The user's age, nationality, etc. can be determined based on algorithms, or users can drag facial key points to perform portrait processing.

[0003] The facial scoring system's functional points, such as age and nationality judgment, are too scattered and incomplete. Currently, mobile applications based on facial recognition are limited to portrait processing, without incorporating knowledge related to facial aesthetics, and lack a solution for recommending information based on facial features.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] Currently, there is a lack of face scoring schemes that incorporate knowledge related to facial aesthetics, and it is impossible to make targeted information recommendations based on facial features. Summary of the invention

[0006] In view of this, the embodiments of the present invention provide a data processing method, an information recommendation method and related devices, which can score faces based on relevant knowledge and aesthetic standards of facial aesthetics, and can make targeted information recommendations based on facial features, which is interesting and provides good user experience.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data processing method is provided.

[0008] A data processing method comprises: locating feature point position information in a face area according to input face image data; calculating a distance feature vector of the face using the feature point position information, wherein the distance feature vector is a vector composed of distance features between feature points; determining label values ​​of each organ through a deep learning network according to the distance feature vector, wherein negative label values ​​in the label values ​​are black label values; calculating an evaluation value corresponding to the face image data according to a preset rule using the distance feature vector and the black label value, wherein the evaluation value is used to evaluate the difference between the face image data and standard face image data having a corresponding standard score.

[0009] Optionally, the step of locating feature point position information in a face area based on input face image data includes: performing principal component analysis on the input face image to obtain a feature face image; locating the face area in the feature face image to obtain a face area image; and locating feature points on the face area image through a first cascade convolutional neural network to obtain feature point position information in the face area.

[0010] Optionally, the step of locating feature points of the face area image through a first cascade convolutional neural network to obtain feature point position information in the face area includes: inputting the face area image into the first layer network of the first cascade convolutional neural network to obtain a minimum bounding box image of the face; obtaining a preset number of feature points of the minimum bounding box image through the second layer network of the first cascade convolutional neural network; in the third layer network of the first cascade convolutional neural network, using the preset number of feature points to crop organs of the minimum bounding box image, and outputting the feature point position information in the face area.

[0011] Optionally, the deep learning network is a trained second cascade convolutional neural network, and the distance feature vector of the face and the evaluation value corresponding to the face image data are calculated by the second cascade convolutional neural network, wherein: the step of calculating the evaluation value corresponding to the face image data according to a preset rule using the distance feature vector and the black label value includes: adding the difference between each distance feature and the corresponding optimal distance according to the scoring weight of each distance feature to obtain a first deduction value; adding each black label value according to the corresponding organ black label weight to obtain a second deduction value; deducting the first deduction value and the second deduction value from the standard score to obtain the evaluation value corresponding to the face image data, and the scoring weight of each distance feature, the organ black label weight, and each optimal distance are obtained by training the second cascade convolutional neural network.

[0012] Optionally, the step of training the second cascade convolutional neural network includes: collecting sample facial image data required for training, and marking the sample facial image data, wherein the marking includes marking evaluation values ​​and marking black label values ​​for selected organs; and training the second cascade convolutional neural network using the marked sample facial image data.

[0013] According to another aspect of an embodiment of the present invention, a method for recommending information is provided.

[0014] A method for recommending information using the data processing results of the data processing method provided by the present invention, wherein the data processing results include the black label value, and the method includes: searching for recommended information using keywords matching the organs with black labels based on the matching relationship between facial organs and keywords, and outputting the searched recommended information.

[0015] According to yet another aspect of an embodiment of the present invention, a method for recommending information is provided.

[0016] A method for recommending information using the data processing results of the data processing method provided by the present invention, wherein the data processing results include evaluation values ​​corresponding to the facial image data, and wherein the method includes: based on the numerical range to which the evaluation values ​​corresponding to the facial image data belong, outputting recommendation information corresponding to the numerical range.

[0017] According to yet another aspect of an embodiment of the present invention, a data processing device is provided.

[0018] A data processing device comprises: a feature point positioning module, used to locate the feature point position information in the face area according to the input face image data; a distance feature vector calculation module, used to calculate the distance feature vector of the face by using the feature point position information, wherein the distance feature vector is a vector composed of the distance features between the feature points; a label distribution determination module, used to determine the label value of each organ through a deep learning network according to the distance feature vector, wherein the negative label value in the label value is a black label value; an evaluation value calculation module, used to calculate the evaluation value corresponding to the face image data according to a preset rule by using the distance feature vector and the black label value, wherein the evaluation value is used to evaluate the difference between the face image data and the standard face image data of the corresponding standard score.

[0019] According to yet another aspect of an embodiment of the present invention, a device for recommending information is provided.

[0020] A device for recommending information using the data processing results of the data processing method provided by the present invention, wherein the data processing results include the black label value, and the device includes: a first information recommendation module, which is used to search for recommended information using keywords matching the organs with black labels based on the matching relationship between facial organs and keywords, and output the searched recommended information.

[0021] According to yet another aspect of an embodiment of the present invention, a device for recommending information is provided.

[0022] A device for recommending information based on the data processing results of the data processing method provided by the present invention, wherein the data processing results include evaluation values ​​corresponding to the facial image data, and the device includes: a second information recommendation module for outputting recommendation information corresponding to the numerical interval to which the evaluation value corresponding to the facial image data belongs.

[0023] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0024] An electronic device comprises: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method or the method for recommending information provided by the present invention.

[0025] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0026] A computer readable medium stores a computer program, which, when executed by a processor, implements the data processing method or information recommendation method provided by the present invention.

[0027] An embodiment of the above invention has the following advantages or beneficial effects: locate the feature point position information in the face area according to the input face image data; calculate the distance feature vector of the face using the feature point position information; determine the label value of each organ through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value; use the distance feature vector and the black label value to calculate the evaluation value corresponding to the face image data according to the preset rules, and the evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score. It can score faces based on the knowledge and aesthetic standards related to face aesthetics, and can make targeted information recommendations based on face features, which is interesting and has a good user experience.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of main steps of a data processing method according to a first embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a data processing flow according to a second embodiment of the present invention;

[0032] Figure 3is a schematic diagram of an interface for marking sample face image data according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram showing various distance features of a face according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of main steps of an information recommendation method according to a third embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of main steps of an information recommendation method according to a fourth embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of a face scoring and information recommendation interface according to an embodiment of the present invention;

[0037] Figure 8 is an interactive schematic diagram of a system for face scoring and information recommendation according to a fifth embodiment of the present invention;

[0038] Fig. 9 is a schematic diagram of main modules of a data processing device according to a sixth embodiment of the present invention;

[0039] Fig.10 is a schematic diagram of main modules of an information recommendation device according to a seventh embodiment of the present invention;

[0040] Fig.11 is a schematic diagram of main modules of an information recommendation device according to an eighth embodiment of the present invention;

[0041] Fig.12 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0042] Fig.13 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0045] Figure 1 It is a schematic diagram of main steps of the data processing method according to the first embodiment of the present invention.

[0046] like Figure 1 As shown, the data processing method of the first embodiment of the present invention mainly includes the following steps S101 to S104.

[0047] Step S101: Locate feature point position information in the face area according to input face image data.

[0048] Specifically, principal component analysis is performed on the input face image to obtain a feature face image; the face area in the feature face image is located to obtain a face area image; feature points are located on the obtained face area image through a first cascade convolutional neural network to obtain feature point position information in the face area.

[0049] Among them, the step of locating feature points of the obtained face area image through the first cascade convolutional neural network to obtain the feature point position information in the face area can specifically include: inputting the face area image into the first layer of the first cascade convolutional neural network to obtain the minimum bounding box image of the face; obtaining a preset number of feature points of the minimum bounding box image through the second layer of the first cascade convolutional neural network; in the third layer of the first cascade convolutional neural network, using the preset number of feature points to crop the organs of the minimum bounding box image, and outputting the feature point position information in the face area. The above three layers of the first cascade convolutional neural network are all convolutional neural networks (CNN).

[0050] Step S102: Calculate the distance feature vector of the face using the feature point position information. The distance feature vector is a vector composed of distance features between feature points.

[0051] Step S103: According to the distance feature vector, the label value of each organ is determined through a deep learning network, and the negative label value in the label value is a black label value.

[0052] The deep learning network may be a trained second cascade convolutional neural network.

[0053] The label value distribution of each organ can be 1 or -1, where -1 represents the black label of the five senses (eyes, nose, mouth, skin, eyebrows). For example, assuming that the label of the eye is a black label, the label value is -1, indicating that the eye is an organ that reduces the evaluation value of the face image.

[0054] Step S104: using the distance feature vector and the black label value, the evaluation value corresponding to the face image data is calculated according to a preset rule.

[0055] The evaluation value is used to evaluate the difference between the face image data and the standard face image data corresponding to the standard score. The standard face image data is the face image data corresponding to the standard score, and the standard face image data can be used as a benchmark for the beauty of the face. Therefore, the evaluation value corresponding to the face image data can be used to evaluate the beauty of the face corresponding to a certain face image data.

[0056] The above steps S102, S103 and S104 can be performed by the trained second cascade convolutional neural network. The feature point position information is input into the first layer of the second cascade convolutional neural network to obtain the distance feature vector of the face; the distance feature vector of the face is input into the second layer of the second cascade convolutional neural network to obtain the label value distribution of each organ; the distance feature vector of the face and the label value distribution of each organ are input into the third layer of the second cascade convolutional neural network to obtain the evaluation value corresponding to the face image data. The above three layers of the second cascade convolutional neural network are all convolutional neural networks (CNN).

[0057] The step of training the second cascade convolutional neural network may include: collecting sample face image data required for training, and marking the sample face image data, the marking includes marking the evaluation value and marking the black label value of the selected organ; and using the marked sample face image data to train the second cascade convolutional neural network. For organs that are not marked with a black label value (-1), the default label value can be 1.

[0058] Step S104 specifically includes: adding the difference between each distance feature in the distance feature vector and the corresponding optimal distance according to the scoring weight of each distance feature to obtain a first deduction value; adding each black label value according to the corresponding organ black label weight to obtain a second deduction value; deducting the first deduction value and the second deduction value from the standard score to obtain an evaluation value corresponding to the facial image data.

[0059] The scoring weight of each distance feature, the organ black label weight, and each optimal distance are obtained by training the second cascade convolutional neural network. The optimal distance corresponding to the distance feature is the optimal distance between feature points.

[0060] Among them, the evaluation values ​​marked in the process of marking the sample facial image data can be combined to extract the common features of the distance features of the sample facial image data with the marked evaluation values ​​above the preset values ​​(for example, 90 points), and then the second cascade convolutional neural network is trained to obtain the optimal distance feature vector, which includes multiple optimal distances between feature points.

[0061] Figure 2 4 is a schematic diagram of a data processing flow according to a second embodiment of the present invention.

[0062] The data processing flow of the second embodiment of the present invention may include the following steps S201 to S206.

[0063] Step S201: Collect sample face image data required for training and label the sample face image data.

[0064] A given face image library can be used to collect sample face image data. Each face in the face image library corresponds to its own picture set, and a face picture set includes multiple pictures of the face. When labeling data, since the scoring of faces is subjective, the amount of sample data should be expanded as much as possible, and as many people as possible should be organized to label the sample face image data in the face image library.

[0065] The content of labeling specifically includes facial scoring and facial feature labeling. Facial scoring refers to facial beauty scoring, and the score range can be 0 to 100. When labeling facial features, black labels for facial features can be provided for labelers to choose from. The content of facial feature labeling includes marking black labels for eyes, nose, mouth, skin, and eyebrows. The black label value is a negative label value in the label value. For example, if the labeler believes that a certain organ of the sample face lowers the aesthetic score of the sample face, a black label value will be marked on the organ.

[0066] The interface diagram for labeling sample face image data can be as follows: Figure 3 In this interface, the labeler can input the face score (a value between 0 and 100) and select the organs that are considered to affect the score (eyes, nose, mouth, skin, eyebrows). The selected organs are marked with black labels, such as -1, and the labels of other organs that are not selected are marked with 1 by default.

[0067] Step S202: Perform principal component analysis (PCA) on the facial image of the user to be evaluated to obtain a characteristic face image.

[0068] Principal component analysis (PCA) can reduce the amount of data when processing face data and training CNN (convolutional neural network), thereby reducing the difficulty of calculation.

[0069] PCA processing includes the following steps: preprocessing of face images; reading in the face image library and training to form a feature subspace; projecting the training image and the test image onto the feature subspace; selecting a certain distance function for recognition and continuously adjusting the feature subspace to make the projection mean square error as small as possible.

[0070] The specific steps of PCA processing of the facial image of the user to be evaluated are as follows: vectorize each facial image in the facial image library to obtain a column vector corresponding to each facial image; average the column vectors of all facial images to obtain an average value vector; subtract each column vector from the average value vector to obtain the difference vector corresponding to each column vector, and all the difference vectors constitute a matrix A; calculate the covariance matrix ∑ of the matrix A; use the singular value decomposition method to calculate the eigenvectors and eigenvalues ​​of the covariance matrix ∑, where the eigenvectors corresponding to the largest k eigenvalues ​​constitute the eigenmatrix, which is the eigenspace, and the column vector (n-dimensional) of the facial image of the user to be evaluated is multiplied by the eigensubspace (eigenmatrix) (i.e., projected to the subspace) to obtain the k-dimensional column vector of the facial image of the user to be evaluated, and the facial image corresponding to the k-dimensional column vector is the eigenface image

[0071] Step S203: Locate the face region in the eigenface image to obtain a face region image.

[0072] The SVM (support vector machine) classifier can be used to locate the face area in the feature face image. The embodiment of the present invention uses cross-validation to determine the penalty parameter C of the SVM, and the kernel function gives priority to using the radial basis kernel function (RBF), constructs a multi-class SVM classifier, and in the multi-class SVM training stage, uses the sample to construct the SVM bipartitioner, and saves the training results (SVMStruct structure) of each SVM bipartitioner into a cell array CASVMStruct of a structure, and finally saves all the information required for multi-class SVM classification into the structure multiSVMStruct and returns; the multiSVMStruct training result is applied to the face sample for face recognition to locate the face area in the picture sample, and prepare for the work of locating the face feature points. Specifically, the multiSVMStruct training result can be used to confirm whether the sample has a face module, and locate the position of the face module in the picture, and the image of the face area is intercepted according to the located position. Through the above method, the face area in the feature face image of the user to be evaluated can be located, and the face area image can be obtained.

[0073] Step S204: locate feature points of the face area image of the user to be evaluated through the first cascade convolutional neural network to obtain feature point position information in the face area.

[0074] The first cascade convolutional neural network (DCNN) includes three levels of CNN (convolutional neural network), which are respectively recorded as level 1, level 2, and level 3. The feature point positioning mainly includes 51 facial feature points in the face contour. The network input and network output of each level of CNN are as follows:

[0075] Level 1:

[0076] Network input: face area image;

[0077] Network output: The minimum bounding box of 51 feature points, including the coordinates of the upper left corner and the lower right corner of the rectangle, is a 4-dimensional vector;

[0078] Level 2:

[0079] Network input: minimum bounding box image;

[0080] Network output: 51 feature points, corresponding to 102 neurons;

[0081] Level 3:

[0082] Network input: Use the 51 points of level 2 to crop the facial features and output the location information of the feature points in the face area. The cropped facial features need to be trained and predicted separately using the labeled sample face image data.

[0083] Network output: more detailed feature points of each organ; using the trained first cascade convolutional neural network, the feature point location information in the face area of ​​the user to be evaluated can be obtained.

[0084] Step S205: constructing a face scoring model using the second cascaded convolutional neural network, and training the face scoring model.

[0085] The face scoring model constructed using the second cascade convolutional neural network (DCNN) includes three levels of CNN (convolutional neural network), which are respectively recorded as the first level (level 1), the second level (level 2), and the third level (level 3). The network input and network output of each level of CNN are as follows:

[0086] Level 1:

[0087] Network input: location information of feature points in the face area;

[0088] Network output: distance feature vector of the face;

[0089] In this level of CNN, the seventeen dimensions required for beauty scoring are calculated based on the position information of the feature points in the face area, and a 17-column*1 distance feature vector corresponding to each sample face is obtained, of which 11 are horizontal distance features and 6 are vertical distance features. The distance feature vector is a vector composed of the distance features between feature points.

[0090] Distance feature, also known as feature quantity, is the distance between facial feature points of a person's face. The distance between facial feature points has an important influence on the beauty of a person's face. Figure 4 The figure is a schematic diagram showing the distance features of a face, which shows F1 to F17, a total of 17 distance features, which constitute the distance feature vector of the face. The detailed description of the 17 distance features is shown in Table 1. The parameters of the convolutional neural network are continuously trained using the feature point position information in the face area of ​​the sample face image data until a relatively accurate distance feature vector of the face is predicted.

[0091] Table 1

[0092]

[0093]

[0094] Level 2:

[0095] Network input: face distance feature vector, including 17 facial features;

[0096] Network output: The label value distribution of each organ is obtained based on the distance feature vector. The dimensions include eyes, nose, mouth, skin, and eyebrows. The label value distribution is 1 or -1, where -1 represents the black label value of the facial features.

[0097] The black label values ​​​​marked when data labeling sample face images with various distance feature vectors are used to continuously train the parameters of the convolutional neural network until a relatively accurate prediction of the label value distribution of each organ is obtained.

[0098] Level 3:

[0099] Network input: distance feature vector and black label value of the face;

[0100] Network output: facial beauty rating.

[0101] Among them, the distance feature vector of the face and the black labels of the facial features are directly used for separate training. Combined with the face score (i.e., beauty score) annotated in the process of labeling the sample face image data, the common features of the distance features are extracted for the sample face image data with a face score above 90 points. Then the network is trained to obtain the optimal distance feature vector, which includes 17 optimal distances (corresponding to F1 to F17 in Table 1).

[0102] When calculating the beauty score, the score weights corresponding to the 17 facial distance features are θ1…θ17, the difference between the 17 distance features and the optimal distance is β1…β17, and the weight of the facial features black label is φ1…φ5;

[0103] The deduction introduced by the difference between the 17 face distance features and the optimal distance is: θ1*β1+θ2*β2+…+θ17*β17;

[0104] The deduction introduced by the black labels of facial features is: φ1+φ2+…+φ5 (in this example, it is assumed that all facial features are marked with black labels, and the black labels are normalized to 1 point);

[0105] Using 100 as the standard score (i.e. full score), the above two items are deducted from the standard score to obtain the facial beauty score = 100-17 points deducted due to the difference between the facial distance features and the optimal distance-points deducted due to the black labels of the facial features.

[0106] During the training phase, the facial scores and annotated facial features black label values ​​of sample facial image data are used to train parameters such as the score weight of each distance feature and the organ black label weight.

[0107] Step S206: inputting the feature point position information in the face area of ​​the user to be evaluated into the trained face scoring model to output the face beauty score of the user.

[0108] The algorithm for calculating the user's facial beauty score has been introduced in detail in step S205 and will not be repeated here. The embodiment of the present invention uses the distance feature vector and the black label value of the face to calculate the facial beauty score, and can score the face based on the knowledge related to facial aesthetics and aesthetic standards, so that the scoring result conforms to the public aesthetics and the user experience is good.

[0109] Figure 5 FIG. 4 is a schematic diagram of main steps of an information recommendation method according to a third embodiment of the present invention.

[0110] The information recommendation method of the third embodiment of the present invention mainly includes the following steps S501 to S504.

[0111] Step S501: Locate feature point position information in the face area according to the input face image data.

[0112] Step S502: Calculate the distance feature vector of the face using the feature point position information. The distance feature vector is a vector composed of distance features between feature points.

[0113] Step S503: According to the distance feature vector, the label value of each organ is determined through a deep learning network, and the negative label value in the label value is a black label value.

[0114] Step S504: According to the matching relationship between the facial organs and the keywords, the recommended information is searched using the keywords matching the organs with black labels, and the searched recommended information is output.

[0115] The recommended information may be, for example, information about skin care, beauty products, etc. related to the organs with black labels, or advertising information displayed in the form of text, pictures, links, etc., or recommended information about skin care, beauty products, etc. based on the user's facial features.

[0116] Taking recommended product information as an example, the black label can correspond to the usage parts (part keywords) of beauty and personal care SKU (stock keeping unit) products, establish a mapping table, and query the recommended SKU products by looking up the table.

[0117] As an optional implementation, after step S503, the distance feature vector and the black label value may be used to calculate the evaluation value corresponding to the face image data according to a preset rule.

[0118] And, optionally, the above step S504 may be executed when the evaluation value corresponding to the face image data is greater than a preset threshold, and the above step S504 may not be executed if the evaluation value is less than or equal to the preset threshold.

[0119] The evaluation value corresponding to the facial image data can be used to evaluate the beauty of the face corresponding to the facial image data.

[0120] Figure 6 FIG. 4 is a schematic diagram of main steps of an information recommendation method according to a fourth embodiment of the present invention.

[0121] The information recommendation method of the fourth embodiment of the present invention mainly includes the following steps S601 to S605.

[0122] Step S601: Locate the feature point position information in the face area according to the input face image data;

[0123] Step S602: Calculate the distance feature vector of the face using the feature point position information, where the distance feature vector is a vector composed of distance features between feature points;

[0124] Step S603: Determine the label value of each organ through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value;

[0125] Step S604: using the distance feature vector and the black label value, calculating the evaluation value corresponding to the face image data according to a preset rule;

[0126] Step S605: According to the numerical range to which the evaluation value corresponding to the face image data belongs, output the recommendation information corresponding to the numerical range.

[0127] The input facial image data may be a user portrait to be scored for beauty.

[0128] The recommended information may be, for example, information about skin care, cosmetics, and other products related to the organs with black labels, or advertising information displayed in the form of text, pictures, links, and the like.

[0129] The evaluation value corresponding to the facial image data can be used to evaluate the beauty of the face corresponding to the facial image data.

[0130] Figure 7 2 is a schematic diagram of a face scoring and information recommendation interface according to an embodiment of the present invention. Taking the recommended product information as an example, according to the introduction of the above embodiments, the face image provided by the user can be scored, and beauty and skin care products can be recommended to the user based on the facial features black labels or the final scores, thereby achieving personalized recommendations.

[0131] The data processing method and the information recommendation method of the embodiments of the present invention can calculate the beauty score of the face based on the input face image data, and make personalized information recommendations based on user characteristics. It is highly interesting and has a good user experience, which can attract users. The obtained scoring results can be shared with other users or social platforms, thereby attracting more users and improving the order conversion rate of the product (or the access rate of information such as advertisements).

[0132] Figure 8 4 is an interactive diagram of a face scoring and information recommendation system according to a fifth embodiment of the present invention.

[0133] The face scoring and information recommendation system of the fifth embodiment of the present invention includes a client and a server.

[0134] Take photos of users in real time through the client, or read photos selected by users, and upload the photos of users to the server;

[0135] The server performs PCA dimensionality reduction on the user's photo, locates the face area, determines the location information of the facial feature points, and returns the facial feature point location information (coordinates) to the client;

[0136] The client displays the location information of facial feature points, calculates the facial beauty score based on the location information of facial feature points, and sends the facial beauty score to the server;

[0137] The server searches for keywords based on the facial beauty score and returns the searched recommendation information;

[0138] The client displays the facial beauty score and recommendation information through the user interface, and displays an entry for sharing the score so that users can share the facial beauty score.

[0139] As an exemplary application scenario, the system of the embodiment of the present invention can be implanted in the e-commerce platform system, and can collect user photos when the user logs in to the e-commerce platform system by scanning the face, without having to take photos of the user separately in real time, or read the photos selected by the user. In addition, the system provides users with the function of taking photos, uploading and scoring, and can call the background interface of the e-commerce platform system to process user image files, including locating the face area, locating the key points of the face, and returning detailed parameters such as the coordinates of the facial features and the scoring results through the interface. Through the embodiment of the present invention, users can receive highly personalized product recommendation services and enhance their sense of social interaction while feeling the fun of the system.

[0140] Fig. 9 4 is a schematic diagram of main modules of a data processing device according to a sixth embodiment of the present invention.

[0141] The data processing device 900 of the sixth embodiment of the present invention mainly includes: a feature point positioning module 901 , a distance feature vector calculation module 902 , a label distribution determination module 903 , and an evaluation value calculation module 904 .

[0142] The feature point positioning module 901 is used to locate the feature point position information in the face area according to the input face image data.

[0143] The distance feature vector calculation module 902 is used to calculate the distance feature vector of the face using the feature point position information. The distance feature vector is a vector composed of distance features between feature points.

[0144] The label distribution determination module 903 is used to determine the label value of each organ through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value.

[0145] The evaluation value calculation module 904 is used to calculate the evaluation value corresponding to the face image data according to preset rules using the distance feature vector and the black label value. The evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score.

[0146] Fig.10 4 is a schematic diagram of main modules of an information recommendation device according to a seventh embodiment of the present invention.

[0147] The information recommendation device 1000 of the seventh embodiment of the present invention mainly includes: a feature point positioning module 1001 , a distance feature vector calculation module 1002 , a label distribution determination module 1003 , and a first information recommendation module 1004 .

[0148] Among them, the feature point positioning module 1001, the distance feature vector calculation module 1002, and the label distribution determination module 1003 have the same functions as the feature point positioning module 901, the distance feature vector calculation module 902, and the label distribution determination module 903, respectively. Therefore, these three modules refer to the introduction of the feature point positioning module 901, the distance feature vector calculation module 902, and the label distribution determination module 903, and will not be repeated here.

[0149] The first information recommendation module 1004 is used to search for recommended information using keywords matching organs with black labels according to the matching relationship between facial organs and keywords, and output the searched recommended information.

[0150] As an optional implementation, the information recommendation device 1000 may further include an evaluation value calculation module 1005 , which has the same function as the evaluation value calculation module 904 , and thus will not be described again.

[0151] And, optionally, when the evaluation value corresponding to the facial image data obtained by the evaluation value calculation module 1005 is greater than a preset threshold, the corresponding function of the first information recommendation module 1004 can be executed; if the evaluation value is less than or equal to the preset threshold, the corresponding function of the first information recommendation module 1004 will not be executed.

[0152] Fig.11 FIG. 4 is a schematic diagram of main modules of an information recommendation device according to an eighth embodiment of the present invention.

[0153] The information recommendation device 1100 of the eighth embodiment of the present invention mainly includes: a feature point positioning module 1101 , a distance feature vector calculation module 1102 , a label distribution determination module 1103 , an evaluation value calculation module 1104 , and a second information recommendation module 1105 .

[0154] The feature point positioning module 1101, the distance feature vector calculation module 1102, the label distribution determination module 1103, and the evaluation value calculation module 1104 have the same functions as the feature point positioning module 901, the distance feature vector calculation module 902, the label distribution determination module 903, and the evaluation value calculation module 904, respectively. Therefore, these four modules can refer to the introduction of the feature point positioning module 901, the distance feature vector calculation module 902, the label distribution determination module 903, and the evaluation value calculation module 904, and will not be repeated here.

[0155] The second information recommendation module 1105 is used to output recommendation information corresponding to the numerical range according to the numerical range to which the evaluation value corresponding to the face image data belongs.

[0156] The specific implementation contents of the data processing device and the information recommendation device in the embodiments of the present invention have been described in detail in the above-mentioned data processing method and information recommendation method, so the repeated contents will not be described here.

[0157] Fig.12 An exemplary system architecture 1200 is shown to which the data processing method and information recommendation method or the data processing device and information recommendation device according to the embodiments of the present invention can be applied.

[0158] like Fig.12 As shown, the system architecture 1200 may include terminal devices 1201, 1202, 1203, a network 1204, and a server 1205. The network 1204 is used to provide a medium for communication links between the terminal devices 1201, 1202, 1203 and the server 1205. The network 1204 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0159] Users can use terminal devices 1201, 1202, and 1203 to interact with server 1205 through network 1204 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1201, 1202, and 1203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0160] The terminal devices 1201 , 1202 , and 1203 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0161] The server 1205 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 1201, 1202, and 1203. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0162] It should be noted that the data processing method and information recommendation method provided in the embodiments of the present invention can be executed by the server 1205 or the terminal devices 1201, 1202, 1203. Accordingly, the data processing device and the information recommendation device are generally arranged in the server 1205 or the terminal devices 1201, 1202, 1203.

[0163] It should be understood that Fig.12 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0164] Reference below Fig.13 , which shows a schematic diagram of the structure of a computer system 1300 of a terminal device or server suitable for implementing an embodiment of the present application. Fig.13 The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0165] like Fig.13 As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the system 1300 are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0166] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0167] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, the above-mentioned functions defined in the system of the present application are executed.

[0168] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0169] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a feature point positioning module, a distance feature vector calculation module, a label distribution determination module, and an evaluation value calculation module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the feature point positioning module may also be described as "a module for locating feature point position information in a face area based on input face image data".

[0171] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: locating the feature point position information in the face area according to the input face image data; calculating the distance feature vector of the face using the feature point position information, the distance feature vector is a vector composed of the distance features between the feature points; determining the label value of each organ through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value; using the distance feature vector and the black label value, calculating the evaluation value corresponding to the face image data according to the preset rules, the evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score.

[0172] According to the technical solution of the embodiment of the present invention, the feature point position information in the face area is located according to the input face image data; the distance feature vector of the face is calculated using the feature point position information; the label value of each organ is determined through a deep learning network according to the distance feature vector, and the negative label value in the label value is a black label value; the evaluation value corresponding to the face image data is calculated according to the preset rules using the distance feature vector and the black label value, and the evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score. The face can be scored based on the knowledge related to face aesthetics and aesthetic standards, and information can be recommended in a targeted manner according to the face features, which is interesting and has a good user experience.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, It is characterized in that include: Locate the feature point position information in the face area according to the input face image data; Calculating a distance feature vector of the face using the feature point position information, wherein the distance feature vector is a vector composed of distance features between feature points; According to the distance feature vector, a label value of each organ is determined through a deep learning network, wherein a negative label value in the label value is a black label value; an organ with a black label value is an organ that reduces the evaluation value of the face image; The evaluation value corresponding to the face image data is calculated according to a preset rule by using the distance feature vector and the black label value, and the evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score.

2. The method according to claim 1, It is characterized in that The step of locating the position information of feature points in the face area according to the input face image data comprises: Performing principal component analysis on the input face image to obtain a characteristic face image; Locating the face region in the eigenface image to obtain a face region image; The feature points of the face area image are located by a first cascade convolutional neural network to obtain the position information of the feature points in the face area.

3. The method according to claim 2, It is characterized in that The step of locating feature points of the face region image by a first cascade convolutional neural network to obtain feature point position information in the face region includes: Inputting the face area image into the first layer of the first cascade convolutional neural network to obtain a minimum bounding box image of the face; Obtaining a preset number of feature points of the minimum bounding box image through the second layer network of the first cascade convolutional neural network; In the third layer network of the first cascade convolutional neural network, the organs of the minimum bounding box image are cropped using the preset number of feature points, and the feature point position information in the face area is output.

4. The method according to claim 1, It is characterized in that The deep learning network is a trained second cascade convolutional neural network, and the distance feature vector of the face and the evaluation value corresponding to the face image data are calculated by the second cascade convolutional neural network, wherein: The step of calculating the evaluation value corresponding to the face image data according to a preset rule by using the distance feature vector and the black label value comprises: The difference between each distance feature and the corresponding optimal distance is added according to the scoring weight of each distance feature to obtain the first deduction value; Add up the black label values ​​according to the corresponding organ black label weights to get the second deduction value; The first deduction value and the second deduction value are deducted from the standard score to obtain an evaluation value corresponding to the facial image data, and the scoring weight of each distance feature, the organ black label weight, and each optimal distance are obtained by training the second cascade convolutional neural network.

5. The method according to claim 4, It is characterized in that The step of training the second cascade convolutional neural network comprises: Collecting sample face image data required for training, and marking the sample face image data, wherein the marking includes marking evaluation values ​​and marking black label values ​​for selected organs; The second cascade convolutional neural network is trained using the labeled sample face image data.

6. A method for using the black label value recommendation information according to any one of claims 1 to 5, It is characterized in that The method comprises: According to the matching relationship between the facial organs and the keywords, the recommended information is searched using the keywords matching the organs with black labels, and the searched recommended information is output.

7. A method for recommending information using evaluation values ​​corresponding to face image data according to any one of claims 1 to 5, It is characterized in that The method comprises: According to the numerical range to which the evaluation value corresponding to the face image data belongs, recommendation information corresponding to the numerical range is output.

8. A data processing device, It is characterized in that include: A feature point positioning module is used to locate the feature point position information in the face area according to the input face image data; A distance feature vector calculation module, used to calculate the distance feature vector of the face using the feature point position information, wherein the distance feature vector is a vector composed of distance features between feature points; A label distribution determination module is used to determine the label value of each organ through a deep learning network according to the distance feature vector, wherein a negative label value in the label value is a black label value; an organ with a black label value is an organ that reduces the evaluation value of the face image; An evaluation value calculation module is used to use the distance feature vector and the black label value to calculate the evaluation value corresponding to the face image data according to a preset rule, and the evaluation value is used to evaluate the difference between the face image data and the standard face image data with the corresponding standard score.

9. A device for using the black label value recommendation information according to any one of claims 1 to 5, It is characterized in that The device comprises: The first information recommendation module is used to search for recommended information using keywords matching organs with black labels according to the matching relationship between facial organs and keywords, and output the searched recommended information.

10. A device for recommending information using evaluation values ​​corresponding to face image data according to any one of claims 1 to 5, It is characterized in that The device comprises: The second information recommendation module is used to output recommendation information corresponding to the numerical interval according to the numerical interval to which the evaluation value corresponding to the face image data belongs.

11. An electronic device, It is characterized in that include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

12. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images