Data processing method and electronic equipment

Abstract

The invention provides a data processing method and electronic equipment, and the method comprises the steps: obtaining feature vectors of facial feature points under all expressions based on facial image data under all expressions; comparing the feature vector under each expression with the feature vector of the basic expression to obtain first feature variations of the facial feature points under different expressions; performing dimension reduction processing on the first feature variable quantity to obtain a second feature variable quantity; based on the time sequence of the facial image data, setting a label of a second feature variable quantity to obtain a third feature variable quantity; the label is used for describing expression features of the second feature variation; calculating a semantic similarity matrix between the third feature variations; determining a semantic weight of the third feature variation based on the semantic similarity matrix to obtain a fourth feature variation; wherein the semantic similarity between the third feature variations is in positive correlation with the corresponding semantic weight; and decoding the fourth characteristic variable quantity to obtain a target characteristic variable quantity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a data processing method and an electronic device. BACKGROUND

[0002] There is a need for facial expression capture in many fields such as virtual reality, augmented reality, animation production, and human-computer interaction. BlendShape, a data form widely used in facial expression modeling, describes different expressions by defining a series of changes in facial shapes. Using BlendShape can quickly produce high-quality facial animation, reduce manual adjustment time, and improve production efficiency.

[0003] In related technologies, professional facial motion capture equipment is needed to obtain BlendShape data. Marking points (such as reflective balls or special patches) are pasted on the face to capture facial movements, and a camera tracks the dynamic positions of these marking points to reconstruct the facial shape. The marking points require high precision and stability for fitting, making the way of obtaining BlendShape data complex. SUMMARY

[0004] The present application provides a data processing method and an electronic device to at least solve the above technical problems in the prior art.

[0005] According to a first aspect of the present application, a data processing method is provided, comprising:

[0006] Based on facial image data under each expression, a feature vector of facial feature points under each expression is obtained; the facial feature points are obtained by marking the face according to a preset marking rule;

[0007] The feature vectors under each expression are compared with the feature vector of the base expression to obtain a first feature change amount of the facial feature points under different expressions;

[0008] The first feature change amount is processed by dimension reduction to obtain a second feature change amount;

[0009] Based on the time sequence of the facial image data, a label of the second feature change amount is set to obtain a third feature change amount; the label is used to describe the expression feature of the second feature change amount;

[0010] A semantic similarity matrix between the third feature change amounts is calculated;

[0011] A semantic weight of the third feature change amount is determined based on the semantic similarity matrix to obtain a fourth feature change amount; wherein the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weight;

[0012] decode the fourth feature variation to obtain a target feature variation.

[0013] In an implementation, before the feature vector of the facial feature point under each expression is obtained based on the facial image data under each expression, the method further comprises:

[0014] display prompt information representing guiding the user to adjust the facial position, the facial angle and make corresponding facial actions;

[0015] obtain the facial image data under each expression based on a time sequence; the time sequence corresponds to the facial image data under each expression.

[0016] In an implementation, the first feature variation of the facial feature point under each expression is obtained by comparing the feature vector under each expression with the feature vector of the basic expression, comprising:

[0017] obtain the first coordinate set of each facial feature point in the basic expression based on the feature vector of the basic expression;

[0018] obtain the second coordinate set of each facial feature point in each expression based on the feature vector under each expression;

[0019] calculate the first feature variation of each expression relative to the basic expression based on the first coordinate set and the second coordinate set.

[0020] In an implementation, the second feature variation is obtained by performing dimension reduction processing on the first feature variation, comprising:

[0021] perform centering processing on the first feature variation to obtain a first vector;

[0022] construct a first vector matrix based on the first vector;

[0023] calculate the covariance matrix of the first vector matrix;

[0024] obtain the eigenvalues and eigenvectors of the covariance matrix;

[0025] sort the eigenvectors in the front according to a preset threshold value based on the principal component analysis method to obtain an eigenvector matrix; wherein, the preset threshold value represents the proportion of the sum of the eigenvalues corresponding to the preset number of eigenvectors in the sum of the total eigenvalues;

[0026] project the first vector matrix onto the eigenvector matrix to obtain the second feature variation.

[0027] In an implementation, the label of the second feature variation is set based on the time sequence of the face image data, to obtain a third feature variation, including:

[0028] The label of the second feature variation is set based on the time sequence and the corresponding image frame in the face image data; the label at least includes a static label representing an expression and a dynamic label representing a transition expression;

[0029] The third feature variation is obtained based on the second feature variation after setting the label.

[0030] In an implementation, the semantic similarity matrix between the third feature variations is calculated, including:

[0031] According to a node rule, a label semantic tree is constructed based on the third feature variations; wherein the node rule represents the hierarchical relationship between the third feature variations; the node rule is used to construct the topological structure, nodes and the correlation between nodes of the label semantic tree;

[0032] The semantic similarity matrix between the third feature variations is obtained based on the label semantic tree.

[0033] In an implementation, the hierarchical relationship of the third feature variations includes a facial motion type, a single feature variation and a composite feature variation; the semantic range of the facial motion type is higher than that of the single feature variation, and the semantic range of the single feature variation is higher than that of the composite feature variation; the label semantic tree is constructed based on the third feature variations according to the node rule, including:

[0034] The label semantic tree is constructed with the facial motion type as the root node, the single feature variation as the intermediate node and the composite feature variation as the leaf node; the label semantic tree includes multiple root nodes, multiple intermediate nodes and multiple leaf nodes.

[0035] In an implementation, the semantic weight of the third feature variation is determined based on the semantic similarity matrix, to obtain a fourth feature variation, including:

[0036] Based on the semantic similarity matrix, the third feature variations are divided into a positive sample set and a negative sample set; the positive sample set includes multiple positive sample pairs, and the negative sample set includes multiple negative sample pairs;

[0037] The first similarity between the positive sample pairs and the second similarity between the negative sample pairs are calculated;

[0038] Based on the first similarity and the second similarity, a loss function comparison loss is obtained.

[0039] Assign a semantic weight to the positive sample pair based on the loss, to obtain a fourth feature change amount; the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weight.

[0040] In an implementable manner, the decoding of the fourth feature change amount to obtain a target feature change amount comprises:

[0041] The hierarchical decoder is used to obtain the weight of each layer of the label semantic tree, and determine a structure loss;

[0042] The weight change of adjacent frame images is calculated to obtain a temporal consistency loss;

[0043] The fourth feature change amount is optimized in combination with the structure loss and the temporal consistency loss, to obtain a target feature change amount.

[0044] According to a second aspect of the present application, an electronic device is provided, comprising:

[0045] at least one processor;

[0046] and a memory in communication connection with the at least one processor; wherein

[0047] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute

[0048] Based on the facial image data under each expression, a feature vector of facial feature points under each expression is obtained; the facial feature points are obtained by marking the face according to a preset marking rule;

[0049] The feature vectors under each expression are compared with the feature vector of the basic expression to obtain a first feature change amount of facial feature points under different expressions;

[0050] The first feature change amount is processed by dimension reduction to obtain a second feature change amount;

[0051] Based on the time sequence of the facial image data, a label of the second feature change amount is set to obtain a third feature change amount; the label is used to describe the expression feature of the second feature change amount;

[0052] The semantic similarity matrix between the third feature change amounts is calculated;

[0053] Based on the semantic similarity matrix, a semantic weight of the third feature change amount is determined to obtain a fourth feature change amount; wherein the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weight;

[0054] The fourth feature variation quantity is decoded to obtain a target feature variation quantity.

[0055] According to a third aspect of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to cause the computer to execute

[0056] Based on the facial image data under each expression, a feature vector of a facial feature point under each expression is obtained; the facial feature point is obtained by marking a face according to a preset marking rule;

[0057] The feature vectors under each expression are compared with the feature vector of the basic expression to obtain a first feature variation quantity of the facial feature point under different expressions;

[0058] The first feature variation quantity is processed by dimension reduction to obtain a second feature variation quantity;

[0059] Based on a time sequence of the facial image data, a label of the second feature variation quantity is set to obtain a third feature variation quantity; the label is used to describe an expression feature of the second feature variation quantity;

[0060] A semantic similarity matrix between the third feature variation quantities is calculated;

[0061] A semantic weight of the third feature variation quantity is determined based on the semantic similarity matrix to obtain a fourth feature variation quantity; wherein the semantic similarity between the third feature variation quantities is positively correlated with the corresponding semantic weight;

[0062] The fourth feature variation quantity is decoded to obtain a target feature variation quantity.

[0063] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0064] The above and other objects, features and advantages of the example embodiments of the present application will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0065] In the drawings, identical or corresponding reference numerals indicate identical or corresponding parts.

[0066] Figure 1 An implementation flowchart of a data processing method according to an embodiment of the present application is shown;

[0067] Figure 2Another implementation flow diagram of the data processing method of the embodiment of the application is shown.

[0068] Figure 3 A structural diagram of the data processing device of the embodiment of the application is shown.

[0069] Figure 4 A structural diagram of an electronic device of the embodiment of the application is shown. DETAILED DESCRIPTION

[0070] To make the purposes, features and advantages of the application more obvious and easy to understand, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0071] A data processing method and an electronic device provided by the application will be described below with reference to the drawings.

[0072] As shown in Figure 1 The application provides a data processing method, which comprises:

[0073] S101, obtaining a feature vector of a facial feature point under each expression based on facial image data under each expression; the facial feature point is obtained by marking a face according to a preset marking rule;

[0074] The data processing method provided by the application can be implemented by an electronic device. The electronic device can be a smart terminal or a server; the smart terminal can be a mobile phone, a personal digital assistant (PDA), a tablet computer, a notebook computer, a desktop computer, etc. The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0075] In the present application, each expression can be, for example, joy, sadness, anger, fear, surprise, etc. The preset marking rule can be a dot marking method. By marking the feature points of the face through the dot marking method, the feature point positions can be, for example, the eye part (the upper eyelid edge, the lower eyelid edge, the pupil center or the iris edge, etc.), the eyebrow part (the inner end, the highest point, the outer end, etc.), the mouth part (the upper lip upper edge, the lower lip lower edge, the corner of the mouth, the upper and lower lip junction line, etc.), the nose part (the nose tip, the two sides of the nose wing, the nose root, etc.), the chin tip, etc. The face feature points are obtained by marking the feature point positions.

[0076] The face image data of the same person when making different expressions is collected by the image collection device. Then, the feature vectors of the face feature points under each expression are obtained according to the face image data under each expression in which the face feature points are marked.

[0077] In the present application, the image data collected by the image collection device is three-dimensional model data of the face, which is stored in the form of point cloud. After obtaining the three-dimensional model data, the three-dimensional model data can also be preprocessed in the present application. The preprocessing includes removing outliers, smoothing filtering, etc., to improve the quality of the face image data.

[0078] S102, comparing the feature vectors under each expression with the feature vector of the basic expression to obtain the first feature change amount of the face feature points under different expressions;

[0079] In the present application, before comparing the feature vectors under each expression with the feature vector of the basic expression, the face image data under different expression states and the face image data of the basic expression (neutral expression or reference state face data) are aligned and matched in space through the marked face feature points, to ensure that all face image data maintains consistent position, scale and structural relationship under the same spatial coordinate system. Thus, the facial morphology difference caused by expression changes (such as smiling, frowning, surprise, etc.) is eliminated, and the data of different expressions can be compared and analyzed under a unified spatial framework.

[0080] In the present application, the difference between the feature vectors of the face feature points under different expressions and the basic state (usually the neutral expression) is compared to quantify and extract the expression features, i.e. the first feature change amount. For example, when smiling, people usually "lift the corners of the mouth" and "wink the corners of the eyes", and in the corresponding difference vector, the y coordinate (vertical direction) of the corner of the mouth changes greatly in the positive direction, and the x coordinate (horizontal direction) of the corner of the eye changes greatly in the negative direction (the distance is reduced). Or when surprised, people usually "raise the eyebrows" and "open the mouth", and the y coordinate of the eyebrow point changes greatly in the positive direction, and the z coordinate (depth direction) of the lip point changes significantly. Therefore, the first feature change amount obtained by the coordinate difference can directly quantify the perception change of human vision.

[0081] S103, dimensionality reduction processing is performed on the first feature change amount to obtain a second feature change amount;

[0082] In this application, principal component analysis is used to perform dimensionality reduction processing on the first feature change amount to obtain the second feature change amount. The principal component analysis is a data dimensionality reduction technique that converts the original data to a new coordinate system through linear transformation, so that the variance of the data in the new coordinate system is concentrated in as few dimensions as possible, thereby reducing the data dimension while preserving the main features of the data as much as possible, achieving the purpose of optimizing and compressing the data.

[0083] S104, based on the time sequence of the face image data, the label of the second feature change amount is set to obtain a third feature change amount; the label is used to describe the expression feature of the second feature change amount;

[0084] It should be noted that when the image acquisition device collects face image data under each expression, the face image data also includes a time sequence. For example, the time sequence for a smiling expression, and the transition time for a smiling to non-smiling expression. Based on the time sequence of the face image data corresponding to different expressions, the label of the second feature change amount is set. The label can be an expression action such as frowning, opening eyes, etc., and can also be an action filter from frowning to relaxing. After labeling the second feature change amount, the third feature change amount is obtained.

[0085] S105, calculating the semantic similarity matrix between the third feature change amounts;

[0086] According to the label between the third feature change amounts, such as smiling, laughing, frowning, anger, or anger, etc. This step can input the third feature change amount into a pre-trained language model, which can calculate the similarity through the label contained in the third feature change amount to construct a similarity matrix. The similarity matrix represents the semantic similarity between the third feature change amounts. In this application, the similarity matrix can be calculated using attribute matching degree calculation, or cosine similarity calculation, which is not limited in this application.

[0087] S106, determining the semantic weight of the third feature change amount based on the semantic similarity matrix to obtain a fourth feature change amount; wherein the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weight;

[0088] In this application, according to the third feature change amount and the corresponding semantic similarity matrix, the semantic weight of each third feature change amount is calculated, and the third feature change amount and the corresponding semantic weight are taken as the fourth feature change amount.

[0089] Wherein, the semantic similarity is positively correlated with the semantic weight, that is, the higher the semantic similarity between the third feature change amounts, the higher the semantic weight, the lower the semantic similarity, the lower the semantic weight.

[0090] S107, the fourth feature change amount is decoded to obtain a target feature change amount.

[0091] It can be understood that the fourth feature change amount is a feature vector after dimension reduction, and therefore, the feature vector after dimension reduction needs to be mapped back to a high-dimensional space. The present application decodes the fourth feature change amount by using a hierarchical decoder to obtain a target feature change amount.

[0092] The data processing method of the present application obtains the first feature change amount by comparing each expression relative to the basic face, clusters the same type of expressions for the deformation data, and simultaneously uses label constraint semantics weight to further optimize the first feature change amount, and finally obtains the target feature change amount. The face image acquisition operation of the present application is simple, and does not require professional equipment operation skills and complex environment construction. It can quickly batch collect feature change amounts of multiple expressions, and improves the efficiency of data collection.

[0093] In some embodiments, as shown in Figure 2 Before obtaining the feature vector of the facial feature points under each expression based on the facial image data under each expression, the method further includes:

[0094] displaying prompt information representing guiding the user to adjust the facial position, facial angle, and make corresponding expression actions;

[0095] obtaining facial image data under each expression based on a time sequence; the time sequence corresponds to the facial image data under each expression.

[0096] When the image acquisition device collects images, it can display prompt information on the display interface. For example, the prompt information can be "Please align the face to the center of the screen frame, make sure the forehead and chin are in the frame", "Please relax your face and make a natural neutral expression", "Please smile: smile and show your teeth", "Please make a surprised expression: raise your eyebrows, open your eyes wide, and slightly open your mouth", etc.

[0097] The user makes corresponding expression actions according to the prompt information, and the image acquisition device collects corresponding images within a specified time, so that the obtained facial image data has a corresponding time sequence. For example, the preset expression type is arranged in order, such as "neutral to smile to anger to surprise to sadness", and each expression corresponds to a time segment. For example, set 1-3 seconds as smile, 3-5 seconds as laughter, etc.

[0098] The application ensures the standardization of the collected facial image data in position and angle through prompt information, and reduces the difficulty of subsequent data preprocessing. The facial image data of the application has a corresponding relationship with the time sequence and expression data, which provides a time dimension basis for analyzing the dynamic changes of expressions (such as the muscle movement process from calm to smile), and facilitates the subsequent setting of labels.

[0099] In some embodiments, the feature vectors under each expression are compared with the feature vector of the basic expression to obtain the first feature change amount of the facial feature points under different expressions, including:

[0100] Based on the feature vector of the basic expression, a first coordinate group of each facial feature point in the basic expression is obtained;

[0101] Based on the feature vector under each expression, a second coordinate group of each facial feature point in each expression is obtained;

[0102] Based on the first coordinate group and the second coordinate group, the first feature change amount of each expression relative to the basic expression is calculated.

[0103] In the application, the feature vector of the basic expression and the feature vector under each expression can be an ordered representation of the coordinate group. In the application, the coordinates can be directly extracted by analyzing the feature vector. According to the time sequence, the coordinate group of the feature vector corresponding to each expression is extracted. Through the difference between the second coordinate group under each expression and the first coordinate group under the basic expression, the first feature change amount of each expression relative to the basic expression is obtained.

[0104] Exemplarily, it is assumed that the facial image data is a set containing n three-dimensional points. Let the feature vector S of the facial feature points of the basic expression be base composed of n three-dimensional point coordinates (x base1 ,y base1 ,z base1 ), (x base2 ,y base2 ,z base2 ), ···, (x basen ,y basen ,z basen ). The feature vector S i of the facial feature points of the i-th expression is composed of n three-dimensional point coordinates (x i1 ,y i1 ,z i1 ), (x i2 ,y i2 ,z i2 ), ···, (x in ,y in ,z in ).

[0105] For the jth point (1≤j≤n) of the ith expression, calculate the first feature change D i The coordinates of the corresponding points in the middle are:

[0106] x Dj =x ij -x basej

[0107] y Dj =y ij -y basej

[0108] z Dj =z ij -z basej

[0109] Thus, D i is composed of n three-dimensional point coordinates (x D1 , y D1 , z D1 ), (x D2 , y D2 , z D2 ), …, (x Dn , y Dn , z Dn ), that is, the first feature change.

[0110] In some embodiments, the first feature change is dimensionally reduced to obtain a second feature change, including:

[0111] The first feature change is centered to obtain a first vector;

[0112] Based on the first vector, a first vector matrix is constructed;

[0113] The covariance matrix of the first vector matrix is calculated;

[0114] The eigenvalues and eigenvectors of the covariance matrix are obtained;

[0115] Based on the principal component analysis method, a preset number of eigenvectors in the front are screened according to a preset threshold value to obtain an eigenvector matrix; wherein the preset threshold value represents the proportion of the sum of the eigenvalues corresponding to the preset number of eigenvectors in the sum of the total eigenvalues;

[0116] The first vector matrix is projected onto the eigenvector matrix to obtain the second feature change.

[0117] In this application, first, the first feature change is centered to obtain a first vector. Then, based on the first vector, a covariance matrix is constructed, and the eigenvalues and eigenvectors of the covariance matrix are obtained. Finally, through the principal component analysis method, a preset number of eigenvectors in the front are screened according to a preset threshold value to obtain an eigenvector matrix.

[0118] Exemplarily, first, all point coordinates in the first feature variation D i are subjected to centering processing. Assuming that the average value of all x coordinates in D i is the average value of y coordinates is and the average value of z coordinates is for each point (x i , y Dj , z Dj ) in D Dj , the centered coordinates are:

[0119]

[0120] By centering the first feature variation D i , a first vector is obtained. The average value of the first vector is zero, so that in subsequent calculation of the covariance matrix, the relative relationship between the facial feature points can be better reflected.

[0121] Then, the first vector is combined into a first vector matrix M. The covariance matrix C of the first vector matrix M is calculated in the following manner,

[0122]

[0123] The covariance matrix can describe the correlation between the first vector in each dimension, and by calculating the covariance matrix, the variation of the first vector in different dimensions and the mutual relationship between the dimensions can be understood.

[0124] Eigenvalue decomposition is performed on the covariance matrix C to obtain eigenvalues λ1, λ2, λ3 and corresponding eigenvectors e1, e2, e3. The eigenvalue represents the variance of the data in the direction of the corresponding eigenvector, and the greater the variance, the greater the change of the data in that direction. According to the order from large to small, the greater the eigenvalue, the more significant the data change in the direction of the corresponding eigenvector. Therefore, according to a threshold (for example, retaining 95% of the data variance), the first k eigenvectors (k≤3) are selected, so that the sum of the eigenvalues corresponding to these eigenvectors accounts for a proportion of the sum of the total eigenvalues reaching the set threshold.

[0125] Assuming that the first k eigenvectors are selected to form an eigenvector matrix E, by selecting the principal components, dimensions that contribute less to the data variance can be discarded, thereby realizing data dimensionality reduction while retaining the main features of the data. Finally, the centered data M is projected onto the selected eigenvector matrix E to obtain the compressed data After the projection operation, the original n x 3 data matrix M is converted into an n x k matrix The data volume is significantly reduced. Since the projection is based on the main change direction of the data (determined by the eigenvector), the main features of the original data are retained as much as possible while reducing the data volume.

[0126] In some embodiments, based on the time sequence of the facial image data, the label of the second feature change amount is set, and the third feature change amount is obtained, including:

[0127] Based on the time sequence in the facial image data and the corresponding image frame, the label of the second feature change amount is set. The label includes at least a static label representing an expression and a dynamic label representing a transition expression.

[0128] Based on the second feature change amount after setting the label, the third feature change amount is obtained.

[0129] In this application, after obtaining the second feature change amount, an artificial semantic label is added to the second feature change amount. Among them, the static label describes a specific expression state, and the dynamic label describes the expression transition process. The feature change amount optimized by the label constraint is the third feature change amount. The third feature change amount is consistent with the label (static state / dynamic transition) while retaining the dynamic information of the image. It can be understood that the third feature change amount is bound with the label and has semantic information, which can more accurately express the semantics.

[0130] Among them, the static label describes the feature change of the "stable expression state". For example, "smile", "frown", "neutral" and the like. The dynamic label describes the feature change of the "expression transition process". For example, "neutral to smile", "frown to relax", "smile to laugh" and the like.

[0131] In some embodiments, the semantic similarity matrix between the third feature change amounts is calculated, including:

[0132] According to the node rule, a label semantic tree is constructed based on the third feature change amount. Among them, the node rule represents the hierarchical relationship between the third feature change amounts. The node rule is used to construct the topological structure, nodes and the association relationship between nodes of the label semantic tree.

[0133] Based on the label semantic tree, the semantic similarity matrix between the third feature change amounts is obtained.

[0134] In the present application, the node rule is used to guide the construction of the topology structure, nodes and the association relationship between nodes of the label semantic tree. According to the node rule, the third feature variation quantity that has been associated with the expression semantic is presented in a tree structure to show the semantic level (root node, intermediate node and leaf node) of the feature variation. Each node corresponds to a type of third feature variation quantity. The semantic similarity matrix is calculated according to the level, node path distance and the like between the nodes of the label semantic tree. The semantic similarity matrix quantifies the association degree of any two third feature variation quantities on the semantic tree.

[0135] In the present application, the third feature variation quantity is organized into a label semantic tree according to the level, the hierarchical relationship is clarified, and the semantic relationship is refined. The numerical value of the semantic similarity matrix can directly correspond to the positional relationship on the label semantic tree. For example, the “smile-related features” are naturally clustered and are far away from the “frown features”. Through the label semantic tree and the semantic similarity matrix, the semantic relationship of the third feature variation quantity is correspondingly displayed, so that the semantic relationship between the third feature variation quantities is more clear, and the accuracy of expression analysis is improved.

[0136] In some embodiments, the hierarchical relationship of the third feature variation quantity includes a facial movement type, a single feature variation quantity and a composite feature variation quantity; the semantic range of the facial movement type is higher than that of the single feature variation quantity, and the semantic range of the single feature variation quantity is higher than that of the composite feature variation quantity; according to the node rule, the label semantic tree is constructed based on the third feature variation quantity, including:

[0137] The label semantic tree is constructed with the facial movement type as the root node, the single feature variation quantity as the intermediate node and the composite feature variation quantity as the leaf node; the label semantic tree includes multiple root nodes, multiple intermediate nodes and multiple leaf nodes.

[0138] In the present application, the hierarchical relationship of the third feature variation quantity includes a facial movement type (such as “expression”, “speaking” and “micro-movement”), a single feature variation quantity (such as “smile” and “frown”) and a composite feature variation quantity (such as “one-sided smile + raised eyebrows”). The nodes of the label semantic tree include root nodes, intermediate nodes and leaf nodes. According to the hierarchical relationship, the label semantic tree is constructed with the facial movement type as the root node, the single feature variation quantity as the intermediate node and the composite feature variation quantity as the leaf node.

[0139] In some embodiments, the semantic weight of the third feature variation quantity is determined based on the semantic similarity matrix to obtain a fourth feature variation quantity, including:

[0140] Based on the semantic similarity matrix, the third feature variation quantity is divided into a positive sample set and a negative sample set; the positive sample set includes multiple positive sample pairs, and the negative sample set includes multiple negative sample pairs;

[0141] calculate a first similarity between positive sample pairs and a second similarity between negative sample pairs;

[0142] based on the first similarity and the second similarity, a contrastive loss through a loss function;

[0143] based on the loss, assign a semantic weight to the positive sample pairs to obtain a fourth feature variation; the semantic similarity between the third feature variations is positively correlated with the corresponding semantic weight.

[0144] Specifically, in the present application, based on the semantic similarity matrix, a similarity threshold is set, and the third feature variation is divided into a positive sample set and a negative sample set through the similarity threshold. For example, the similarity threshold of "smile mouth corner amplitude" and "smile apple muscle bulge" is 0.85, and the similarity threshold of "neutral to smile rate" and "smile mouth corner amplitude" is 0.65, both of which are positive samples. And the similarity threshold of "smile mouth corner amplitude" and "frown between the eyebrows" is 0.3, which is a negative sample.

[0145] The present application can use cosine similarity to calculate the similarity of positive sample pairs in the positive sample set and negative sample pairs in the negative sample set. The feature distance of the positive sample pairs is narrowed and the distance of the negative sample pairs is pushed away through the loss function, while the semantic similarity is introduced as a weight. The loss function is

[0146]

[0147] where f(x) is the feature mapping (vector after dimension reduction) of the three-dimensional point cloud, τ is the temperature coefficient, and the semantic weight is added for correction: for positive sample pairs According to the weight assigned by the semantic similarity matrix (the higher the similarity, the greater the weight), the semantic constraint is strengthened.

[0148] In the present application, the third feature variation is weighted and adjusted based on the semantic similarity matrix and the gradient of the contrastive loss, so that the fourth feature variation not only retains the dynamic information of the third feature variation, but also forms a clustering structure consistent with the semantic label in the feature space due to the introduction of the semantic weight.

[0149] In some embodiments, the fourth feature variation is decoded to obtain a target feature variation, including:

[0150] The hierarchical decoder is used to obtain the weight of each layer of the label semantic tree to determine the structure loss;

[0151] The weight change of adjacent frame images is calculated to obtain a temporal consistency loss;

[0152] The fourth feature variation is optimized to obtain the target feature variation in combination with the structure loss and the temporal consistency loss.

[0153] In this application, a hierarchical decoder is designed corresponding to the label semantic tree. Each layer of the label semantic tree (root node, intermediate node, leaf node) corresponds to the output of the weight of the corresponding node. For example, the top layer weight: the activation weight of the root node and the first level node (such as "facial movement type" in "expression" accounts for 0.8, "speaking" accounts for 0.2). The middle layer weight: the weight of the intermediate node (such as "smile" weight 0.6, "open eyes" weight 0.3). The bottom layer weight: the correction weight of the leaf node (composite / individualized action) (such as "one-sided smile" offset -0.1). Further output the weight vector, each sub-vector corresponds to the weight distribution of a layer.

[0154] According to the weight change of the adjacent frame image, the time sequence consistency loss is obtained,

[0155]

[0156] Among them, is the time sequence consistency loss, W t is the weight vector of the fourth feature change amount of the t-th frame, W t+1 is the weight vector of the fourth feature change amount of the t-th frame.

[0157] The total loss function is

[0158]

[0159] Among them, Ltotal is the total loss, and a, b, and g are balance coefficients.

[0160] The application combines and , while minimizing the feature reconstruction loss (the error between the reconstructed three-dimensional point cloud and the original data), to optimize the fourth feature change amount.

[0161] The application constrains the semantic hierarchical relationship through the structure loss, guarantees the motion continuity through the time sequence loss, and balances the fidelity by combining the feature reconstruction loss. Finally, the target feature change amount is the optimal solution of "semantic logic combined with time sequence law combined with original information".

[0162] As Figure 3 shown, the application provides a data processing device, comprising:

[0163] The acquisition module 301 is configured to obtain the feature vector of the facial feature point under each expression based on the facial image data under each expression; the facial feature point is obtained by marking the face according to a preset marking rule;

[0164] The comparison module 302 is configured to compare the feature vectors under each expression with the feature vector of the basic expression to obtain the first feature change amount of the facial feature point under different expressions.

[0165] a dimension reduction module 303, configured to perform dimension reduction processing on the first feature variation to obtain a second feature variation;

[0166] a setting module 304, configured to set a label of the second feature variation based on a time sequence of the facial image data, to obtain a third feature variation; the label is used to describe an expression feature of the second feature variation;

[0167] a calculation module 305, configured to calculate a semantic similarity matrix between the third feature variations;

[0168] a determination module 306, configured to determine a semantic weight of the third feature variation based on the semantic similarity matrix, to obtain a fourth feature variation; wherein the semantic similarity between the third feature variations is positively correlated with the corresponding semantic weight;

[0169] a decoding module 307, configured to decode the fourth feature variation to obtain a target feature variation.

[0170] The data processing apparatus provided in the application obtains the feature vectors of the facial feature points in each expression based on the facial image data in each expression; the facial feature points are obtained by marking the face according to a preset marking rule; the comparison module 302 compares the feature vectors in each expression with the feature vector of the basic expression to obtain the first feature variation of the facial feature points in different expressions; the dimension reduction module 303 performs dimension reduction processing on the first feature variation to obtain the second feature variation; the setting module 304 sets the label of the second feature variation based on the time sequence of the facial image data to obtain the third feature variation; the label is used to describe the expression feature of the second feature variation; the calculation module 305 calculates the semantic similarity matrix between the third feature variations; the determination module 306 determines the semantic weight of the third feature variation based on the semantic similarity matrix to obtain the fourth feature variation; wherein the semantic similarity between the third feature variations is positively correlated with the corresponding semantic weight; and the decoding module 307 decodes the fourth feature variation to obtain the target feature variation.

[0171] It should be noted that the data processing apparatus of the embodiments of the application has similar principles to the data processing method described above in solving problems, and therefore the implementation process and implementation principles, advantages of the data processing apparatus can be referred to the description of the implementation process and implementation principles, advantages of the method described above, and the repeated parts will not be described herein.

[0172] The embodiments of the application provide an electronic device, comprising:

[0173] at least one processor; and

[0174] a memory connected in communication with the at least one processor; wherein,

[0175] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform

[0176] Based on the facial image data under each expression, a feature vector of a facial feature point under each expression is obtained; the facial feature point is obtained by marking the face according to a preset marking rule;

[0177] The feature vectors under each expression are compared with the feature vector of the basic expression to obtain a first feature change amount of the facial feature point under different expressions;

[0178] The first feature change amount is processed by dimension reduction to obtain a second feature change amount;

[0179] Based on the time sequence of the facial image data, a label of the second feature change amount is set to obtain a third feature change amount; the label is used to describe the expression feature of the second feature change amount;

[0180] A semantic similarity matrix between the third feature change amounts is calculated;

[0181] A semantic weight of the third feature change amount is determined based on the semantic similarity matrix to obtain a fourth feature change amount; wherein the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weight;

[0182] The fourth feature change amount is decoded to obtain a target feature change amount.

[0183] The embodiment of the application provides a non-transient computer readable storage medium storing computer instructions, and the computer instructions are used to enable a computer to execute

[0184] Based on the facial image data under each expression, a feature vector of a facial feature point under each expression is obtained; the facial feature point is obtained by marking the face according to a preset marking rule;

[0185] The feature vectors under each expression are compared with the feature vector of the basic expression to obtain a first feature change amount of the facial feature point under different expressions;

[0186] The first feature change amount is processed by dimension reduction to obtain a second feature change amount;

[0187] Based on the time sequence of the facial image data, a label of the second feature change amount is set to obtain a third feature change amount; the label is used to describe the expression feature of the second feature change amount;

[0188] A semantic similarity matrix between the third feature change amounts is calculated;

[0189] The semantic weights of the third feature change amounts are determined based on a semantic similarity matrix, to obtain fourth feature change amounts; wherein the semantic similarity between the third feature change amounts is positively correlated with the corresponding semantic weights.

[0190] The fourth feature change amounts are decoded to obtain target feature change amounts.

[0191] According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

[0192] Figure 4 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0193] As shown in Figure 4 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0194] Various components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, and the like; an output unit 807, such as various types of displays, speakers, and the like; the storage unit 808, such as a magnetic disk, an optical disk, and the like; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0195] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the image processing method by any other suitable means, such as by means of firmware.

[0196] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0197] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, fully on a machine and partially on a remote machine or entirely on a remote machine or server.

[0198] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0199] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0200] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0201] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0202] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0203] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0204] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, comprising: Based on facial image data under various expressions, feature vectors of facial feature points under each expression are obtained; The facial feature points are obtained by marking the face according to a preset marking rule; The feature vectors of each expression are compared with the feature vectors of the basic expression to obtain the first feature change of the facial feature points under different expressions. The first feature change is subjected to dimensionality reduction processing to obtain the second feature change. Based on the time series of the facial image data, a label is set for the second feature change to obtain a third feature change; the label is used to describe the facial expression features of the second feature change. Calculate the semantic similarity matrix between the changes in the third feature; The semantic weights of the third feature changes are determined based on the semantic similarity matrix to obtain the fourth feature change; wherein the semantic similarity between the third feature changes is positively correlated with the corresponding semantic weights. The fourth feature change is decoded to obtain the target feature change.

2. The method according to claim 1, before obtaining the feature vectors of facial feature points under each expression based on facial image data under each expression, further comprising: The system displays prompts to guide users in adjusting facial position, facial angle, and making corresponding facial expressions. Facial image data for various expressions was obtained based on time series data. The time series corresponds to facial image data for each expression.

3. The method according to claim 1, wherein comparing the feature vectors of each expression with the feature vectors of the basic expression to obtain the first feature change of facial feature points under different expressions includes: Based on the feature vector of the basic expression, the first coordinate group of each facial feature point in the basic expression is obtained; Based on the feature vectors of each expression, the second coordinate set of each facial feature point in each expression is obtained; Based on the first coordinate set and the second coordinate set, calculate the first feature change of each expression relative to the base expression.

4. The method according to claim 1, wherein the dimensionality reduction processing of the first feature change to obtain the second feature change includes: The first feature change is centered to obtain the first vector; Based on the first vector, construct the first vector matrix; Calculate the covariance matrix of the first vector matrix; Obtain the eigenvalues ​​and eigenvectors of the covariance matrix; Based on principal component analysis, a preset number of eigenvectors are selected and sorted using a preset threshold to obtain an eigenvector matrix; wherein the preset threshold represents the proportion of the sum of the eigenvalues ​​corresponding to the preset number of eigenvectors to the sum of the total eigenvalues. Projecting the first vector matrix onto the feature vector matrix yields the second feature change.

5. The method according to claim 2, wherein the step of setting a label for the second feature change based on the time series of the facial image data to obtain the third feature change includes: Based on the time series and corresponding image frames in facial image data, a label is set for the second feature change. The labels include at least static labels representing facial expressions and dynamic labels representing transitional facial expressions; The third feature change is obtained based on the second feature change after the label is set.

6. The method according to claim 1, wherein calculating the semantic similarity matrix between the third feature changes comprises: A tag semantic tree is constructed based on the node rules and the third feature variation; wherein, the node rules represent the hierarchical relationship between the third feature variations; the node rules are used to construct the topology, nodes, and relationships between nodes of the tag semantic tree; Based on the label semantic tree, a semantic similarity matrix is ​​obtained between the third feature changes.

7. The method according to claim 6, wherein the hierarchical relationship of the third feature change includes facial movement type, single feature change, and composite feature change; the semantic range of the facial movement type is higher than the semantic range of the single feature change, and the semantic range of the single feature change is higher than the semantic range of the composite feature change; the step of constructing a label semantic tree based on the third feature change according to node rules includes: A semantic tree for labels is constructed using facial movement type as the root node, single feature change as the intermediate node, and composite feature change as the leaf node. The tag semantic tree includes multiple root nodes, multiple intermediate nodes, and multiple leaf nodes.

8. The method according to claim 7, wherein determining the semantic weight of the third feature change based on the semantic similarity matrix to obtain the fourth feature change comprises: Based on the semantic similarity matrix, the change in the third feature is divided into a positive sample set and a negative sample set; The positive sample set includes multiple positive sample pairs, and the negative sample set includes multiple negative sample pairs; Calculate the first similarity between the positive sample pairs and the second similarity between the negative sample pairs; Based on the first similarity and the second similarity, the loss is compared using a loss function; Based on the loss, semantic weights are assigned to the positive sample pairs to obtain the fourth feature change; the semantic similarity between the third feature changes is positively correlated with the corresponding semantic weights.

9. The method according to claim 8, wherein decoding the fourth feature change to obtain the target feature change includes: A hierarchical decoder is used to obtain the weights of each layer of the tag semantic tree and determine the structural loss. The weight changes of adjacent frames are calculated to obtain the temporal consistency loss; By combining the structural loss and temporal consistency loss, the fourth feature change is optimized to obtain the target feature change.

10. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform... Based on facial image data under various expressions, feature vectors of facial feature points under each expression are obtained; the facial feature points are obtained by marking the face according to a preset marking rule; The feature vectors of each expression are compared with the feature vectors of the basic expression to obtain the first feature change of the facial feature points under different expressions. The first feature change is subjected to dimensionality reduction processing to obtain the second feature change. Based on the time series of the facial image data, a label is set for the second feature change to obtain a third feature change; the label is used to describe the facial expression features of the second feature change. Calculate the semantic similarity matrix between the changes in the third feature; The semantic weights of the third feature changes are determined based on the semantic similarity matrix to obtain the fourth feature change; wherein the semantic similarity between the third feature changes is positively correlated with the corresponding semantic weights. The fourth feature change is decoded to obtain the target feature change.

Images