Confused expression recognition method and device based on emotion matrix and high aggregation subgraph network, equipment and storage medium

Abstract

The application provides a confusion expression recognition method and device based on an emotion matrix and a high aggregation subgraph network, equipment and a storage medium. It relates to the technical field of image classification processing. The method comprises: constructing a predefined emotion matrix; extracting a feature vector of an input face image, performing feature fusion on the feature vector and an embedding vector of a corresponding expression category in the emotion matrix to generate a node feature; constructing an adjacency matrix based on the feature vectors of all face images, and generating a global affinity graph according to the adjacency matrix and the node feature; inputting the node feature as a vertex and the global affinity graph into a global graph convolution network to output a vertex confidence; taking each vertex as an anchor point, screening high-confidence neighborhood nodes based on the vertex confidence, and constructing a high aggregation subgraph; inputting the high aggregation subgraph into a local graph convolution network, and outputting an expression classification result through a fully connected layer. The method can more accurately capture unique feature information, thereby improving the accuracy of recognition.

Description

Technical Field

[0001] This application relates to the field of image classification and processing technology, and in particular to a method, apparatus, device, and storage medium for recognizing confused expressions based on an emotion matrix and a highly aggregated subgraph network. Background Technology

[0002] For recognizing confused expressions, the inherent ambiguity and subtlety of these expressions make it even more difficult to accurately capture and define their features under the interference of complex external factors. The quality of training data also poses a significant obstacle to improving the performance of expression recognition models. For example, datasets collected from the internet commonly exhibit class imbalance and unclear features. Class imbalance leads to overemphasis on certain expression categories during training, while neglecting others; unclear features blur the boundaries between different expression categories, making clear definition difficult. This not only presents a significant challenge to annotation, making it difficult to guarantee the accuracy and reliability of the annotation results, but also greatly affects the model's learning performance. Consequently, the model's recognition efficiency drops drastically when faced with complex and diverse expressions, especially for expressions like the confused expression, whose features are inherently less prominent, making recognition significantly more difficult.

[0003] For example, when specific facial movement units such as "slightly furrowed eyebrows" and "slightly drooping lips" are detected, the expression is very likely to be "confused." If a combination of movement units such as "widened eyes," "lifted upper eyelids," and "slightly open mouth" is detected, the facial expression is more likely to be "surprised." As for a "neutral" expression, its typical characteristics are that the facial muscles are relatively relaxed, the features are naturally relaxed without obvious muscle contraction or relaxation, the eyes are calm and indifferent, and the corners of the mouth are slightly closed, without any obvious upward or downward tilt.

[0004] Table 1 below presents the typical features of seven basic facial expressions and a confused expression, along with the main facial movement units involved. It is clear that the main movement units of the confused expression largely overlap with those of the seven basic facial expressions, undoubtedly highlighting the difficulty of recognizing a confused expression.

[0005] Table 1. Typical characteristics of basic facial expressions and confused expressions, and the main facial movement units involved.

[0006]

[0007]

[0008] Current research on facial expression recognition mainly focuses on seven commonly used basic facial expressions, and the average accuracy of these seven expressions is often used as the evaluation metric. In-depth analysis of the confusion matrices of some current state-of-the-art models reveals that while the overall accuracy of these models is good, this is primarily due to the high accuracy of some easily recognizable expressions. Taking the POSTER++ model as an example, its accuracy in recognizing the "happy" expression far exceeds that of other expressions; however, its accuracy for easily confused expressions such as anger, disgust, and fear is below 75%. Nevertheless, because the high accuracy of the "happy" expression boosts the overall level, the model's overall accuracy still reaches a relatively high level.

[0009] Therefore, conducting research on easily confused facial expression recognition is particularly necessary. Confused expressions, as one of the more difficult to recognize, provide an ideal starting point for in-depth exploration. Focusing on confused expressions helps to gain a deeper understanding of the performance and shortcomings of models when handling complex facial features and relationships, thereby enabling targeted optimization and improvement of the models. After completing research on confused expression recognition and achieving good results, the accumulated experience and technology can be naturally extended to multi-expression classification tasks. This approach of starting with a specific problem and gradually expanding the research scope is conducive to a more robust advancement of easily confused facial expression recognition technology.

[0010] The inherent ambiguity and uncertainty of confused expressions, coupled with interference from complex external factors and limitations in training data, make accurate recognition of confused expressions an extremely challenging task. In-depth research and modeling of the relationship between facial action units and facial expressions are crucial for improving the performance of confused expression recognition. Especially for confused expressions with ambiguous features that are difficult to accurately judge, how to more precisely capture their unique feature information to improve recognition accuracy is a pressing technical problem that needs to be solved. Summary of the Invention

[0011] This application provides a method, device, equipment, and storage medium for recognizing confused facial expressions based on an emotion matrix and a highly clustered subgraph network. Addressing the semantic ambiguity of confused facial expressions, this paper innovatively constructs a graph convolutional neural network framework based on an emotion matrix. By establishing an expression-emotion dimension mapping matrix, the theory of emotion dimension in psychology is transformed into a computable mathematical representation. A dual-path analysis mechanism of global affinity graph and local highly clustered subgraph is designed to achieve multi-granularity modeling from macroscopic expression distribution patterns to microscopic AU (Affinity Entity) association features.

[0012] Firstly, this application provides a method for recognizing confused expressions based on an emotion matrix and a highly aggregated subgraph network, including:

[0013] Construct a predefined emotion matrix; wherein the emotion matrix includes rows, columns and embedding vectors, the rows correspond to expression categories, the columns correspond to the sentiment dimension, and the embedding vectors represent the weight values ​​of expression categories in the sentiment dimension;

[0014] Extract the feature vector from the input facial image, and fuse the feature vector with the embedding vector of the corresponding expression category in the emotion matrix to generate node features;

[0015] An adjacency matrix is ​​constructed based on the feature vectors of all facial images, and a global affinity graph is generated based on the adjacency matrix and node features.

[0016] Using the node features as vertices, the global affinity graph is input into a global graph convolutional network, and the vertex confidence is output.

[0017] Using each vertex as an anchor point, high-confidence neighboring nodes are selected based on vertex confidence to construct a highly clustered subgraph;

[0018] The highly aggregated subgraph is input into a local graph convolutional network, and the expression classification result is output through a fully connected layer.

[0019] In one possible design, the feature vector and the embedding vector corresponding to the expression category in the emotion matrix are fused by directly concatenating the feature vector and the embedding vector, or by performing a linear transformation on the embedding vector using a learnable weight matrix and then fusing it with the feature vector.

[0020] In one possible design, an adjacency matrix is ​​constructed based on the feature vectors of all facial images, including:

[0021] Represent the feature vectors of all facial images as an n×d feature matrix:

[0022]

[0023] Where: F is the feature matrix; n is the total number of images; d is the dimension of the feature vector; f is the matrix element of the feature matrix, representing the feature vectors of different facial images in different dimensions;

[0024] Calculate the dot product matrix of the characteristic matrix F:

[0025]

[0026] Wherein: S ij F represents the feature dot product of the i-th and j-th facial images. i and F j Let f represent the feature vectors of the i-th and j-th facial images, respectively. i,k This represents the feature vector of the i-th facial image in dimension k;

[0027] Calculate the Euclidean norm of each eigenvector and normalize it. The formula is as follows:

[0028]

[0029] Where: ||F i || represents the normalized Euclidean norm, k represents the dimension index, and i represents the index of the face image;

[0030] Store the normalized Euclidean norm as a diagonal matrix D:

[0031] D=diag(||F1||,||F2||,…,||F n ||) (10)

[0032] Among them: ||F1||, ||F2|| and ||F n || denotes the normalized Euclidean norm of the 1st, 2nd, and nth facial images, respectively, and diag denotes the operation of constructing a diagonal matrix;

[0033] By combining the dot product matrix with the normalized Euclidean norm, the cosine similarity matrix can be calculated:

[0034]

[0035] Where: C ij Let represent the cosine similarity matrix between the i-th facial image and the j-th facial image;

[0036] Using facial images as vertices, for each vertex, select the k vertices with the highest similarity to that vertex in the cosine similarity matrix to form a k-nearest neighbor relationship, represented as:

[0037]

[0038] Where: N k (i) represents the k-nearest neighbor set of vertex i, Top-k represents the operation of taking the top k largest elements, C i,1 C i,2 and C i,n Represents the cosine similarity associated with vertex i;

[0039] Based on the k-nearest neighbor relationships, an adjacency matrix of the global affinity graph is constructed.

[0040] In one possible design, the adjacency matrix is ​​represented as:

[0041]

[0042] Where: A represents the adjacency matrix, N k(i) represents the set of k-nearest neighbors of vertex i, N k (j) represents the set of k-nearest neighbors of vertex j, where i and j represent the indices of the face image or vertex.

[0043] In one possible design, vertex confidence is determined using the following formula:

[0044]

[0045] Where: N i y represents the number of vertices i and y. i y represents the true label of vertex i. j This represents the true label of vertex j. This represents the cosine similarity between vertices i and j. This is an indicator function.

[0046] In one possible design, after using the node features as vertices, inputting the global affinity graph into a global graph convolutional network, and outputting vertex confidence, the method further includes performing a graph convolution operation based on the graph input to the global graph convolutional network using the following formula to obtain the embedding features of each vertex:

[0047]

[0048] in: c represents the embedding feature of vertex j in the l-th layer. ij W represents the normalization coefficient. l Let N represent the weight matrix of the l-th layer, σ(·) represent the activation function, and N... i This represents the number of vertices i. This represents the embedding feature of vertex i in the (l+1)th layer.

[0049] In one possible design, the computation process of inputting the highly aggregated subgraph into a local graph convolutional network and outputting the expression classification result through a fully connected layer can be represented as follows:

[0050] f(H L = softmax(W) L+1 H L +b L+1 (16)

[0051] Among them: W L+1 and b L+1 These are the weight matrix and bias vector of the fully connected layer, where softmax represents the activation function, and H... L This represents the concatenation of all vertex embedding features to form a vertex feature matrix, f(H). L ) represents a probability matrix.

[0052] Secondly, this application provides a confused expression recognition device based on an emotion matrix and a highly clustered subgraph network, the device comprising:

[0053] The emotion matrix construction module is configured to construct a predefined emotion matrix; wherein the emotion matrix includes rows, columns and embedding vectors, the rows correspond to expression categories, the columns correspond to the sentiment dimension, and the embedding vectors represent the weight values ​​of expression categories in the sentiment dimension;

[0054] The feature fusion module is configured to extract feature vectors from the input facial image, fuse the feature vectors with the embedding vectors of the corresponding expression categories in the emotion matrix, and generate node features.

[0055] The affinity graph construction module is configured to construct an adjacency matrix based on the feature vectors of all facial images, and generate a global affinity graph based on the adjacency matrix and node features;

[0056] The confidence calculation module is configured to use the node features as vertices, input the global affinity graph into a global graph convolutional network, and output the vertex confidence.

[0057] The confidence filtering module is configured to use each vertex as an anchor point to filter high-confidence neighborhood nodes based on the vertex confidence, and construct a highly clustered subgraph.

[0058] The confused expression classification module is configured to input the highly aggregated subgraph into a local graph convolutional network and output the expression classification result through a fully connected layer.

[0059] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the confused expression recognition method based on emotion matrix and highly clustered subgraph network as described in the first aspect and various possible designs of the first aspect.

[0060] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the confused expression recognition method based on the emotion matrix and highly aggregated subgraph network described in the first aspect and various possible designs of the first aspect.

[0061] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the confused expression recognition method based on an emotion matrix and a highly aggregated subgraph network as described in the first aspect and various possible designs of the first aspect.

[0062] The method, apparatus, device, and storage medium for recognizing confused expressions based on emotion matrices and highly clustered subgraph networks provided in this application have at least the following beneficial effects:

[0063] (1) By using the jointly extracted feature map, the potential correlation information between the expression category and the emotion dimension is cleverly introduced and deeply integrated with the image features to innovatively construct the emotion matrix.

[0064] (2) By adopting a global-local combined convolutional neural network architecture, namely GCN-A and GCN-S, the full-range and multi-level extraction of facial expression features is realized, which significantly improves the recognition accuracy of confused expressions and successfully breaks through the limitations of traditional methods in processing such complex expressions.

[0065] (3) It innovatively integrates emotional semantic embedding with visual features to enhance the model's semantic reasoning ability for ambiguous expressions. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 A structural diagram of an expression recognition model based on an emotion matrix and GCN provided in an embodiment of this application;

[0068] Figure 2 A flowchart illustrating a method for recognizing confused expressions based on an emotion matrix and a highly aggregated subgraph network, provided in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the high-aggregation subgraph construction process provided in an embodiment of this application;

[0070] Figure 4 The confusion matrix of the method provided in the embodiments of this application on the AffectNet dataset;

[0071] Figure 5 The confusion matrix of the method provided in the embodiments of this application on the RAF-AU dataset;

[0072] Figure 6 The method provided in this application provides a visualization of the emotion graph network before applying it to RAF-AU.

[0073] Figure 7 A visualization of the method provided in this application embodiment after applying a sentiment graph network to RAF-AU;

[0074] Figure 8 This is a structural diagram of a confused expression recognition device based on an emotion matrix and a highly aggregated subgraph network, provided in an embodiment of this application.

[0075] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0077] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0078] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0080] To effectively identify confused expressions and resolve the issue of confusion between confused expressions and other expressions, this application provides a method for recognizing confused expressions based on an emotion matrix and a highly aggregated subgraph network (GCN). This method can be implemented based on an expression recognition model based on an emotion matrix and a GCN, the structure of which is as follows: Figure 1 As shown, x1, x2, and xn represent the input image data, including confused expression images and other expression images. The model first performs feature extraction on the confused expression images and other expression images, resulting in a set of features, which form a set of features F. p .

[0081]

[0082] Where: n represents the number of facial images, d is the feature dimension, R is the set of real numbers, and pi represents the i-th facial image. Simultaneously, this is combined with the emotion matrix E. prior Additional sentiment semantic embeddings are introduced for each expression category. Then, image features and sentiment embeddings are combined through feature fusion to generate vertex features for graph convolution operations.

[0083] The goal is to predict the expression label y for each facial image i (i∈{1,2,…,n}) by capturing the latent relationships between expressions. i (y = 1 or 0). In particular, it accurately identifies easily confused expressions of disbelief.

[0084] To achieve this, based on the vertex classification framework of graph data, this paper aims to implement vertex category prediction in facial expression recognition tasks. By constructing and analyzing the structural features of affinity graphs, facial expression information can be effectively extracted from facial images. In the affinity graph construction process, a vertex mapping method is used to transform each facial image into vertex units in the graph structure. Specifically, a k-nearest neighbor graph is constructed based on the cosine similarity measure of the initial feature matrix. The cosine similarity algorithm is used to determine the k nearest neighbors for each vertex, and a complete global affinity graph G is formed through the connection relationships between nodes. This graph structure can be mathematically represented by the initial vertex feature matrix and the symmetric adjacency matrix: when vertices i and j are connected, the matrix elements are the cosine similarity values ​​between them; if there is no connection, the matrix elements are assigned a value of zero.

[0085] Each facial expression is recognized in a global-local manner by two modules, GCN-A and GCN-S. GCN-A takes the entire global affinity graph G as input and performs inference via a graph convolutional neural network. GCN-A outputs vertex confidence scores to detect the global distribution of facial expressions on the affinity graph. Then, using each image as an anchor, a high-aggregation subgraph, called HAS, is constructed based on the vertex confidence scores and the high-order neighbors of the top embedding function. This will be further explained in subsequent embodiments. Finally, HAS is used as input to GCN-S, which learns from GCN-S with shared parameters to predict the class of each vertex.

[0086] Specifically, such as Figure 2 The diagram shown is a flowchart of a method for recognizing confused expressions based on an emotion matrix and a highly clustered subgraph network provided in this application. The method includes the following steps S100-S600.

[0087] S100: Construct a predefined emotion matrix; where the emotion matrix includes rows, columns and embedding vectors, rows correspond to expression categories, columns correspond to the sentiment dimension, and the embedding vectors represent the weight values ​​of expression categories in the sentiment dimension.

[0088] For example, as shown in Table 2, the sentiment matrix E prior It is a predefined embedding matrix used to describe the relationship between expression categories and sentiment dimensions. Its rows represent expression categories (confusion, surprise, neutral, disgust), columns represent sentiment dimensions ("uncertainty", "positive sentiment", "negative sentiment", "intensity"), and matrix values ​​are corresponding weights, ranging from [0,1].

[0089] Table 2 Emotion Matrix

[0090] expression Uncertainty Positive emotions negative emotions strength Puzzled 0.9 0.2 0.1 0.4 surprise 0.6 0.4 0.2 0.8 neutral 0.5 0.2 0.2 0 disgust 0.2 0.1 0.8 0.7

[0091] The emotion matrix provides a high-level semantic embedding for each expression category, which helps the model establish semantic distinctions between expression categories.

[0092] S200: Extract the feature vector of the input facial image, fuse the feature vector with the embedding vector of the corresponding expression category in the emotion matrix, and generate node features.

[0093] In some embodiments, to fully utilize the semantic information of the sentiment matrix, its embedding vector is combined with the feature vector f extracted from the image. visual The nodes are fused together to generate the final node features f. final The fusion method is as follows:

[0094] The embedding vector E in the sentiment matrix prior [i] and the feature vector f of the facial image final [i] Concatenate to form new node features:

[0095] f final =Concat(f visual [i],E prior [i])(2)

[0096] Concat(·) represents the vector concatenation operation.

[0097] In some implementations, a learnable weight matrix W can be introduced. prior A linear transformation is performed on the embedding vectors in the sentiment matrix to adapt them to the feature learning space of the GCN:

[0098]

[0099] The adjusted emotion embedding is then fused with the feature vector of the facial image.

[0100] The final node features f visual The vertex features used as input to the GCN are further used to learn the relationships between nodes through graph convolution operations. The introduction and fusion of the sentiment matrix not only provides high-level semantic information for confused expression recognition, but also enhances the model's generalization ability to scenarios with insufficient visual features.

[0101] S300: Construct an adjacency matrix based on the feature vectors of all facial images, and generate a global affinity graph based on the adjacency matrix and node features.

[0102] The construction of the adjacency matrix is ​​one of the key points. The crucial element in constructing a high-quality graph structure in this embodiment lies in the accurate measurement of feature vector similarity, a step that directly determines the performance of the graph convolutional model in the task of recognizing confused facial expressions. Cosine similarity is introduced as an effective tool for evaluating the similarity of feature vectors. Its mathematical essence lies in transforming complex relationships in high-dimensional space into geometric analyses of vector directions. This measurement method focuses on calculating the cosine value of the angle between vectors, representing the intrinsic correlation of data points through spatial direction rather than amplitude differences. The traditional Euclidean distance, which emphasizes amplitude comparison, is susceptible to interference under scenarios such as changes in lighting and fluctuations in facial expression intensity. The evaluation strategy based on directional similarity precisely compensates for this deficiency, providing a more stable mathematical framework for facial expression feature modeling.

[0103] The definition of cosine similarity is as follows:

[0104]

[0105] Wherein: F i and F j Let i and j represent the feature vectors of the i-th and j-th facial images, respectively.

[0106]

[0107]

[0108] D represents the dimension of the feature vector. The range of cosine similarity is [-1, 1], where 1 represents perfect similarity and -1 represents perfect opposites.

[0109] In some embodiments, the adjacency matrix is ​​constructed based on the feature vectors of all facial images in the following manner:

[0110] The feature vectors of all facial images are F = {F1, F2, ..., F...} n} can be represented as an n×d characteristic matrix:

[0111]

[0112] Where: F is the feature matrix; n is the total number of images; d is the dimension of the feature vector; f is the matrix element of the feature matrix, representing the feature vectors of different facial images in different dimensions.

[0113] Calculate the dot product matrix S of the characteristic matrix F:

[0114]

[0115] Wherein: S ij F represents the feature dot product of the i-th and j-th facial images. i and F j Let f represent the feature vectors of the i-th and j-th facial images, respectively. i,k Let represent the feature vector of the i-th facial image in dimension k.

[0116] The dot product matrix S is an n×n symmetric matrix.

[0117] Calculate the Euclidean norm of each vector and use it for normalization:

[0118]

[0119] Where: ||F i || represents the normalized Euclidean norm, k represents the dimension index, and i represents the index of the face image.

[0120] Then the norm is stored as a diagonal matrix D:

[0121]

[0122] Among them: ||F1||, ||F2|| and ||F n || denotes the normalized Euclidean norm of the 1st, 2nd, and nth facial images, respectively, and diag denotes the operation of constructing a diagonal matrix.

[0123] By combining the dot product matrix S with the normalized norm, the cosine similarity matrix can be calculated:

[0124]

[0125] Where: C ij Let C represent the cosine similarity matrix between the i-th and j-th facial images, and let C be the final cosine similarity matrix with a value range of [-1, 1].

[0126] For each vertex i, select the k vertices with the highest similarity (excluding itself) from the cosine similarity matrix C to form a k-nearest neighbor relationship. Let N k Let (i) represent the set of k-nearest neighbors of vertex i, then:

[0127]

[0128] Where: N k (i) represents the k-nearest neighbor set of vertex i, Top-k represents the operation of taking the top k largest elements, C i,1 C i,2 and C i,n This represents the cosine similarity associated with vertex i.

[0129] Based on k-nearest neighbor relationships, construct the adjacency matrix of the affinity graph.

[0130]

[0131] Where: A represents the adjacency matrix, N k (i) represents the set of k-nearest neighbors of vertex i, N k (j) represents the set of k-nearest neighbors of vertex j, where i and j represent the indices of the face image or vertex.

[0132] The adjacency matrix A, as a sparse matrix, is used to represent the similarity associations between vertices. A k-nearest neighbor graph is constructed based on the cosine similarity of eigenvectors. This method effectively reveals the potential association patterns between facial expression data. Cosine similarity, as a unitless standardized metric, avoids computational biases caused by differences in the dimensions of eigenvectors due to its geometric properties. This metric effectively eliminates interference from feature amplitude and improves the recognition efficiency of affinity maps for facial expression features. Experiments demonstrate that this angle-similarity-based modeling strategy can increase inter-class distances when processing micro-expression recognition tasks.

[0133] S400: Using node features as vertices, the global affinity graph is input into a global graph convolutional network, and the vertex confidence is output.

[0134] S500: Using each vertex as an anchor point, high-confidence neighboring nodes are selected based on vertex confidence to construct a highly clustered subgraph.

[0135] Highly aggregated subgraphs can be constructed through steps S400 and S500, such as Figure 3 The diagram shown illustrates the high-aggregation subgraph construction process provided in this application embodiment. The process of constructing a high-aggregation subgraph (HAS) specifically includes:

[0136] A vertex confidence-based discrimination mechanism was first constructed within the global prediction framework. High-confidence facial expression samples exhibit significant same-class clustering characteristics in their spatial distribution, while low-confidence samples show cross-class adjacency characteristics. This phenomenon is particularly evident in the confusion region between confused and non-confused expressions. Based on these findings, this paper innovatively proposes a neighborhood-optimized vertex confidence definition method and designs a corresponding confidence estimation network architecture. By introducing an adaptive aggregation function, deep modeling and learning of the complex distribution characteristics of facial expression data are achieved.

[0137] The formula for calculating vertex confidence is as follows:

[0138]

[0139] Where N i This represents the number of vertices i. For indicator functions, y i This represents the true label of vertex i, with a value of 0 or 1, indicating whether it is confused or not confused. This represents the cosine similarity between vertices i and j.

[0140] GCN-A is an L-layer network. Taking G (a graph composed of interconnected feature vectors) as input, GCN-A estimates the confidence of each vertex. Through forward propagation, it learns the activation function and weight parameters. The transformation function is learned using the input features and the features aggregated from the embedded neighborhood. The specific graph convolution operation is as follows:

[0141]

[0142] in: c represents the embedding feature of vertex j in the l-th layer. ij W represents the normalization coefficient. l Let N represent the weight matrix of the l-th layer, σ(·) represent the activation function, and N... i This represents the number of vertices i. This represents the embedding feature of vertex i in the (l+1)th layer.

[0143] It should be noted that c ij These are coefficients calculated based on vertex confidence, for example,

[0144] Based on vertex confidence, high-order neighbors are found, which can help construct a highly clustered subgraph. Finally, the top-level embedding features of these neighbors are used to construct the Highly Clustered Subgraph (HAS). By using the embedding features of these neighbors, a globally meaningful highly clustered subgraph can be constructed. First, first-order neighbor nodes with high confidence are found, and nodes with low confidence are deleted. Then, second-order neighbor nodes with high confidence are found. The vertices and the selected neighbors form a highly clustered subgraph.

[0145] Specifically, the process involves finding first-order neighbor nodes with high confidence, deleting nodes with low confidence, and then finding second-order neighbor nodes with high confidence. The vertices and the selected neighbors then form a highly clustered subgraph. The specific implementation is as follows:

[0146] For each anchor point i (i.e., each vertex), iterate through all its first-order neighbors. For directly connected neighbors, if the confidence of neighbor node j is greater than a certain set threshold T (e.g., 0.5), retain the neighbor; otherwise, delete the neighbor. This process filters for high-confidence first-order neighbors. Based on the same principle, further filter for high-confidence second-order neighbors based on the high-confidence first-order neighbors, and then iterate through the first-order neighbors of j again. Note that the first-order neighbors of j include anchor point i and other nodes m. For each neighbor m of j (m ≠ i), if the confidence of m is greater than the threshold T and m is not yet included in the current HAS, add it. It is important to note that at this point, only second-order neighbors are considered, i.e., nodes m reached from anchor point i through an intermediate node j, and only high-confidence m are retained.

[0147] S600: Input the highly aggregated subgraph into the local graph convolutional network, and output the expression classification result through the fully connected layer.

[0148] In some embodiments, each highly aggregated subgraph is used as input to a second graph convolutional neural network (Local Graph Convolutional Network GCN-S), and graph convolution forward propagation is performed as described above. All nodes are concatenated together to form a node feature matrix, which is then input into a fully connected layer. The fully connected formula is as follows:

[0149] f(H L = softmax(W) L+1 H L +b L+1 (16)

[0150] Among them: W L+1 and b L+1 These are the weight matrix and bias vector of the fully connected layer, where softmax represents the activation function, and H... L This represents the concatenation of all vertex embedding features to form a vertex feature matrix, f(H). LThe probability matrix is ​​represented by , and a graph convolutional neural network is trained using the backpropagation algorithm to obtain the representation of each node. And the classification results of the images.

[0151] To verify the effectiveness of the confused expression recognition method based on emotion matrix and highly aggregated subgraph network proposed in this application, this embodiment conducted extensive experiments on the dataset and analyzed the model from three aspects: overall performance evaluation, comparison of different methods, and ablation experiments of key modules.

[0152] To deeply analyze the model's classification performance on the AffectNet and RAF-AU datasets, a confusion matrix was constructed for detailed analysis. The confusion matrix, as a powerful tool for evaluating classification model performance, clearly presents the matching between the model's predictions and the true labels.

[0153] like Figures 4 to 7 As shown, a visual comparison of the results before and after applying this method clearly reveals the significant differences. Compared to the original method that did not incorporate highly clustered subgraphs and a sentiment matrix, the organic combination and application of these two elements makes the previously ambiguous and unclear boundaries between confused and non-confused features much clearer. In actual classification, images that were easily confused in the past due to similar features are no longer easily misclassified, greatly improving the accuracy and reliability of classification.

[0154] To verify the effectiveness of the proposed confused facial expression recognition model based on the emotion matrix and graph convolutional neural network (GCN), its performance was compared with several mainstream facial expression recognition methods on RAF-AU and AffectNet. These methods include state-of-the-art facial expression recognition methods based on convolutional neural networks and lightweight models, such as EmoNeXt, RepVGG, POSTER++, and ResEmoteNet, as well as fusion attention network (MVTS) models. Simultaneously, the parameter count of each model was compared, and their performance, generalization ability, and complexity were comprehensively evaluated.

[0155] Table 3 Comparative Experiments

[0156]

[0157] The model or method proposed in this application achieves state-of-the-art classification performance on both datasets, reaching 82.07% accuracy on the AffectNet dataset and 88.42% accuracy on the RAF-AU dataset. Compared with existing methods, the performance advantages of this model are manifested in the following aspects:

[0158] RAF-AU dataset: Compared to the attention-based MVTS model, our model achieves a 3.28 percentage point improvement in accuracy. This indicates that our model can better handle fine-grained emotion classification tasks on complex facial expression datasets.

[0159] AffectNet dataset: On this dataset, the accuracy improvement of the proposed model is more significant, exceeding MVTS by 3.38 percentage points. As a more diverse and noisy dataset, AffectNet allows the proposed model to effectively improve classification robustness by combining the sentiment matrix with the GCN module.

[0160] This result fully demonstrates the high feasibility of the proposed method for confused expression recognition on the RAF-AU and AffectNet datasets.

[0161] To further verify the effectiveness of the proposed confused expression recognition model based on the emotion matrix and highly clustered subgraphs, a series of ablation experiments were designed. Key modules of the model were progressively removed or replaced, and the specific impact of each module on model performance and parameter count was analyzed. The experiments were conducted on the RAF-AU and AffectNet datasets.

[0162] To investigate the independent contributions of each component in the model, ablation experiments were conducted using the following four model configurations:

[0163] The model using MVTS for feature extraction only extracts feature vectors from images for classification, without including advanced modules such as GCN or sentiment matrix. Building upon MVTS, a single-stage GCN module is added to capture the global relationship structure between features, evaluating the performance improvement of graph convolution on classification. In addition to feature extraction and GCN, a sentiment matrix is ​​introduced to enhance the discriminative power of expression categories through high-level semantic embedding.

[0164] Building upon the aforementioned model, a two-stage design, GCN-A and GCN-S, was incorporated. GCN-A handles global inference to calculate anchor confidence and constructs highly clustered subgraphs based on the confidence scores. GCN-S then refines local features based on these subgraphs. The specific results are shown in Table 4.

[0165] Table 4 Ablation experimental results under different model configurations

[0166]

[0167] As shown in Table 4, graph convolutional networks can effectively capture the global relationship structure between features, significantly improving the accuracy of expression classification. Even without introducing an emotion matrix or highly clustered subgraphs, the model's accuracy on the AffectNet and RAF-AU datasets improved by 0.98 and 1.83 percentage points, respectively, using only a single-stage GCN.

[0168] The emotion matrix provides independent semantic embeddings for each expression category. Fusing these embeddings with visual features enhances the model's ability to handle complex expression classification tasks and improves its robustness. The accuracy improvement is even more significant on the noisy AffectNet dataset.

[0169] GCN-A constructs highly clustered subgraphs through global inference and removes low-confidence nodes, thus reducing noise interference. GCN-S performs detailed learning of local feature relationships, further improving the accuracy of fine-grained classifications such as confused expressions. The combination of global inference and local aggregation constitutes the key to improving model performance.

[0170] The full model has a larger number of parameters (77.58MB) than the basic MVTS model (45.64MB). After introducing a single-stage GCN or sentiment matrix, the model configuration has a limited increase in the number of parameters compared to the former, but the performance is significantly improved.

[0171] This application also provides a confused expression recognition device based on an emotion matrix and a highly aggregated subgraph network, such as... Figure 8 As shown, the confused expression recognition device based on the emotion matrix and highly clustered subgraph network includes:

[0172] The emotion matrix construction module 801 is configured to construct a predefined emotion matrix; wherein the emotion matrix includes rows, columns and embedding vectors, the rows correspond to expression categories, the columns correspond to the emotion dimension, and the embedding vectors represent the weight values ​​of expression categories on the emotion dimension;

[0173] The feature fusion module 802 is configured to extract feature vectors from the input facial image, fuse the feature vectors with the embedding vectors of the corresponding expression categories in the emotion matrix, and generate node features.

[0174] The affinity graph construction module 803 is configured to construct an adjacency matrix based on the feature vectors of all facial images, and generate a global affinity graph based on the adjacency matrix and node features;

[0175] The confidence calculation module 804 is configured to use the node features as vertices, input the global affinity graph into a global graph convolutional network, and output the vertex confidence.

[0176] The confidence filtering module 805 is configured to use each vertex as an anchor point to filter high-confidence neighborhood nodes based on vertex confidence and construct a highly clustered subgraph.

[0177] The confused expression classification module 806 is configured to input the highly aggregated subgraph into a local graph convolutional network and output the expression classification result through a fully connected layer.

[0178] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0179] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0180] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0181] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0182] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the confused expression recognition method based on the emotion matrix and highly aggregated subgraph network described in the above embodiments.

[0183] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the confused expression recognition method based on the emotion matrix and highly aggregated subgraph network in the above embodiments.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0185] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0186] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0187] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0188] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0189] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0190] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0191] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0192] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0193] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A confusion expression recognition method based on an emotion matrix and a high aggregation subgraph network, characterized in that, The method comprises: constructing a predefined emotion matrix; wherein the emotion matrix comprises rows, columns and embedding vectors, the rows correspond to expression categories, the columns correspond to emotional dimensions, and the embedding vectors represent weight values of expression categories in emotional dimensions; extracting a feature vector of an input facial image, and performing feature fusion on the feature vector and an embedding vector of a corresponding expression category in the emotion matrix to generate a node feature; constructing an adjacency matrix based on the feature vectors of all facial images, and generating a global affinity graph according to the adjacency matrix and the node feature; taking the node feature as a vertex, inputting the global affinity graph into a global graph convolution network, and outputting vertex confidence; taking each vertex as an anchor point, screening high-confidence neighborhood nodes based on the vertex confidence, and constructing a high-aggregation subgraph; inputting the high-aggregation subgraph into a local graph convolution network, and outputting an expression classification result through a fully connected layer; the vertex confidence is determined by the following formula: (14) wherein: N i denotes a vertex i of the graph, y i denotes a true label of a vertex i , y j denotes a true label of a vertex j , denotes and a cosine similarity between two vertices, is an indicator function.

2. The method of claim 1, wherein the method is performed by a computer system. The way of performing feature fusion on the feature vector and the embedding vector of the corresponding expression category in the emotion matrix includes directly splicing the feature vector and the embedding vector or performing linear transformation on the embedding vector through a learnable weight matrix and then fusing the embedding vector with the feature vector. 3.The confusion expression recognition method based on emotion matrix and high aggregation sub-graph network according to claim 1, characterized in that, Constructing an adjacency matrix based on the feature vectors of all facial images comprises: The feature vectors of all face images are represented as a feature matrix: : (7) wherein: F is a feature matrix; is the total number of images; is the dimension of the feature vector; f is a matrix element of the feature matrix, representing the feature vector of different facial images in different dimensions. Dot product matrix of the computed feature matrix Dot product matrix of the computed feature matrix (8) wherein: denotes the i feature point product of the j m-th face image and the feature vector of the m-th face image, denotes the feature vector of the f m-th face image, i,k denotes the i feature vector of the k m-th face image in dimension calculating the Euclidean norm of each feature vector and normalizing it, and the calculation formula is: (9) wherein: denotes the normalized Euclidean norm, k denotes the dimension index, i denotes the index of the face image; Store the normalized Euclidean norm as a diagonal matrix : (10) wherein: , and respectively denote the normalized Euclidean norm of the 1st, 2nd and 3rd n side face image, diag denotes the operation of constructing a diagonal matrix; combining the dot product matrix with the normalized Euclidean norm to calculate the cosine similarity matrix: (11) wherein: C ij denotes the i cosine similarity matrix of the j cosine similarity matrix of the with the face images as vertices, for each vertex, selecting, in the cosine similarity matrix, the vertex with the highest similarity to the vertex k k a nearest neighbor relationship, denoted as:​ (12) wherein: denotes a vertex k-neighbors of a vertex, Top-k denotes an operation of taking the k largest elements, C i,1 , C i,2 and C i,n denotes a cosine similarity related to vertices i ​ Based on the k-nearest neighbor relationship, an adjacency matrix of the global affinity graph is constructed.

4. The confusion expression recognition method based on emotion matrix and high aggregation sub-graph network according to claim 1 or 3, characterized in that, The adjacency matrix is represented as: (13) wherein: A denotes an adjacency matrix, denotes a k-neighborhood of a vertex N k ( j ) denotes a k-neighborhood of a vertex j i and j denote an index of a face image or a vertex.​​ 5. The method of claim 1, wherein the method is performed by a computer system. After taking the node feature as a vertex, inputting the global affinity graph into a global graph convolution network, and outputting vertex confidence, the method further comprises performing graph convolution operation based on the global graph convolution network through the following formula to obtain each vertex embedding feature: (15) in: Indicates the first vertex in layer j Embedding features, Represents the normalization coefficient. Indicates the first The weight matrix of the layer, This represents the activation function. N i Represents vertices i Quantity, Indicates the first Vertex in +1 layer i Embedded features.

6. The method of claim 1, wherein the method is performed by a computer system. The calculation process of inputting the high-aggregation subgraph into a local graph convolution network and outputting an expression classification result through a fully connected layer is represented as: (16) where: and are the weight matrix and bias vector of the fully connected layer, softmax denotes the activation function, H L denotes the concatenation of all vertex embedding features to form the vertex feature matrix, f ( H L ) denotes the probability matrix.

7. A device for confusion expression recognition based on emotion matrix and high aggregated subgraph network, characterized in that, The device comprises: an emotion matrix construction module configured to construct a predefined emotion matrix; wherein the emotion matrix comprises rows, columns and embedding vectors, the rows correspond to expression categories, the columns correspond to emotional dimensions, and the embedding vectors represent weight values of expression categories in emotional dimensions; a feature fusion module configured to extract a feature vector of an input facial image, and perform feature fusion on the feature vector and an embedding vector of a corresponding expression category in the emotion matrix to generate a node feature; an affinity graph construction module configured to construct an adjacency matrix based on the feature vectors of all facial images, and generate a global affinity graph according to the adjacency matrix and the node feature; a confidence calculation module configured to take the node feature as a vertex, input the global affinity graph into a global graph convolution network, and output vertex confidence; the vertex confidence is determined by the following formula: (14) wherein: N i denotes a vertex i of the graph, y i denotes a true label of a vertex i , y j denotes a true label of a vertex j , denotes and a cosine similarity between two vertices, is an indicator function; a confidence screening module configured to take each vertex as an anchor point, screen high-confidence neighborhood nodes based on the vertex confidence, and construct a high-aggregation subgraph; a confusion expression classification module configured to input the high-aggregation subgraph into a local graph convolution network, and output an expression classification result through a fully connected layer.

8. An electronic device, comprising: comprise: a processor, and a memory connected with the processor in communication; the memory stores computer-executed instructions; the processor executes the computer-executed instructions stored in the memory to implement the confusion expression recognition method based on the emotion matrix and the high-aggregation subgraph network according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the confusion expression recognition method based on the emotion matrix and the high-aggregation subgraph network according to any one of claims 1-6.

Images