A multi-domain feature-level fusion expression recognition method based on graph neural network

Abstract

The present invention discloses a multi-domain feature-level fusion expression recognition method based on a graph neural network, which relates to the field of computer vision technology, including firstly, data enhancement of the original image, based on key point detection, the facial region is divided according to the three-part and five-eye region; secondly, the facial graph structure is constructed, and corresponding weights are assigned to each edge according to the region where the feature points are located, and the weighted distance information is obtained, and stored in the adjacency matrix; then the spectral domain features and spatial domain features are obtained through the graph neural network, and the local features and context features are obtained through the multi-level convolutional neural network and ViT respectively; after that, multi-domain features are fused through the feature fusion network guided by structural features, and then the expression recognition results are obtained. Therefore, the use of the above-mentioned multi-domain feature-level fusion expression recognition method based on a graph neural network can enable the model to understand the changes in expression more deeply, enhance the recognition ability, and improve the robustness and stability of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-domain feature-level fusion expression recognition method based on a graph neural network. Background Art

[0002] The change of facial expression is often used as an important basis for judging the change of emotion. According to the difference in duration and amplitude, facial expressions are divided into macro-expressions and micro-expressions. Existing research divides the facial expression recognition task into two tasks: macro-expression recognition and micro-expression recognition. Different models are trained using macro-expression datasets and micro-expression datasets respectively, and the trained models are used to detect macro-expressions and micro-expressions respectively. Compared with macro-expressions, micro-expressions have a short duration and low amplitude, so different features need to be extracted for macro-expressions and micro-expressions respectively. Accurately extracting features with differences can effectively improve the recognition accuracy.

[0003] At present, feature extraction methods can be divided into traditional methods and deep learning methods. Traditional methods include HOG features, LBP features, SIFT features, color features, etc., but traditional methods are easily disturbed by external factors such as illumination changes, posture changes and occlusion. Deep learning methods are mainly based on convolutional neural networks, which can automatically extract features and are widely used in expression recognition tasks. However, existing research mainly extracts features in the spatial domain of images, and does not consider the relationship between image structural features, local features and global features. Although convolution has a strong ability to express local features of images, it cannot reflect the relationship between features, which will seriously affect the accuracy of emotion recognition.

[0004] On the other hand, due to the large size of the image, it is necessary to downsample the image into a smaller image, which will lead to information loss and decreased reasoning accuracy. In addition, expressions are private, there are few public expression datasets, and the data distribution between classes in the dataset is uneven. With a small amount of training data, the classification accuracy is low and the model generalization is poor.

[0005] Therefore, it is necessary to provide a multi-domain feature-level fusion expression recognition method that can accurately extract distinctive features with a small amount of training data and realize macro-expression and micro-expression recognition under the same recognition framework. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-domain feature-level fusion expression recognition method based on graph neural network, which can make full use of the features of each level of the image, enhance the understanding of the image, and improve the accuracy of expression recognition.

[0007] To achieve the above object, the present invention provides a multi-domain feature-level fusion expression recognition method based on graph neural network, comprising the following steps:

[0008] S1. Perform data enhancement on the original image, including image rotation, color conversion, image scaling and adding Gaussian noise; perform key point detection on the facial image, and segment the face according to the three-part and five-eye divisions based on the key point detection results;

[0009] S2. Construct an undirected weighted graph of the face according to the divided facial regions, assign corresponding weights to each edge according to the regions where the feature points are located, obtain weighted distance information, and store it in an adjacency matrix;

[0010] S3. Based on the adjacency matrix and the undirected weighted graph, spectral domain features and spatial domain features are obtained through the spectral domain feature extraction branch and the spatial domain feature extraction branch respectively; local features are extracted on the original image through a multi-level convolutional neural network; context features of the image are extracted on the original image through ViT;

[0011] S4. Cascade the spectral domain features and spatial domain features to obtain structural features, and use the feature fusion network guided by structural features to fuse multi-domain features. In addition, introduce the self-attention module to calculate the attention function of each feature to obtain the fused features, and then obtain the expression recognition results.

[0012] Preferably, constructing the undirected weighted graph of the face includes using the pixel points of the divided facial region image as nodes of the graph structure and the distances between the nodes as edges of the graph structure.

[0013] Assign corresponding weights to each edge according to the area where the feature points are located, including determining the corresponding weights according to the Manhattan distance between the area where the current node is located and the area where the node to be connected is located, as follows:

[0014] ;

[0015] ;

[0016] In the formula, For Node With Node The Manhattan distance between , Node ,node The coordinates of For Node and nodes The weight of the edge between them.

[0017] Preferably, the spectral domain feature extraction branch includes converting the image into the spectral domain using a graph Fourier transform, and then using a convolutional network to extract spectral domain features.

[0018] Preferably, the spatial feature extraction branch includes convolving the nodes of the undirected weighted graph and their adjacent nodes to obtain the spatial features.

[0019] Preferably, the multi-stage convolutional neural network includes a Conv module and a MFEM module;

[0020] Among them, the Conv module consists of three parts: convolution layer, normalization layer and activation function, which are used to downsample the feature map, normalize and perform nonlinear processing on the data respectively;

[0021] The MFEM module is used to extract multi-scale features.

[0022] Preferably, the feature fusion network guided by structural features includes taking structural features as query matrix Q, local features as key matrix K and context features as value matrix V, calculating the similarity between the query matrix and the key matrix through scaled dot product operation, and obtaining the weight coefficient of the value matrix, thereby obtaining the fused features.

[0023] Therefore, the present invention adopts the above-mentioned multi-domain feature-level fusion expression recognition method based on graph neural network, which has the following technical effects:

[0024] (1) On the one hand, local feature extraction makes the convolutional neural network more robust to background noise and lighting changes, which helps to improve the stability of expression recognition. On the other hand, ViT can directly model the relationship between various parts of the image through the self-attention mechanism. This global vision enables the model to effectively capture the subtle connections between facial features and help better understand complex expression changes.

[0025] (2) Considering the spatial and spectral features of the graph structure and using them to guide local features and contextual features for multi-domain feature fusion can improve the model's ability to resist interference and adapt to complex scenarios. At the same time, the multi-layer convolutional neural network enables the model to have a deeper understanding of expression changes and enhance its ability to recognize expressions.

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a multi-domain feature-level fusion expression recognition method based on graph neural network;

[0028] Figure 2 A facial region result graph divided according to feature point detection in an embodiment of a multi-domain feature-level fusion expression recognition method based on a graph neural network;

[0029] Figure 3The present invention is a schematic diagram of an embodiment of a multi-domain feature-level fusion expression recognition method based on a graph neural network, in which feature points are used as graph structure nodes;

[0030] Figure 4 It is a schematic diagram of an embodiment of a multi-domain feature-level fusion expression recognition method based on a graph neural network, using pixels as graph structure nodes;

[0031] Figure 5 It is a schematic diagram of constructing an adjacency matrix in an embodiment of a multi-domain feature-level fusion expression recognition method based on a graph neural network;

[0032] Figure 6 The present invention is a schematic diagram of a feature fusion module guided by structural features in an embodiment of a multi-domain feature-level fusion expression recognition method based on a graph neural network. DETAILED DESCRIPTION

[0033] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0034] According to the characteristics of the motion trajectory of the facial feature points of macro expressions and micro expressions, it is found that compared with macro expressions, micro expressions have a shorter duration and lower intensity. Since the existing methods cannot detect macro expressions and micro expressions at the same time, the present invention provides a multi-domain feature-level fusion expression recognition method based on graph neural network, such as Figure 1 As shown, the following steps are included:

[0035] S1. First, data enhancement is performed on the original image, including image rotation, color transformation, image scaling and adding Gaussian noise.

[0036] Then, the enhanced image is subjected to facial feature point detection, and the facial region is divided into three sections and five eyes according to the position relationship of the feature points, for a total of 15 regions, such as Figure 2 shown.

[0037] Among them, the three parts refer to the length ratio of the face, which divides the length of the face into three equal parts, namely from the forehead hairline to the brow bone, from the brow bone to the bottom of the nose, and from the bottom of the nose to the chin, each of which accounts for 1 / 3 of the length of the face. The five eyes refer to the width ratio of the face, which is divided into five equal parts based on the length of the eye shape, from the left hairline to the right hairline, which contains the width of five eyes. The width of the face is divided into five equal parts, namely from the outer side of the left eye to the left hairline, the left eye area, between the left and right eyes, the right eye area, and from the outer side of the right eye to the right hairline.

[0038] S2: Based on the divided facial regions, use pixels as nodes of the graph structure , connect the nodes and The edge of the graph structure ,node and The weight between , using undirected weighted graph to build the graph structure of the face .node The neighborhood set of .

[0039] When the image quality is good, the facial feature point detection algorithm can be used to detect facial feature points and use the facial feature points as nodes, such as Figure 3 However, in general, due to the influence of posture and image quality, facial feature points cannot be detected. In this case, the pixel points are used as nodes, and the distance between nodes is used as edges. The graph structure can be established through edges and nodes, as shown in Figure 4 Using pixels as nodes gets rid of the limitation of feature point detection and improves versatility and universality.

[0040] However, when the image resolution is large, using the adjacency matrix to store the graph structure will increase the matrix dimension, resulting in increased memory overhead. Based on this, in this embodiment, for each node, only the nodes closest to the node are connected. Nodes, such as Figure 5 This is because when facial muscles move, they have a greater impact on their surroundings, but less impact on muscles farther away, which can effectively reduce the dimension of the matrix.

[0041] The process of constructing the adjacency matrix is ​​as follows:

[0042] First, take the pixel in the upper left corner of the image as the origin and establish a Euclidean coordinate system. When there are nodes, the Euclidean distance between a node and other nodes is calculated respectively.

[0043] Secondly, different weights are assigned to nodes between different connection areas. Specifically, the area is divided according to the three courts and five eyes, the Manhattan distance between the area where the current node is located and the area where the node to be connected is located is calculated, and then the corresponding weight is determined.

[0044] The expression of Manhattan distance is:

[0045] ;

[0046] In the formula, For Node With Node The Manhattan distance between , Node ,node The coordinates of .

[0047] like Figure 2 As shown in the figure, if node A is located in area X and node B is located in area Y, then the Manhattan distance is D. The weight calculation method between nodes A and B is: , multiply the weight by the Euclidean distance between the nodes to obtain the weighted Euclidean distance.

[0048] Then, the weighted Euclidean distances are sorted from small to large, and the adjacency matrix is ​​established based on the sorting results. The value of the adjacency matrix is ​​the weighted Euclidean distance between two feature points. For unconnected nodes, the distance is 0, and the expression is:

[0049] ;

[0050] In the formula, is the constructed adjacency matrix element; Representative Node The neighborhood set of , which can construct adjacency matrices of different sizes according to different K values, thereby controlling the complexity.

[0051] At this point, the node The degree matrix According to the obtained adjacency matrix and degree matrix, the spectral domain feature extraction of subsequent graph structures can be performed through graph neural networks.

[0052] S3. First, based on the adjacency matrix and graph structure, spectral domain features and spatial domain features are obtained through spectral domain feature extraction branch and spatial domain feature extraction branch respectively.

[0053] Specifically, in the spectral domain feature extraction branch, the graph structure is transformed from the spatial domain to the frequency domain through the graph Fourier transform, and then the spectral domain feature extraction is performed using the graph neural network. In the spatial domain feature extraction branch, the graph nodes and their adjacent nodes are convolved through the graph neural network to extract multi-scale spatial domain features. Multi-scale spatial domain features can give it a larger receptive field and reflect the changes in the distance between feature points under different expressions.

[0054] Next, a multi-level convolutional neural network is used to extract local features from the data-enhanced image. Specifically, it includes four stages: the first stage only contains the Conv module, and the second to fourth stages are composed of the Conv and MFEM (Multi-scale feature extraction module) modules. Among them, the Conv module consists of three parts: the convolution layer, the normalization layer, and the activation function. The functions of the three parts are to downsample the feature map, normalize, and perform nonlinear processing on the data; the function of the MFEM module is to propose multi-scale features. The network can effectively extract tiny local features to alleviate the impact of facial deformation, angle changes, and partial occlusion, improve the accuracy of expression recognition, and make the model suitable for complex scenes with background noise and lighting changes, with stronger robustness and stability.

[0055] Then, by extracting the contextual features of the image through ViT, the self-attention mechanism can directly model the relationship between the various parts of the image, effectively handle long-distance dependencies, and identify the interactions between different areas of the face (such as eyes, mouth, eyebrows, etc.), thereby improving the recognition accuracy.

[0056] S4, cascading the spectral domain features and the spatial domain features, i.e., feature-level fusion, to obtain the structural features; then, based on the structural features, a feature fusion network guided by structural features is used to perform multi-domain feature fusion, such as Figure 6 shown.

[0057] Specifically, the feature fusion network guided by structural features uses a scaled dot product attention module to capture the interaction relationship within the features, takes structural features, local features, and context features as input, and calculates the query matrix Q, key matrix K, and value matrix V, respectively. These matrices are obtained by linear transformation of the input feature vector and the learned weight matrix, as follows:

[0058] , , ;

[0059] In the formula, , , They are structural features, local features, and contextual features, respectively. , and are the corresponding transformation weight matrices respectively.

[0060] Next, the query matrix Q, key matrix K, and value matrix V are divided into multiple "heads" and the attention function is operated independently ( ), which is used to capture the global features between features and then realize multi-domain feature fusion, as follows:

[0061] ;

[0062] ;

[0063] In the formula, is the output weight matrix, For multi-domain feature fusion, For channel connection, For the The attention characteristics of the head, is the dimension of the key vector.

[0064] Then, the fused features are used as the input of the fully connected layer to obtain the expression recognition results.

[0065] Therefore, the present invention adopts the above-mentioned multi-domain feature-level fusion expression recognition method based on graph neural network, which fully considers the influence of different acquisition devices, different image resolutions, different lighting, occlusion and a small amount of training data on the recognition accuracy, and can accurately recognize expression and micro-expression images at the same time.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-domain feature-level fusion expression recognition method based on graph neural network, characterized in that: include: S1, perform data enhancement on the original image, then perform key point detection on the facial image, and segment the face into three parts and five eyes according to the key point detection results; S2. Based on the divided facial regions, an undirected weighted graph is used to construct a facial graph structure, and corresponding weights are assigned to each edge according to the region where the feature points are located, so as to obtain weighted distance information and store it in an adjacency matrix. S3, based on the adjacency matrix and the graph structure, spectral domain features and spatial domain features are obtained through the spectral domain feature extraction branch and the spatial domain feature extraction branch respectively; The local features of the data-enhanced image are extracted through a multi-level convolutional neural network on the original image; the contextual features of the data-enhanced image are extracted through ViT on the original image; S4, cascading the spectral domain features and the spatial domain features to obtain the structural features, using the feature fusion network guided by the structural features to fuse the multi-domain features, and introducing the self-attention module to calculate the attention function of each domain feature to obtain the fusion feature, and then obtain the expression recognition result; Among them, the feature fusion network guided by structural features includes taking structural features as query matrix Q, local features as key matrix K and context features as value matrix V, calculating the similarity between the query matrix and the key matrix through scaled dot product operation, and obtaining the weight coefficient of the value matrix, thereby obtaining the fused features.

2. According to the multi-domain feature-level fusion expression recognition method based on graph neural network in claim 1, it is characterized in that: Constructing an undirected weighted graph of the face includes using the image pixels of the divided facial region as nodes of the graph structure and the distances between the nodes as edges of the graph structure.

3. According to the multi-domain feature-level fusion expression recognition method based on graph neural network in claim 1, it is characterized in that: Assign corresponding weights to each edge according to the area where the feature points are located, including determining the corresponding weights according to the Manhattan distance between the area where the current node is located and the area where the node to be connected is located, as follows: ； ； In the formula, For Node With Node The Manhattan distance between , Node ,node The coordinates of For Node and nodes The weight of the edge between them.

4. The multi-domain feature-level fusion expression recognition method based on graph neural network according to claim 1 is characterized in that: The spectral domain feature extraction branch includes using graph Fourier transform to convert the image into the spectral domain, and then using a convolutional network to extract spectral domain features.

5. The multi-domain feature-level fusion expression recognition method based on graph neural network according to claim 1 is characterized in that: The spatial feature extraction branch includes using the spatial convolution model to perform convolution operations on the nodes of the graph structure and their adjacent nodes to obtain spatial features.

6. The multi-domain feature-level fusion expression recognition method based on graph neural network according to claim 1 is characterized in that: The multi-level convolutional neural network includes Conv module and MFEM module; Among them, the Conv module consists of three parts: convolution layer, normalization layer and activation function, which are used to downsample the feature map, normalize and perform nonlinear processing on the data respectively; The MFEM module is used to extract multi-scale features.

Images