Emotion recognition system based on graph attention inter-neighbor aggregation in missing mode

Abstract

The invention, which relates to the image processing field, discloses an emotion recognition system based on image attention inter-neighbor aggregation in a missing mode, comprising the following steps: acquiring a face image and voice of a user; the collected data are sent to the CPU processor and are stored; the processed data are sent to a graph network construction module, and two relational graphs are constructed according to the time sequence and the relation between different speakers; relation semantics are injected through a graph relation perception module, and a graph attention module carries out adaptive reweighting on neighbors; obtaining robust fusion modal representation through a graph inter-neighbor aggregation module; the modal information of the missing modal is drawn according to the available modal information through a reconstruction generation module; and outputting and displaying an identification result by using the classification network. According to the system, the recognition robustness and the complementation capability under the condition of mode missing are remarkably improved, the limitation that an existing emotion recognition system is not high in accuracy under the condition of mode missing is solved, and the application range of the emotion recognition system under the scene of multi-person conversation is widened.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an emotion recognition system based on graph attention inter-neighbor aggregation under missing modalities. BACKGROUND

[0002] Emotion recognition has a wide range of applications in human-computer interaction, mental health analysis, etc. In recent years, deep learning has received widespread attention, and its main advantage is that it can use a large amount of data set for training, thereby learning the most obvious features exhibited in these data.

[0003] Emotion recognition data is diverse, including speech, images, text, etc., which provides rich data for deep learning model training. However, in the real world, due to device failure, data damage, privacy issues, or occlusion in multi-person dialogue scenarios, missing modalities often occur, which may lead to a decrease in model performance.

[0004] Currently, multi-modal emotion recognition models trained on complete data often perform poorly when tested on incomplete data (i.e., missing modalities), with low accuracy. This limits the use of emotion recognition systems, especially in multi-person dialogue scenarios. Therefore, how to utilize existing modality information to handle missing modalities and improve the recognition robustness and completion ability of the model under modality missing conditions is crucial for the practical application of emotion recognition systems. SUMMARY

[0005] (I) Technical problems solved To address the shortcomings of the prior art, the present application provides an emotion recognition system based on graph attention inter-neighbor aggregation under missing modalities. The present application aims to solve the limitations of existing emotion recognition systems in terms of low accuracy under missing modalities, significantly improve the recognition robustness and completion ability under modality missing conditions, and increase the use of emotion recognition systems in multi-person dialogue scenarios.

[0006] (II) Technical solutions To achieve the above purpose, the present application provides an emotion recognition system based on graph attention inter-neighbor aggregation under missing modalities, comprising the following steps: S1: Collecting the facial images and speech of the user; S2: Sending the collected facial images and speech to the CPU processor for data processing; S3: Adding a pre-trained optimal model under the directory of the system; S4: Sending the processed data to the optimal model, constructing two relationship graphs of temporal relationship and speaker relationship through the graph network construction module, and injecting relationship semantics through the graph relationship perception module. S5: Adaptive reweighting of neighbors is performed through the graph attention module, and neighborhood aggregation is performed through the graph neighbor aggregation module to obtain a robust fusion modality representation; S6: Then, through the reconstruction generation module, the modal information of the missing modal is dynamically drawn based on all currently available modal information; S7: Use a classification network to output the recognition results and display the results in the display window; The optimal model in step S3 is obtained through the following training method: The first step is to preprocess the acquired facial images and speech to extract initial speech-level feature representations for the three modalities of vision, speech, and text. The second step is to build and train a network model. The specific training steps are as follows: (1) Multimodal feature extraction network: The starting point of the model is a multimodal feature extraction network, which is used to extract the initial feature representation at the discourse level from the preprocessed raw data. It contains three parallel branches that adapt to different modal characteristics. The branches include a two-dimensional convolution that processes the visual modality. The module includes two one-dimensional convolutions, one for speech and one for text, which handle the latter two modalities respectively. The module, with all branches, aims to capture the time-series dynamics of their respective modalities and aggregate them into a unified discourse-level representation. , as input to the graph network building module; (2) Graph Network Construction Module: The graph network construction module is used to convert serialized features into structured graph data and receive the utterance-level representation output by the previous module. Valid bit mask Speaker markers First, filter out invalid statements. It is then expanded into a unified intra-batch node matrix. Next, the module constructs two multi-relationship graphs within each dialogue segment: a time graph. The temporal dependence of discourse is captured through three types of directed edges: self-loops, past to present, and present to future; and a speaker diagram is also provided. According to the speaker ,Right now Establish as many as A relation edge is used to simulate the dynamic interaction in a dialogue, and optionally subject to a time window. Constraints, ultimately outputting shared node features X And two sets of diagram structure information ( ); (3) Graph Relationship Awareness Module: The graph relationship awareness module uses nodes X As input, respectively in the time graph and speaker graph Convolution operation with relation-specificity is performed on the upper graph The update rule of the node of the first layer is defined as: , where is the activation function, is the degree normalization term, is the relation-specific projection matrix; the process produces two relation-aware representations , which are fused to obtain ; ; (4) Graph attention inter-neighbor aggregation module: the graph attention part can realize adaptive reweighting, and the representation is sent into a graph attention layer , which encodes the type of edge as attention attribute, and through the multi-head attention mechanism: dynamically learns the importance of different neighbors, and finally fuses the results of the two graphs and , and stabilization processing to obtain ; the graph inter-neighbor aggregation part is responsible for consistency refinement of the output of the graph attention module, which receives , and performs a permutation-invariant aggregation operation on the neighborhood of each node : to collect the statistical information of the neighbor nodes; and through an update function: combine the representation of the node itself with the aggregation information of its neighborhood, after stacking multiple layers, output the final refined node representation ; (5) Reconstruction generation and classification module: the final reconstruction generation and classification module is responsible for generating predictions and handling missing modalities; on the one hand, it concatenates the refined features of the graph network with the original node features into , and restores it to a sequence form , and then sends it into the classification head to calculate the main task loss ; on the other hand, the reconstruction generation network is introduced as an auxiliary task, which uses a learnable generation prompt, a missing signal label, and a cross-modal to explicitly generate missing modal information; the reconstruction loss of the generation network is denoted as , which is combined with the main task loss by weighting: , so as to improve the robustness of the model to missing modalities in joint optimization, and use the classification head to make predictions during inference.

[0007] As optimization, in the first step of preprocessing the collected facial images and voice, the coordinates of key points such as eye corners, nose tips, mouth corners and eyebrow contours are accurately located for each facial image using the OpenCV tool; the perceptual characteristics of the tone are obtained by extracting the MFCCs spectrum of the voice; the voice is converted into text, the pre-trained Universal Sentence Encoder embedding model is used to convert the text into embedding vectors, and the data of the three modalities are segmented into the same segments according to the time stamp in the text.

[0008] As optimization, in the second step of constructing and training the network model, the total number of four-fifths of the multi-modal data is input into the network model as a training sample to train the network structure module, and the trained network is obtained; the remaining one-fifth of the data is used as a test sample, and the data is input into the trained network to test the accuracy of the model.

[0009] As optimization, in step S4, the graph network construction module is responsible for constructing two multi-relation graphs: a time graph that captures the temporal dependence of the speech through three directed edges of self-loop, past-to-present and present-to-future; and a speaker graph that establishes up to K kinds of relationship edges according to the relationship between different speakers to simulate the interactive dynamics in the dialogue.

[0010] As optimization, in step S5, the neighbors are adaptively reweighted through the graph attention module; then the neighborhood aggregation is performed through the graph inter-neighbor aggregation module to obtain a robust fused modal representation.

[0011] As optimization, in step S6, the data passes through the reconstruction generation module, which dynamically draws the modal information of the missing modal according to all available modal information at present.

[0012] (Three) beneficial effects The present application provides a depression risk assessment system based on prompt learning under missing modal, which has the following beneficial effects: The present application models the "time-speaker" double relationship through the graph network construction module, dynamically learns the neighbor node weight through the graph attention inter-neighbor aggregation mechanism, fully excavates the multi-modal associated semantics, and makes the emotion recognition accuracy in the missing scene significantly improved through the joint optimization of classification and reconstruction loss in the reconstruction generation module; supports multi-person dialogue scene, and can accurately capture the emotion transmission in the dialogue by means of the "self-connected edge-interconnected edge" design and time window constraint of the speaker graph, and is compatible with single-person interactive scene, and only needs to set the speaker mark as a fixed value to automatically adapt to scene switching. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The flowchart of the emotion recognition system of the present application.

[0014] Figure 2 A framework diagram of the emotion recognition system of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0016] Please refer to Figure 1 and Figure 2 , the present application provides a technical solution: As Figure 2 shown, model training and emotion recognition are completed in the following steps: First, pre-process the collected facial images and speech: use the OpenCV tool to accurately locate the coordinates of key points such as eye corners, nose tips, mouth corners, and eyebrow contours in each facial image; extract the MFCCs spectrum of the speech to obtain the perceptual characteristics of the tone; convert the speech into text, convert the text into an embedding vector using a pre-trained Universal Sentence Encoder (USE) embedding model, and divide the data of the three modalities into the same segments according to the time stamp in the text.

[0017] Second, optimal network training: construct and train a network model, input the total number of four-fifths of the multi-modal data as training samples into the network model, train the network structure module, and obtain the trained network; input the remaining one-fifth of the data as test samples into the trained network to test the accuracy of the model.

[0018] The specific steps of training the network structure module are as follows: (1) Multi-modal feature extraction network: the starting point of the model is a multi-modal feature extraction network, which is responsible for extracting speech-level initial feature representation from the pre-processed original data, including three parallel branches that adapt to different modal characteristics: a two-dimensional convolution module for processing the visual modality, and two one-dimensional convolution modules for processing the speech and text modalities, respectively. All branches aim to capture the time series dynamics of the respective modalities, and output unified speech-level representation as input for the graph network construction module.

[0019] (2) Graph network construction module: the subsequent graph network construction module is responsible for converting the serialized features into structured graph data. It receives the speech-level representation , valid bit mask and speaker label , first filter out invalid utterances ( ) and unfold them into a unified intra-batch node matrix . Then, the module builds two multi-relational graphs within each dialogue: a temporal graph , which captures the temporal dependencies of utterances through three kinds of directed edges: self-loop, past now and future , and a speaker graph , which builds up to kinds of relational edges according to speaker ( ) to simulate the interactive dynamics in dialogues and optionally constrained by a time window . The final output shares the node features X and two sets of graph structure information ( ).

[0020] (3) Graph relation-aware module: The graph relation-aware module takes node X as input and performs relation-specific convolution operations on the temporal graph and speaker graph respectively. For the th layer of any graph , the update rule of node is defined as: , where is the activation function, is the degree normalization term, is the relation-specific projection matrix. The process produces two relation-aware representations , which are fused to obtain .

[0021] (4) Graph attention inter-neighbour aggregation module: The graph attention part can achieve adaptive re-weighting, and the representation is sent into a graph attention layer , which encodes the types of edges as attention properties, through a multi-head attention mechanism: dynamically learns the importance of different neighbours, and finally fuses the results of the two graphs and goes through , and stabilization processing to obtain . The graph inter-neighbour aggregation part is responsible for consistency refinement of the output of the graph attention module. This module receives , and performs a permutation-invariant aggregation operation on the neighbourhood of each node : , to collect the statistics of neighbor nodes. Through an update function: Combining the representation of the node itself with the aggregated information of its neighborhood, after stacking multiple layers, the final refined node representation is output .

[0022] (6) Reconstruction generation and classification module: the final reconstruction generation and classification module is responsible for generating predictions and handling modality missing. On the one hand, it concatenates the refined features of the graph network with the original node features into , and restores them to a sequence form , and then sends them to the classification head to calculate the main task loss . On the other hand, the reconstruction generation network is introduced as an auxiliary task, which uses a learnable generation prompt, a missing signal label and cross-modal to explicitly generate missing modal information. The reconstruction loss of the generation network is denoted as , which is combined with the main task loss with a weight: so as to improve the robustness of the model to missing modalities in joint optimization, and use the classification head to make predictions during inference.

[0023] Based on the above steps, as shown in Figure 1 , an emotion recognition system based on graph attention inter-neighbor aggregation under missing modalities, the specific operation steps are as follows: S1: Collect the face image and voice of the user; S2: Send the collected face image and voice to the CPU processor for data processing; S3: Add the optimal model obtained by training under the directory of the system; S4: Send the processed data to the graph network construction module to construct two relationship graphs of time relationship and speaker relationship; inject relationship semantics through the graph relationship perception module; S5: The neighbors are adaptively reweighted through the graph attention module; the neighborhood aggregation is performed through the graph inter-neighbor aggregation module to obtain a robust fusion modal representation; S6: Then, through the reconstruction generation module, the modal information of the missing modal is dynamically drawn according to all available modal information; S7: The recognition result is output by the classification network and displayed on the display window.

[0024] In the training process, the model gradually learns the correlation rules between multi-modalities and the generation logic of the missing modalities through joint optimization; during inference, even if there is modal missing, the reconstruction generation module can dynamically draw the modal information of the missing modal according to all available modal information, ensuring the accuracy and stability of emotion recognition.

[0025] While embodiments of the application have been shown and described, it is to be understood that the application is not limited to the details of the embodiments described, since numerous changes, modifications, substitutions and variations can be made thereto without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. An emotion recognition system based on graph attention and neighbor aggregation in the absence of modalities, characterized in that: Includes the following steps: S1: Collects the user's facial image and voice; S2: The captured facial images and voice data are sent to the CPU processor for data processing; S3: Add a pre-trained optimal model to the directory of this system; S4: The processed data is fed into the optimal model, and two relationship graphs, time relationship and speaker relationship, are constructed through the graph network construction module. Relationship semantics are injected through the graph relationship perception module. S5: Adaptive reweighting of neighbors is performed through the graph attention module, and neighborhood aggregation is performed through the graph neighbor aggregation module to obtain a robust fusion modality representation; S6: Then, through the reconstruction generation module, the modal information of the missing modal is dynamically drawn based on all currently available modal information; S7: Use a classification network to output the recognition results and display the results in the display window; The optimal model in step S3 is obtained through the following training method: The first step is to preprocess the acquired facial images and speech to extract initial speech-level feature representations for the three modalities of vision, speech, and text. The second step is to build and train a network model. The specific training steps are as follows: (1) Multimodal feature extraction network: The starting point of the model is a multimodal feature extraction network, which is used to extract the initial feature representation at the discourse level from the preprocessed raw data. It contains three parallel branches that adapt to different modal characteristics. The branches include a two-dimensional convolution that processes the visual modality. The module includes two one-dimensional convolutions, one for speech and one for text, which handle the latter two modalities respectively. The module, with all branches, aims to capture the time-series dynamics of their respective modalities and aggregate them into a unified discourse-level representation. , as input to the graph network building module; (2) Graph Network Construction Module: The graph network construction module is used to convert serialized features into structured graph data and receive the utterance-level representation output by the previous module. Valid bit mask Speaker markers First, filter out invalid statements. It is then expanded into a unified intra-batch node matrix. Next, the module constructs two multi-relationship graphs within each dialogue segment: a time graph. The temporal dependence of discourse is captured through three types of directed edges: self-loop, past to present, and present to future. And a picture of the speaker. According to the speaker ,Right now Establish as many as A relation edge is used to simulate the dynamic interaction in a dialogue, and optionally subject to a time window. Constraints, ultimately outputting shared node features X And two sets of diagram structure information ( ); (3) Graph Relationship Awareness Module: The graph relationship awareness module uses nodes X As input, respectively in the time graph And speaker diagram Perform relation-specific convolution operations on any graph. The Layers, nodes The update rule is defined as follows: ,in, For activation function, It is a degree normalization term. It is a relation-specific projection matrix; the process generates two relation-aware representations. , obtained through fusion ; (4) Graph Attention Neighbor Aggregation Module: The graph attention part can achieve adaptive reweighting and fusion representation. Feed into a graph attention layer This layer encodes the edge type as an attention attribute, using a multi-head attention mechanism: The importance of different neighbors is dynamically learned, and the results of the two graphs are ultimately merged and processed. , and Stabilization treatment yielded The graph neighbor aggregation section is responsible for refining the consistency of the output of the graph attention module. This module receives... and for each node neighborhood Perform an aggregation operation that does not change the substitution: This is done to collect statistical information from neighboring nodes; and through an update function: By combining the node's own representation with the aggregated information of its neighborhood, and stacking multiple layers, the final refined node representation is output. ; (5) Reconstruction Generation and Classification Module: The final reconstruction generation and classification module is responsible for generating predictions and handling modality loss; on the one hand, it will generate the features refined by the graph network. Compared with the original node features spliced ​​as and restore it to its sequence form. Then it is sent to the classification head to calculate the main task loss. On the other hand, a reconstruction-generative network was introduced as an auxiliary task, which utilizes learnable generative cues, missing signal labels, and cross-modal features. To explicitly generate the missing modal information; the reconstruction loss of this generative network is denoted as... Combined with the main task loss weighting: This improves the model's robustness to missing modalities during joint optimization and allows the use of a classification head for prediction during inference.

2. The emotion recognition system based on graph attention neighbor aggregation in a missing modality according to claim 1, characterized in that: In the first step of preprocessing the acquired facial images and speech, OpenCV tools were used to accurately locate the coordinates of key points such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the outlines of the eyebrows in each facial image; the MFCC spectrum of the speech was extracted to obtain the perceptual characteristics of the timbre; the speech was converted into text, and the pre-trained Universal Sentence Encoder embedding model was used to convert the text into embedding vectors, and the data of the three modalities were segmented into the same segments according to the timestamps in the text.

3. The emotion recognition system based on graph attention neighbor aggregation in a missing modality according to claim 1, characterized in that: In the second step of constructing and training the network model, four-fifths of the total multimodal data are used as training samples to input into the network model to train the network structure modules and obtain the trained network; the remaining one-fifth of the data is used as test samples and input into the trained network to test the accuracy of the model.

4. The emotion recognition system based on graph attention neighbor aggregation in a missing modality according to claim 1, characterized in that: In step S4, the graph network construction module is responsible for constructing two multi-relationship graphs: a time graph, which captures the temporal dependencies of discourse through three types of directed edges: self-loops, past to present, and present to future; It also includes a speaker graph, which establishes up to K types of relationship edges based on the relationships between different speakers to simulate the interactive dynamics in a dialogue.

5. The emotion recognition system based on graph attention neighbor aggregation in a missing modality according to claim 1, characterized in that: In step S5, the neighbors are adaptively reweighted by the graph attention module; then, the neighborhood aggregation is performed by the graph neighbor aggregation module to obtain a robust fusion modality representation.

6. The emotion recognition system based on graph attention neighbor aggregation in a missing modality according to claim 1, characterized in that: In step S6, the data is processed by the reconstruction generation module, which dynamically draws the modal information of the missing modal based on all currently available modal information.

Images