Multi-modal emotion recognition system and method based on large language model enhancement

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a multi-mode emotion recognition system and method based on big language model enhancement, and the method comprises the steps: extracting conversation-level features of an audio mode, a text mode and a visual mode from a conversation; performing feature extraction by using a large language model, and constructing an alignment enhancement embedding representation of corresponding modal optimization by using shallow embedding representations of an audio modal and a visual modal; further decoupling to obtain shared features and specific features of each modal, and uniformly projecting the shared features of each modal to a text semantic space; and finally, combining the specific features of each modal with the projected shared features, and outputting an emotion prediction result. According to the multi-modal emotion recognition method, semantic comprehension and cross-modal alignment capability of a large language model are fully utilized, consistency and accuracy of multi-modal emotion recognition are improved through accurate representation alignment and fusion, and the multi-modal emotion recognition method can be widely applied to scenes such as intelligent customer service, emotion analysis, virtual assistants and human-computer interaction and has important practical value and application prospect.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to affective computing, multimodal learning and large language model application technology, and in particular to a multimodal affective recognition system and method based on large language model enhancement. Background Technology

[0002] With the development of artificial intelligence technology, affective computing, especially multimodal affective recognition, has become an important research direction in the field of affective analysis. Multimodal affective recognition aims to extract and fuse affective information from multiple modalities (such as text, audio, and visual) to improve the accuracy and robustness of affective recognition. Compared with traditional single-modal affective recognition methods, multimodal affective recognition tasks can make full use of complementary information from different modalities, thereby significantly improving the effect of affective recognition. Especially in dialogue systems, affective recognition not only relies on the content of the text, but also needs to consider the intonation of speech, changes in emotional tone, and visual information such as facial expressions.

[0003] However, existing multimodal emotion recognition methods often focus on aligning consistency information between modalities, neglecting the specific cues in emotional expression inherent in each modality. For example, sarcastic language in text, pitch variations in speech, and micro-expressions in facial features are all modality-specific emotional information that traditional methods have failed to fully utilize. Furthermore, existing multimodal fusion methods typically employ contrastive learning and cross-modal attention mechanisms, but these methods mostly focus only on shared emotional features between modalities, failing to effectively capture modality-specific emotional details, thus limiting the accuracy and robustness of emotion recognition.

[0004] MGCMA (Multimodal Graph Convolutional Matching Attention) methods align audio, text, and visual information through contrastive learning, leveraging the similarity between the three modalities for feature learning. These methods excel at extracting consistent emotional information across different modalities, particularly focusing on the extraction and fusion of shared features. SAC-CMA employs a cross-modal attention mechanism, aiming to capture emotional information by enhancing the correlation between modalities; this approach improves emotional recognition accuracy by maximizing intermodal consistency. Joyful methods extract modal-shared features through a global context fusion module and combine it with a specific modality fusion module to extract modality-specific emotional information; this method attempts to address the problem of extracting modality-specific emotional information through global feature fusion.

[0005] These methods typically employ contrastive learning or cross-modal attention mechanisms to align emotional information across different modalities, attempting to improve the accuracy of emotion recognition through a unified semantic space. However, these methods often neglect the unique emotional information specific to each modality, leading to information loss during alignment, such as intonation variations in speech, micro-expressions in facial expressions, and sarcastic language in text. This unique emotional information represents the distinctive way each modality expresses emotion and is crucial for the accuracy of emotion recognition. Moreover, these methods lack explicit supervision mechanisms.

[0006] Methods based on cross-modal attention mechanisms require significant computational resources for consistency alignment operations between modalities. These methods typically involve handling numerous feature alignment and weighting operations, resulting in high computational complexity. Consequently, they fail to meet real-time requirements in practical applications, especially in dialogue systems and sentiment analysis scenarios that demand rapid responses, leading to slow response times.

[0007] Furthermore, current multimodal emotion recognition methods often mix shared features and modality-specific features, failing to effectively decouple them. This makes it difficult for the model to capture the unique emotional information of each modality when fusing multimodal information, and may also lead to excessive mixing of shared features. The inability to effectively separate shared features from modality-specific features affects the independence of feature representation and the accurate transmission of emotional information, thus impacting the accurate extraction of emotional information. If modality-specific emotional features are not effectively preserved, the final emotion recognition result will lose some important emotional details. In addition, the failure to effectively orthogonally decouple and projectively align features results in a certain degree of complexity and information loss in modality feature fusion, failing to reduce intermodal differences, and therefore performing worse than expected in cross-modal alignment.

[0008] Therefore, existing multimodal emotion recognition methods often overlook the unique emotional cues in each modality (such as text, speech, and vision) when processing different modalities, such as intonation variations in speech, micro-expressions in facial expressions, and sarcastic language in text. These emotional cues are crucial for the accuracy and robustness of emotion recognition, but traditional methods mainly focus on mining information shared between modalities, failing to effectively preserve and utilize the unique emotional features of each modality. Summary of the Invention

[0009] The purpose of this invention is to provide a multimodal emotion recognition system based on a large language model. By embedding the large language model into the feature extraction and alignment process, and combining orthogonal decoupling strategy and projection alignment, it can effectively preserve and separate modality-specific emotional information, while enhancing the semantic consistency between modalities. This significantly improves the accuracy and robustness of emotion recognition and solves the problems of insufficient feature alignment, insufficient modality decoupling, and loss of emotional information in the prior art.

[0010] The present invention also provides a multimodal emotion recognition method based on Large Language Model (LLM) enhancement.

[0011] To achieve the above objectives, the present invention employs the following technical solutions.

[0012] This invention provides a multimodal emotion recognition system based on a large language model enhancement, comprising:

[0013] A single-modal dialogue-level feature extraction module is used to extract dialogue-level features from the audio modality, text modality, and visual modality of the dialogue, respectively.

[0014] The LLM large language model is used to extract shallow features from audio modal dialogue-level features and visual modal dialogue-level features to obtain shallow embedding representations of the corresponding modalities, and to extract deep features from text modal dialogue-level features to obtain deep embedding representations of the text modalities.

[0015] The encoding module includes a shared encoder, an audio-specific encoder, a text-specific encoder, and a visual-specific encoder. The shared encoder takes the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, and encodes the shared features of the corresponding modalities. The audio-specific encoder, text-specific encoder, and visual-specific encoder encode the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality separately, obtaining the specific features of the corresponding modalities. The alignment-enhanced embedding representation of the audio modality is obtained by superimposing the shallow embedding representation of the audio modality with the corresponding dialogue-level features; the alignment-enhanced embedding representation of the visual modality is obtained by superimposing the shallow embedding representation of the visual modality with the corresponding dialogue-level features.

[0016] The projection module is used to project the shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality onto the text space to obtain the projected shared features of the corresponding modality.

[0017] The decoding module includes an audio decoder, a text decoder, a visual decoder, and a text-specific decoder. The audio decoder, text decoder, and visual decoder reconstruct corresponding modal features based on shared features and specific features of each modality, respectively. The text-specific decoder reconstructs text modality-specific features based on the projection shared features of the audio modality and the projection shared features of the visual modality.

[0018] A classifier is used to predict sentiment based on specific features of each modality and projection-shared features.

[0019] In one possible implementation, the single-modal dialogue-level feature extraction module includes an audio feature extraction unit, a text feature extraction unit, and a visual feature extraction unit. The audio feature extraction unit includes an audio encoder and an audio projector. The audio encoder uses a Whisper-Large model to extract audio features from the dialogue, and then the audio projector obtains audio modal dialogue-level features. The text feature extraction unit includes a text encoder and a fully connected layer. The text encoder uses a Baichuan-13B model to extract text features from the dialogue, and then the fully connected layer further processes these features to obtain text modal dialogue-level features. The visual feature extraction unit includes a visual encoder and a visual projector. The visual encoder uses a CLIP-ViT-Large model to extract visual features from the dialogue, and then the visual projector obtains visual modal dialogue-level features. The audio projector and the visual projector have the same structure, including two layers of multilayer perceptrons (MLPs) and a GELU activation function arranged sequentially.

[0020] In one possible implementation, for both audio and visual modalities, shallow embedding feature extraction is performed using the shallow layers of a Large Language Model (LLM) (serving as supervisory layers, the lower half of the LLM, typically layers 1 to 16) to construct supervisory signals, gradually improving system performance. The features extracted from each shallow layer are weighted and fused to obtain the shallow embedding representation for the corresponding modality. This shallow embedding representation can be used to construct alignment supervisory signals to optimize the audio and visual projectors, resulting in an alignment-enhanced embedding representation. For the text modality, deep embedding feature extraction is performed using the Large Language Model, with the final output serving as the deep embedding representation for the text modality.

[0021] In one possible implementation, the shared encoder, audio-specific encoder, text-specific encoder, and vision-specific encoder have the same structure, each consisting of two fully connected layers.

[0022] In one possible implementation, the audio decoder, text decoder, visual decoder, and text-specific decoder have the same structure, each consisting of two fully connected layers.

[0023] In one implementation, the classifier comprises a fully connected layer, two multilayer perceptron (MLP) layers, and a ReLU activation function arranged sequentially.

[0024] This invention also provides a training method for a multimodal emotion recognition system based on a large language model enhancement, which includes the following steps:

[0025] The single-modal dialogue-level feature extraction module extracts dialogue-level features from the dialogue for audio modality, text modality, and visual modality respectively.

[0026] The LLM large language model is used to extract shallow features from audio and visual dialogue-level features to obtain shallow embedding representations for the corresponding modalities, and to extract deep features from text dialogue-level features to obtain deep embedding representations for the text modalities. The shallow embedding representations of the audio modal are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the audio modalities; the shallow embedding representations of the visual modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the visual modalities.

[0027] Using the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, the shared features of the corresponding modalities are encoded separately through a shared encoder; the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality are encoded separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder, respectively, to obtain the specific features of the corresponding modalities.

[0028] The shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality are projected onto the text space through the projection module to obtain the projected shared features of the corresponding modality.

[0029] The corresponding modal features are reconstructed by the audio decoder, text decoder, and visual decoder based on the shared features and specific features of each modality; the specific features of the text modality are reconstructed by the text-specific decoder based on the projection shared features of the audio modality and the projection shared features of the visual modality.

[0030] Based on the specific features of each modality and the projection-shared features, a classifier is used to predict sentiment, and the sentiment prediction results are obtained.

[0031] The audio embedding alignment loss is constructed based on the alignment enhancement embedding representation of the audio modality and the dialogue-level features, and the visual embedding alignment loss is constructed based on the alignment enhancement embedding representation of the visual modality and the dialogue-level features. The audio embedding alignment loss and the visual embedding alignment loss together constitute the total embedding alignment loss.

[0032] Decoupling loss is constructed based on the shared and specific features of each modality, and reconstruction loss is constructed based on the alignment-enhanced embedding representation of the audio modality and the visual modality, the deep embedding representation of the text modality, and the corresponding modal features of the reconstruction. The decoupling loss and the reconstruction loss together form an orthogonal decoupling loss.

[0033] A cross-modal reconstruction loss is constructed based on the modal-specific features of the reconstructed text and the deep embedding representation of the text modality;

[0034] A classification loss is constructed based on the sentiment prediction results and the corresponding sentiment labels of the dialogue.

[0035] The alignment loss, orthogonal decoupling loss, cross-modal reconstruction loss, and classification loss constitute a joint objective function. By optimizing the system parameters through the joint objective function, a well-trained multimodal emotion recognition system is obtained.

[0036] This invention also provides a multimodal emotion recognition method based on a large language model enhancement, which uses a trained multimodal emotion recognition system and follows these steps:

[0037] The single-modal dialogue-level feature extraction module extracts dialogue-level features from the dialogue for audio modality, text modality, and visual modality respectively.

[0038] The LLM large language model is used to extract shallow features from audio and visual dialogue-level features to obtain shallow embedding representations for the corresponding modalities, and to extract deep features from text dialogue-level features to obtain deep embedding representations for the text modalities. The shallow embedding representations of the audio modal are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the audio modalities; the shallow embedding representations of the visual modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the visual modalities.

[0039] Using the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, the shared features of the corresponding modalities are encoded separately through a shared encoder; the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality are encoded separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder, respectively, to obtain the specific features of the corresponding modalities.

[0040] The shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality are projected onto the text space through the projection module to obtain the projected shared features of the corresponding modality.

[0041] Based on the specific features of each modality and the shared features of projection, a classifier is used to predict sentiment, and the sentiment prediction results are obtained.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Specifically, the objectives of this invention can be explained from the following aspects:

[0044] 1) This invention refines and aligns audio and visual modal features by introducing a large language model (LLM); by introducing fine-grained supervised embedding and directional alignment supervision mechanisms, it significantly improves the feature alignment effect between audio, text and visual modalities, ensures semantic consistency between modalities, and enables audio, text and visual modalities to express emotional information more collaboratively in emotion recognition.

[0045] 2) This invention proposes an orthogonal decoupling strategy, which introduces orthogonal constraints by setting a shared encoder and a single-modal specific encoder to ensure the independence between shared features and specific features. This strategy can minimize the similarity between shared features and specific features, and minimize information redundancy. Furthermore, it maintains the integrity of emotion-related information in the decoupling process through single-modal reconstruction loss, ensuring that each modality can independently express emotion-related information, thereby achieving effective capture of emotions and improving the accuracy and robustness of emotion recognition, especially in complex dialogue and multimodal scenarios.

[0046] 3) This invention proposes a cross-modal projection and alignment strategy, specifically projecting shared features onto a common text space and reconstructing text modal features through a text-specific decoder, enabling better semantic alignment of shared features across different modalities and enhancing cross-modal consistency. This method can effectively improve the semantic consistency between audio, visual, and text modalities, allowing the system to better understand and integrate information from different modalities, thereby improving the robustness and accuracy of emotion recognition in varied contexts.

[0047] 4) This invention further improves the accuracy of sentiment prediction by combining specific features of audio, text and visual modalities with context-enhanced projection-shared features as input to the classifier; in this way, the system can better understand and process sentiment information of different modalities, and improve the robustness of sentiment recognition in different contexts.

[0048] 5) This invention designs a joint objective function, including multiple loss terms such as orthogonal decoupling loss, directional alignment supervision loss, and cross-modal reconstruction loss, to comprehensively optimize the performance of the multimodal emotion recognition model. This ensures the optimization of the multimodal emotion recognition model at multiple levels, enabling the system to better retain emotional information and improve accuracy during the effective fusion of features from various modalities.

[0049] Therefore, this invention combines the powerful semantic understanding capabilities of Large Language Models (LLM) to extract emotional features from multiple modalities such as speech, text, and vision. Through precise feature alignment and fusion methods, it improves the consistency and accuracy of multimodal information. It can be widely used in fields such as intelligent customer service, sentiment analysis, virtual assistants, and human-computer interaction, and has important practical value and broad application prospects. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the principle of a multimodal emotion recognition system based on a large language model.

[0051] Figure 2 This is a schematic diagram of the training process for a multimodal emotion recognition system enhanced by a large language model.

[0052] Figure 3 This is a schematic diagram of the process of a multimodal emotion recognition method based on a large language model.

[0053] Figure 4 t-SNE graphs for the multimodal emotion recognition system based on large language model enhancement with and without the present invention. Detailed Implementation

[0054] The technical solutions of various embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1

[0056] This embodiment provides a multimodal emotion recognition system based on large language model enhancement (LDPA-SA), the principle block diagram of which is shown below. Figure 1 As shown, it includes: a single-modal dialogue-level feature extraction module, an LLM large language model, an encoding module, a projection module, a decoding module, and a classifier.

[0057] 1. Single-modal dialogue-level feature extraction module

[0058] The single-modal dialogue-level feature extraction module is used to extract dialogue-level features from the audio modality, text modality, and visual modality of the dialogue.

[0059] The single-modal dialogue-level feature extraction module includes an audio feature extraction unit, a text feature extraction unit, and a visual feature extraction unit. The audio feature extraction unit includes an audio encoder and an audio projector. The audio encoder uses a Whisper-Large model to extract audio features from the dialogue, which are then processed by the audio projector to obtain audio-modal dialogue-level features. The text feature extraction unit includes a text encoder and a fully connected layer. The text encoder uses a Baichuan-13B model to extract text features from the dialogue, which are then further processed by the fully connected layer to obtain text-modal dialogue-level features. The visual feature extraction unit includes a visual encoder and a visual projector. The visual encoder uses a CLIP-ViT-Large model to extract visual features from the dialogue, which are then processed by the visual projector to obtain visual-modal dialogue-level features. The audio projector and visual projector have the same structure, including two layers of multilayer perceptrons (MLPs) and a GELU activation function arranged sequentially.

[0060] In the specific implementation, for those containing A dialogue of words (Right now The single-modal dialogue-level feature extraction module described above extracts single-modal dialogue-level features for each modality. Specifically, audio features are extracted from the dialogue using Whisper-Large, and then audio modal dialogue-level features are obtained using an audio projector. Text features are extracted using the Baichuan-13B model and then further processed through fully connected layers to obtain text modality dialogue-level features. Visual features are extracted from the dialogue using CLIP-ViT-Large, and then visual modal dialogue-level features are obtained through a visual projector. . Indicates dialogue The Middle The single-modal features of a discourse, among which, Indicates the modality type (audio, text, visual).

[0061] 2. LLM Large Language Model

[0062] The LLM large language model is used to perform shallow feature extraction on audio modality dialogue-level features and visual modality dialogue-level features to obtain shallow embedding representations of the corresponding modalities, and to perform deep feature extraction on text modality dialogue-level features to obtain deep embedding representations of the text modality.

[0063] In this embodiment, the Large Language Model (LLM) specifically used is Vicuna-v1.5-7B. This embodiment utilizes refined audio and visual embeddings from the shallow layers of the LLM to construct supervised audio and visual embeddings. The shallow layers, used for supervision, are the lower half of the LLM, typically layers 1 to 16. Using refined audio and visual embeddings from layers 1 to 16 to construct supervisory signals gradually improves model performance. The deep layers are the upper half of the LLM, typically layers 16 to 32. For both audio and visual modalities, the features extracted from each shallow layer are weighted and fused to obtain the corresponding shallow embedding representation, which is used to optimize the output of the audio and visual projectors to obtain an aligned and enhanced embedding representation. For the text modality, deep embedding features are extracted using the LLM, with the final output serving as the deep embedding representation for the text modality.

[0064] The final supervised audio embedding (i.e., the shallow embedding representation of the audio modality) is a weighted sum of the shallow refined audio embeddings of these LLMs, calculated as follows:

[0065] ; ;

[0066] in, This represents the final shallow audio embedding, with a dimension of . L is the number of audio frames. It is the embedded dimension. Let represent the refined audio embedding of the i-th layer of the LLM, and k is the total number of shallow LLM layers used to construct the supervisory signal, where k=16. Indicates assignment to the first Layer weights.

[0067] Similarly, for visual modalities, the formula for calculating the constructed supervised visual embedding (i.e., the shallow embedding representation of the visual modality) is as follows:

[0068] ; ;

[0069] in, This represents the final visual shallow embedding representation. This represents the refined visual embedding at the i-th layer of the LLM. Shallow audio embedding representation. and shallow visual embedding representation It can be used to construct alignment supervision signals to optimize the audio projector and the visual projector, resulting in a corresponding alignment-enhanced embedding representation. and .

[0070] ; ;

[0071] For text modalities, the final output of the large language model is used as the deep embedding representation of the text modality, which is... .

[0072] 3. Encoding Module

[0073] The encoding module includes a shared encoder, an audio-specific encoder, a text-specific encoder, and a vision-specific encoder. The shared encoder takes the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, and encodes the shared features of the respective modalities. The audio-specific encoder, text-specific encoder, and vision-specific encoder encode the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality separately, obtaining the specific features of the respective modalities.

[0074] In specific implementations, the shared encoder, audio-specific encoder, text-specific encoder, and vision-specific encoder have the same structure, all consisting of two fully connected layers.

[0075] The shared encoder is used by all modalities to capture consistent semantics across modalities. For each modality, its shared features are represented as follows:

[0076] ;

[0077] in, Indicates a shared encoder. This represents the shared features of mode m. Parameters representing the shared encoder.

[0078] Audio-specific encoders, text-specific encoders, and vision-specific encoders extract specific features for each modality separately to preserve the specific semantics of that modality, as follows:

[0079] ;

[0080] in, This represents a specific encoder for mode m. This represents a specific characteristic of mode m. This represents the parameters of the encoder specific to the corresponding mode.

[0081] 4. Projection module

[0082] For shared spatial feature projection, this embodiment selects the text modality as a reference, projecting the shared features of the visual and audio modalities into the text space. Therefore, this embodiment uses a projection module to project the shared features of the audio modality, the visual modality, and the text modality into the text space to obtain the corresponding modal's projected shared features.

[0083] In the specific implementation, the calculation formula for projecting the shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality onto the text space is as follows:

[0084] ;

[0085] in, Represents the learnable projection matrix. , These represent shared features projected from the audio and visual modalities, respectively.

[0086] 5. Decoding module,

[0087] The decoding module is primarily used to calculate the reconstruction loss. It includes an audio decoder, a text decoder, a visual decoder, and a text-specific decoder. The audio decoder, text decoder, and visual decoder reconstruct the corresponding modal features based on the shared and specific features of each modality. The text-specific decoder reconstructs the text modality-specific features based on the projected shared features of the audio modality and the projected shared features of the visual modality. The audio decoder, text decoder, visual decoder, and text-specific decoder have the same structure, each consisting of two fully connected layers.

[0088] In the specific implementation, the audio decoder, text decoder, and visual decoder reconstruct the original single-modal features of each modality based on the decoupled shared features and specific features, as shown in the formula:

[0089] ;

[0090] in, Representing modes The corresponding decoder, Modes representing reconstruction The single-modal characteristics, This represents the parameters of the corresponding decoder.

[0091] Using text-specific decoders (i.e.) Based on the projection sharing features of audio and visual modalities (i.e.) and ) and specific features of the text modality (i.e. Reconstruct the original text modality-specific features to improve the semantic consistency of these projected shared features, as shown in the formula:

[0092] ;

[0093] in, and They represent passing through and Modality-specific features of the reconstructed text.

[0094] 6. Classifier

[0095] A classifier is used to predict sentiment based on specific features of each modality and projection-shared features.

[0096] In its specific implementation, the classifier consists of a fully connected layer, two multilayer perceptron (MLP) layers, and a ReLU activation function arranged sequentially.

[0097] Specific features of the three modalities (audio, text, and visual) ) and context-enhanced projection-shared features ( (The two parts are then combined to form a classifier.) The input will give the sentiment prediction result. The formula is:

[0098] .

[0099] This embodiment also provides a training method for the aforementioned multimodal emotion recognition system based on a large language model enhancement, such as... Figure 2 As shown, it includes the following steps:

[0100] S1 extracts dialogue-level features from the dialogue in audio, text, and visual modes respectively through a single-modal dialogue-level feature extraction module.

[0101] S2 uses an LLM large language model to perform shallow feature extraction on audio modality dialogue-level features and visual modality dialogue-level features to obtain shallow embedding representations of the corresponding modalities, and performs deep feature extraction on text modality dialogue-level features to obtain deep embedding representations of the text modality; the shallow embedding representations of the audio modality are superimposed with the corresponding dialogue-level features to obtain alignment-enhanced embedding representations of the audio modality; the shallow embedding representations of the visual modality are superimposed with the corresponding dialogue-level features to obtain alignment-enhanced embedding representations of the visual modality.

[0102] S3 takes the alignment-enhanced embedding representation of the audio modality, the deep embedding representation of the text modality, and the alignment-enhanced embedding representation of the visual modality as input, and encodes the shared features of the corresponding modalities through a shared encoder; and encodes the alignment-enhanced embedding representation of the audio modality, the deep embedding representation of the text modality, and the alignment-enhanced embedding representation of the visual modality separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder to obtain the specific features of the corresponding modalities.

[0103] S4, through the projection module, projects the shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality onto the text space to obtain the projected shared features of the corresponding modality.

[0104] S5 reconstructs the corresponding modal features based on the shared and specific features of each modality through the audio decoder, text decoder, and visual decoder; and reconstructs the specific features of the text modality based on the projection shared features of the audio modality and the projection shared features of the visual modality through the text-specific decoder.

[0105] S6, based on the specific features of each modality and the shared features of projection, performs sentiment prediction through a classifier to obtain the sentiment prediction result.

[0106] S7. Based on the alignment enhancement embedding representation of the audio modality and the dialogue-level features, construct the audio embedding alignment loss. Based on the alignment enhancement embedding representation of the visual modality and the dialogue-level features, construct the visual embedding alignment loss. The audio embedding alignment loss and the visual embedding alignment loss together constitute the total embedding alignment loss.

[0107] Since different audio frames and image patches contribute unequally to the model's text output, stronger supervision should be applied to the initial audio and visual embeddings corresponding to more important audio frames and image patches. This embodiment uses the attention scores generated by the text portion at each layer of the LLM to quantify the importance of each audio frame and each image patch, and uses this as the supervision strength. Specifically, the formula for calculating the audio supervision strength is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] Here, the j-th text token is used as the query vector Q, and the audio features output from the i-th layer of the LLM are used as the key vector K and the value vector V. Indicates the dimension of a vector; This represents the attention score of the j-th text token to the l-th audio frame at the i-th layer of the LLM; k represents the layer number of the LLM, which is related to the construction... k is the same; J represents the number of text tokens; This represents the average attention score obtained by the l-th audio frame across all relevant layers and all text tokens; The normalized average attention score is used as the degree of supervision for the embedding of the l-th initial audio frame.

[0112] The calculation formula for visual supervision intensity is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] Here, the j-th text token is used as the query vector Q′, and the visual features output from the i-th layer of the LLM are used as the key vector K′ and the value vector V′. Indicates the dimension of a vector; This represents the attention score of the j-th text token to the l′-th image patch in the i-th layer of the LLM. This represents the average attention score obtained by the l′ image patch across all relevant layers and all text tokens; The normalized average attention score is used as the degree of supervision for the l′ initial visual embedding.

[0117] Based on the aforementioned supervision strength, this embodiment also proposes directional alignment supervision to guide the initial audio embedding. and visual embedding Orientation alignment. By minimizing their orientation on the unit hypersphere. To reduce the angle between them, the distance (i.e., their Euclidean distance) is used. For audio modalities, directional alignment supervision is defined as the audio embedding alignment loss. :

[0118] ;

[0119] in, Represents the audio modal dialogue-level feature matrix The embedding vector of the l-th audio frame, It is the input embedding generated by the audio projector; L represents the number of audio frames. Alignment-enhanced embedding representation matrix representing audio modalities The embedding vector of the l-th audio frame in the data; It is a normalization term; before calculating the direction, the embedding vector is first normalized. . It is the normalized average attention score for audio.

[0120] Similarly, for the visual modality, directional alignment supervision is defined as the visual embedding alignment loss. :

[0121] ;

[0122] in, Represents the visual modality dialogue-level feature matrix The Image patch embedding vectors, It is the input embedding generated by the visual projector; Alignment-enhanced embedding representation matrix representing visual modalities The first in Image patch embedding vectors; Indicates the number of image blocks. It is a normalization term; It is the normalized average visual attention score.

[0123] By applying supervised embedding and directional alignment supervision, the initial unimodal features of the audio and visual modalities can be aligned with the shallow embeddings, resulting in aligned and enhanced embedding representations, thereby improving overall multimodal understanding. This process further enhances the performance of emotion recognition by accurately aligning the features of each modality. The total loss for aligning the initial unimodal features of the audio and visual modalities with the refined embeddings is [not specified in the original text]. as follows:

[0124] .

[0125] S8 constructs a decoupling loss based on the shared and specific features of each modality, and constructs a reconstruction loss based on the alignment enhancement embedding representation of the audio modality and the visual modality, the deep embedding representation of the text modality, and the corresponding modal features of the reconstruction. The decoupling loss and the reconstruction loss together form an orthogonal decoupling loss.

[0126] This embodiment proposes an orthogonal decoupling strategy. By introducing orthogonal constraints, it ensures the spatial distinguishability of shared features and specific features among different modes. The orthogonal decoupling loss based on this strategy is also presented. Including decoupling loss and reconstruction loss Through decoupling loss Promote discriminability between shared features and specific features; and through single-modal reconstruction loss Preserve the emotion-related information in both types of features. Therefore, It can be represented as:

[0127] ;

[0128] in, and This is a hyperparameter.

[0129] This embodiment applies orthogonal constraints to minimize the similarity between shared features and specific features, ensuring they capture different semantic aspects by being forced to adopt a 90-degree angle. This constraint applies both within the same modality and between different modalities. Therefore, the decoupling loss... It can be calculated as follows:

[0130] ;

[0131] Reconstruction loss Represented as:

[0132] .

[0133] S9 constructs a cross-modal reconstruction loss based on the reconstructed text modality-specific features and the deep embedding representation of the text modality.

[0134] This embodiment also proposes a cross-modal feature projection and alignment strategy to improve cross-modal semantic consistency. Specifically, shared features are first projected into a common text space, and then aligned. To further enhance cross-modal consistency, the projected shared features are used together with text-specific features to reconstruct the original text's unimodal features.

[0135] In the specific implementation, cross-modal reconstruction loss is based on a cross-modal reconstruction strategy. Represented as:

[0136] .

[0137] S10: Construct a classification loss based on the sentiment prediction results and the corresponding sentiment labels of the dialogue.

[0138] In this embodiment, the classification loss is expressed as: It uses cross-entropy loss, where, Indicates the corresponding emotional label in the dialogue. This indicates the sentiment prediction result.

[0139] S11, the alignment total loss, orthogonal decoupling loss, cross-modal reconstruction loss and classification loss form a joint objective function; by optimizing the system parameters through the joint objective function, a well-trained multimodal emotion recognition system is obtained.

[0140] The joint objective function is expressed as:

[0141] .

[0142] This embodiment also provides a multimodal emotion recognition method based on a large language model enhancement, such as... Figure 3 As shown, the following steps are performed using a trained multimodal emotion recognition system:

[0143] L1 extracts dialogue-level features from the dialogue in audio, text, and visual modes respectively through a single-modal dialogue-level feature extraction module.

[0144] L2 uses an LLM large language model to perform shallow feature extraction on audio and visual modal dialogue-level features to obtain shallow embedding representations of the corresponding modalities, and performs deep feature extraction on text modal dialogue-level features to obtain deep embedding representations of the text modalities. The shallow embedding representations of the audio modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the audio modalities. The shallow embedding representations of the visual modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the visual modalities.

[0145] L3 takes the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as inputs. It encodes the shared features of the corresponding modalities through a shared encoder. It then encodes the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder, respectively, to obtain the specific features of the corresponding modalities.

[0146] L4 uses a projection module to project the shared features of the audio modality, the visual modality, and the text modality onto the text space to obtain the projected shared features of the corresponding modality.

[0147] L5, based on the specific features of each modality and the shared features of projection, uses a classifier to predict sentiment and obtain the sentiment prediction results.

[0148] To verify the superiority of this invention, its performance in multimodal emotion recognition tasks was verified through performance comparison with several existing technologies. The experimental setup was as follows: batch size of 32, training on IEMOCAP for 100 loops, and training on MELD for 30 loops. The optimizer used was Adam, and the optimizer parameters were as follows. =0.9, The learning rate is initialized to 0.0001.

[0149] The audio encoder, visual encoder, text encoder, and LLM big oracle model of the above multimodal emotion recognition system are obtained through pre-training, and the hyperparameters in the model are locked.

[0150] For the IEMOCAP dataset, the multimodal emotion recognition system is trained and tested using the 5-fold cross-validation training method according to steps S1-S12 given above. The samples in the dataset are divided into five batches. In each loop, four batches are used for training and one batch is used for testing. Finally, the prediction results of the five parts are averaged. This process is repeated until the number of loop iterations is reached to obtain the trained multimodal emotion recognition system.

[0151] For the MELD dataset, following the defined training, validation, and test sets, the system is first trained using the training set according to steps S1-S12. After each training iteration, the system is validated using the validation set, until the required number of iterations is reached, resulting in a trained multimodal emotion recognition system. The trained system is then tested on the test set.

[0152] To objectively evaluate the technical effectiveness of this invention, it was compared with several state-of-the-art methods (such as DialogGCN, MMGCN, MMDFN, MGLRA, and Joyful), and the results are shown in Table 1. Experimental results show that LDPA-SA of this invention achieves significantly better performance than existing technologies on the IEMOCAP and MELD datasets, especially in terms of accuracy and weighted F1 score in emotion recognition tasks, demonstrating stronger performance.

[0153] Table 1 Comparison of emotion recognition performance of different methods on the IEMOCAP and MELD datasets.

[0154]

[0155] The DiaRNN model mentioned above can be found in Majumder, N. et al.'s Dialogue RNN: An attentive RNN for emotion detection inconversations. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 6818-6825 (2019)). The MMGCN model mentioned above can be found in Wei, Y. et al.'s MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1437-1445 (2019)).

[0156] The MMDFN model mentioned above can be found in Hu, D. et al.'s MM-DFN: Multimodal dynamic fusion network for emotion recognition in conversations. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing.pp. 7037-7041. IEEE (2022)).

[0157] The aforementioned SCMM model can be found in Yang, H. et al.'s paper, "Self-adaptive context and modal-interaction modeling for multimodal emotion recognition." (Yang, H., Gao, X., Wu, J., Gan, T., Ding, N., Jiang, F., Nie, L.: Self-adaptive context and modal-interaction modeling for multimodal emotion recognition. In: Findings of the Association for Computational Linguistics.pp. 6267-6281 (2023)).

[0158] The Joyful model mentioned above can be found in Joyful: Joint modality fusion and graph contrastive learning for multimodal emotion recognition, published by Li, D. et al. (Li, D., Wang, Y., Funakoshi, K., Okumura, M.: Joyful: Joint modality fusion and graph contrastive learning for multimodal emotion recognition. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 16051-16069 (Dec 2023). https: / / doi.org / 10.18653 / v1 / 2023.emnlp-main.996).

[0159] The SACCMA model mentioned above can be found in Guo, L. et al., "Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation" (Guo, L., Song, Y., Ding, S.: Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation. Knowledge-Based Systems 296, 111969 (2024)).

[0160] The MGLRA model mentioned above can be found in Meng, T., et al., "Recursive Learning of Masked Graphs for Multimodal Emotion Recognition in Conversation" (Meng, T., Zhang, F., Shou, Y., Shao, H., Ai, W., Li, K.: Masked graph learning with recurrent alignment for multimodal emotion recognition in conversation. IEEE / ACM Transactions on Audio, Speech, and Language Processing (2024)).

[0161] To demonstrate the discriminability of the feature representations, the clustering performance of the six sentiment categories in the IEMOCAP dataset before and after applying the LDPA-SA method is compared by visualizing the t-SNE projection, as shown below. Figure 4 As shown, the clustering effect of sentiment categories becomes more pronounced after applying the LDPA-SA method compared to when it is not applied. From Figure 4 As can be seen, especially in the two prominent areas of the figure, the clusters of happiness and excitement are more prominent, while the clusters of anger and frustration are also more significant. This indicates that the present invention can significantly improve the performance of emotion classification.

[0162] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multimodal emotion recognition system based on large language model enhancement, characterized in that, include: A single-modal dialogue-level feature extraction module is used to extract dialogue-level features from the audio modality, text modality, and visual modality of the dialogue, respectively. The LLM large language model is used to extract shallow features from audio modal dialogue-level features and visual modal dialogue-level features to obtain shallow embedding representations of the corresponding modalities, and to extract deep features from text modal dialogue-level features to obtain deep embedding representations of the text modalities. The encoding module includes a shared encoder, an audio-specific encoder, a text-specific encoder, and a vision-specific encoder. The shared encoder takes the alignment-enhanced embedding representation of the audio modality, the deep embedding representation of the text modality, and the alignment-enhanced embedding representation of the visual modality as input, and encodes the shared features of the corresponding modalities respectively. The audio-specific encoder, the text-specific encoder, and the vision-specific encoder encode the alignment-enhanced embedding representation of the audio modality, the deep embedding representation of the text modality, and the alignment-enhanced embedding representation of the visual modality separately, and obtain the specific features of the corresponding modalities. The alignment-enhanced embedding representation of the audio modality is obtained by superimposing the shallow embedding representation of the audio modality with the corresponding dialogue-level features; The alignment-enhanced embedding representation of the visual modality is obtained by superimposing the shallow embedding representation of the visual modality with the corresponding dialogue-level features; The projection module is used to project the shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality onto the text space to obtain the projected shared features of the corresponding modality. The decoding module includes an audio decoder, a text decoder, a visual decoder, and a text-specific decoder. The audio decoder, text decoder, and visual decoder reconstruct the corresponding modal features based on the shared features and specific features of each modality. The text-specific decoder reconstructs text-specific features based on the projection-shared features of the audio modality and the projection-shared features of the visual modality; A classifier is used to predict sentiment based on specific features of each modality and projection-shared features.

2. The multimodal emotion recognition system based on large language model enhancement according to claim 1, characterized in that, The single-modal dialogue-level feature extraction module includes an audio feature extraction unit, a text feature extraction unit, and a visual feature extraction unit; the audio feature extraction unit includes an audio encoder and an audio projector; the text feature extraction unit includes a text encoder and a fully connected layer; and the visual feature extraction unit includes a visual encoder and a visual projector.

3. The multimodal emotion recognition system based on large language model enhancement according to claim 2, characterized in that, The audio encoder uses the Whisper-Large model to extract audio features from the dialogue, and then uses an audio projector to obtain audio modal dialogue-level features. The text encoder uses the Baichuan-13B model to extract text features from the dialogue, and then uses a fully connected layer for further processing to obtain text modal dialogue-level features. The visual encoder uses the CLIP-ViT-Large model to extract visual features from the dialogue, and then uses a visual projector to obtain visual modal dialogue-level features. The audio projector and visual projector have the same structure, including two layers of multilayer perceptrons and a GELU activation function arranged sequentially.

4. The multimodal emotion recognition system based on large language model enhancement according to claim 1, characterized in that, For audio and visual modalities, shallow embedding features are extracted using the shallow layers of the large language model, and the features extracted from each shallow layer are weighted and fused to serve as the shallow embedding representation of the corresponding modality. For text modalities, deep embedding features are extracted using the large language model, and the final output serves as the deep embedding representation of the text modality.

5. The multimodal emotion recognition system based on large language model enhancement according to claim 1, characterized in that, The shared encoder, audio-specific encoder, text-specific encoder, and vision-specific encoder have the same structure, each consisting of two fully connected layers; the audio decoder, text decoder, vision decoder, and text-specific decoder have the same structure, each consisting of two fully connected layers; the classifier consists of a fully connected layer, two multilayer perceptrons, and a ReLU activation function arranged sequentially.

6. A training method for a multimodal emotion recognition system based on large language model enhancement, characterized in that, Includes the following steps: The single-modal dialogue-level feature extraction module extracts dialogue-level features from the dialogue for audio modality, text modality, and visual modality respectively. The LLM large language model is used to extract shallow features from audio and visual dialogue-level features to obtain shallow embedding representations for the corresponding modalities, and to extract deep features from text dialogue-level features to obtain deep embedding representations for the text modalities. The shallow embedding representations of the audio modal are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the audio modalities; the shallow embedding representations of the visual modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the visual modalities. Using the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, the shared features of the corresponding modalities are encoded separately through a shared encoder; the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality are encoded separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder, respectively, to obtain the specific features of the corresponding modalities. The shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality are projected onto the text space through the projection module to obtain the projected shared features of the corresponding modality. The corresponding modal features are reconstructed by the audio decoder, text decoder, and visual decoder based on the shared and specific features of each modality. Text modality-specific features are reconstructed using a text-specific decoder based on the projection-shared features of the audio modality and the projection-shared features of the visual modality. Based on the specific features of each modality and the projection-shared features, a classifier is used to predict sentiment, and the sentiment prediction results are obtained. The audio embedding alignment loss is constructed based on the alignment enhancement embedding representation of the audio modality and the dialogue-level features, and the visual embedding alignment loss is constructed based on the alignment enhancement embedding representation of the visual modality and the dialogue-level features. The audio embedding alignment loss and the visual embedding alignment loss together constitute the total embedding alignment loss. Decoupling loss is constructed based on the shared and specific features of each modality, and reconstruction loss is constructed based on the alignment-enhanced embedding representation of the audio modality and the visual modality, the deep embedding representation of the text modality, and the corresponding modal features of the reconstruction. The decoupling loss and the reconstruction loss together form an orthogonal decoupling loss. A cross-modal reconstruction loss is constructed based on the modal-specific features of the reconstructed text and the deep embedding representation of the text modality; A classification loss is constructed based on the sentiment prediction results and the corresponding sentiment labels of the dialogue. The alignment total loss, orthogonal decoupling loss, cross-modal reconstruction loss, and classification loss constitute the joint objective function; By combining objective functions, the system parameters are optimized to obtain a well-trained multimodal emotion recognition system.

7. The training method for a multimodal emotion recognition system based on large language model enhancement according to claim 6, characterized in that, Audio embedding alignment loss Represented as: ； in, Represents the audio modal dialogue-level feature matrix The embedding vector of the l-th audio frame; Alignment-enhanced embedding representation matrix of audio modalities The embedding vector of the l-th audio frame in the data; Represents the normalization term; represents the normalized average audio attention score; L represents the number of audio frames. ； ； ； Here, the j-th text token is used as the query vector Q, and the audio feature output from the i-th layer of the LLM is used as the key vector K. Indicates the dimension of a vector; This represents the attention score of the j-th text token to the l-th audio frame in the i-th layer of the LLM; k represents the number of layers in the LLM; and J represents the number of text tokens. This represents the average attention score obtained by the l-th audio frame across all relevant layers and all text tokens; Visual embedding alignment loss Represented as: ； in, Represents the visual modality dialogue-level feature matrix The Image patch embedding vectors; Alignment-enhanced embedding matrix representing visual modalities The first in Image patch embedding vectors; Represents the normalization term; This represents the normalized average visual attention score. Indicates the number of image patches; ； ； ； Here, the j-th text token is used as the query vector Q′ (Query), and the visual features output from the i-th layer of the LLM are used as the key vector K′ (Key). Indicates the dimension of a vector; This represents the attention score of the j-th text token to the l′-th image patch in the i-th layer of the LLM; This represents the average attention score obtained by the l′ image patch across all relevant layers and all text tokens; The total embedding alignment loss is expressed as: 。 8. The training method for a multimodal emotion recognition system based on large language model enhancement according to claim 6, characterized in that, Decoupling loss Represented as: ； in, This represents the shared features of mode m. This represents a specific characteristic of mode m; The reconstruction loss is expressed as: ； in, Modes representing reconstruction Single-modal characteristics; Representing modes Single-modal characteristics; The orthogonal decoupling loss is expressed as: ； in, and For hyperparameters; Cross-modal reconstruction loss Represented as: ； in, and They represent passing through and Modality-specific features of the reconstructed text; , These represent shared features projected from the audio and visual modalities, respectively. ； ； in, Indicates a text-specific decoder; This represents the learnable projection matrix.

9. The training method for a multimodal emotion recognition system based on large language model enhancement according to claim 6, characterized in that, Classification loss is expressed as It uses cross-entropy loss, where, Indicates the corresponding emotional label in the dialogue. This indicates the sentiment prediction result.

10. A multimodal emotion recognition method based on large language model enhancement, characterized in that, The multimodal emotion recognition system according to any one of claims 1 to 5 is used in accordance with the following steps: The single-modal dialogue-level feature extraction module extracts dialogue-level features from the dialogue for audio modality, text modality, and visual modality respectively. The LLM large language model is used to extract shallow features from audio and visual dialogue-level features to obtain shallow embedding representations for the corresponding modalities, and to extract deep features from text dialogue-level features to obtain deep embedding representations for the text modalities. The shallow embedding representations of the audio modal are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the audio modalities; the shallow embedding representations of the visual modalities are superimposed with the corresponding dialogue-level features to obtain the alignment-enhanced embedding representations of the visual modalities. Using the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality as input, the shared features of the corresponding modalities are encoded separately through a shared encoder; the alignment-enhanced embedding representations of the audio modality, the deep embedding representations of the text modality, and the alignment-enhanced embedding representations of the visual modality are encoded separately through an audio-specific encoder, a text-specific encoder, and a visual-specific encoder, respectively, to obtain the specific features of the corresponding modalities. The shared features of the audio modality, the shared features of the visual modality, and the shared features of the text modality are projected onto the text space through the projection module to obtain the projected shared features of the corresponding modality. Based on the specific features of each modality and the shared features of projection, a classifier is used to predict sentiment, and the sentiment prediction results are obtained.

Images