Multi-mode sentiment classification method based on multi-view interaction representation

Abstract

The invention discloses a multi-modal sentiment classification method based on multi-view interaction representation, and relates to the technical field of multi-modal information processing, and the method comprises the steps: collecting and preprocessing the voice, facial expression and text data of a user, and generating a multi-modal data packet; based on the multi-modal data packet, performing feature extraction by adopting a hierarchical attention mechanism to obtain an aligned voice expression text feature sequence, and generating a multi-modal feature flow; performing sentiment classification on the sentiment benchmark result by adopting a time sequence gating network to obtain a service state vector and a conflict view vector; according to the multi-view fusion emotion feature sequence, a time sequence gating network and a multi-mode recognition model are adopted for recognition, and a matched emotion classification result and a natural language reply are generated in combination with the service state. According to the method, sentiment classification is carried out on the sentiment benchmark result by adopting the time sequence gating network, the service state vector and the conflict view vector are obtained, and synchronous description of the sentiment state and the context is realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a multimodal emotion classification method based on multi-perspective interactive representation. Background Technology

[0002] Multimodal sentiment classification technology is gradually becoming the direction of sentiment recognition. Existing multimodal sentiment classification methods mainly rely on extracting features from different data modalities and performing sentiment analysis and classification through various algorithms. Typically, the sentiment information of each modality is analyzed separately, and multimodal fusion is achieved through weighted fusion and feature concatenation. Existing technologies are widely used in the fields of sentiment recognition, customer service systems, and intelligent assistants, achieving significant classification results.

[0003] However, existing methods still have certain shortcomings in multimodal data fusion. On the one hand, existing feature extraction often relies on the emotional information of a single modality, ignoring the deep interaction between modalities, which limits the improvement of the accuracy of emotion classification. On the other hand, many methods cannot achieve strict synchronization and effective alignment in time sequence. Especially when emotional fluctuations are significant, the time alignment and data synchronization between modalities are difficult to process, affecting the accuracy and stability of emotion classification. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multimodal emotion classification method based on multi-view interactive representation to solve the problems of insufficient temporal accuracy and inadequate intermodal information fusion.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a multimodal emotion classification method based on multi-view interactive representations. The method includes: collecting and preprocessing user speech, facial expression, and text data to generate a multimodal data package; extracting features from the multimodal data package using a hierarchical attention mechanism to obtain aligned speech, facial expression, and text feature sequences, generating a multimodal feature stream; performing cross-modal fusion on the multimodal feature stream to obtain speech, facial expression, text, and preliminary trimodal emotion vectors, generating an emotion baseline result; classifying the emotion baseline result using a temporal gating network to obtain a business state vector and a conflict perspective vector; generating a multi-view fused emotion feature sequence based on the business state vector and conflict perspective vector, using a multi-view interaction graph combined with emotion state and emotion classification; and recognizing the multi-view fused emotion feature sequence using a temporal gating network and a multimodal recognition model, and generating a matching emotion classification result and a natural language response based on the business state.

[0007] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the specific steps for collecting and preprocessing user voice, facial expression, and text data to generate a multimodal data package are as follows. By collecting users' voice, facial expressions, and text data, synchronous startup and tag binding are performed to obtain a set of text fragments; The text fragment set is subjected to noise reduction, frame segmentation, and segmentation normalization to obtain a standardized sequence of speech, facial expression, and text data with uniform time stamps; The standardized sequences are preprocessed and integrated to obtain integrated multi-source content identified by session and time, and multimodal data packets are generated.

[0008] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the following steps are taken: Based on multimodal data packets, a hierarchical attention mechanism is used for feature extraction to obtain aligned speech-expression-text feature sequences, generating a multimodal feature stream. The audio segments, video frames, and text segments of the multimodal data packets are synchronized and processed to obtain the original multimodal sequence; Based on the original multimodal sequence, feature extraction is used to perform frame segmentation and frequency domain analysis on the speech signal, face and facial region cropping on the video frame, and word segmentation and encoding on the text content to obtain a multi-channel sequence of initial speech features, initial facial expression features, and initial text features. Based on multi-channel sequences, a hierarchical attention mechanism is used to aggregate and filter initial speech features, initial facial expression features, and initial text features to generate aligned speech feature sequences, facial expression feature sequences, and text feature sequences, thereby obtaining a multimodal feature stream.

[0009] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the preliminary three-modal emotion vector refers to the emotion representation vector that comprehensively reflects the emotional information of the three modalities after cross-modal fusion of multimodal feature streams.

[0010] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the specific steps for performing cross-modal fusion of multimodal feature streams to obtain speech-expression text and preliminary trimodal emotion vectors, and generating emotion benchmark results, are as follows. The speech features, facial expression features, and text features in the multimodal feature stream are aligned and processed to obtain the aligned speech feature segments, facial expression feature segments, and text feature segment sequences. The emotion coding method is used to analyze and compress sequences of speech feature segments, facial expression feature segments, and text feature segments to obtain a set of single-modal emotion vectors; Cross-modal fusion is performed on a set of single-modal emotion vectors, and emotion information is jointly extracted through a hierarchical attention mechanism to generate a preliminary trimodal emotion vector that simultaneously represents the overall emotional state of speech, facial expression, and text, as well as the corresponding emotion benchmark results.

[0011] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the temporal gating network refers to the process of using a temporal gating network to classify and identify emotion benchmark results and multi-view fused emotion feature sequences, generating business state vectors, conflict perspective vectors, and emotion classification results.

[0012] As a preferred embodiment of the multimodal sentiment classification method based on multi-view interactive representation described in this invention, the following steps are taken: Sentiment classification is performed on the sentiment baseline results using a temporal gating network to obtain the business state vector and conflict perspective vector. The three-modal emotion vectors at different times of the emotion benchmark results are filtered and weighted by time dimension to obtain the emotion representation sequence; Based on the emotion representation sequence, feature extraction is used to perform frame-by-frame and frequency domain analysis on the emotion features of each time slice to generate a business state vector; Based on the emotion representation sequence and business state vector, a temporal gating network is used for emotion classification to obtain a conflict perspective vector that reflects modal consistency.

[0013] As a preferred embodiment of the multimodal emotion classification method based on multi-view interaction representation described in this invention, the step of generating a multi-view fused emotion feature sequence based on business state vectors and conflict perspective vectors, using multi-view interaction graphs combined with emotion state emotion classification, is as follows: By analyzing the business state vector and conflict perspective vector, a set of emotional perspective nodes in the business context is obtained, and the relationship between emotional perspective nodes and conflict perspectives is determined, forming a multi-perspective interaction graph structure. Information is exchanged in a multi-view interactive graph structure, and node representations are aggregated by combining emotional state and sentiment classification to generate a multi-view fused emotional feature sequence.

[0014] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the multimodal recognition model is specifically constructed as follows: A multimodal recognition model is constructed based on a data processing layer, a time-series processing layer, and a discriminant output layer; The data processing layer synchronously collects the user's voice, facial expressions, and text data, performs preprocessing, and constructs aligned multimodal data packets; The temporal processing layer extracts and fuses features from various modalities through a hierarchical attention mechanism to form a unified multimodal feature stream; The discriminant output layer performs emotion classification and semantic matching based on a temporal gating network and a multi-view interaction graph, resulting in a multimodal emotion recognition model.

[0015] As a preferred embodiment of the multimodal emotion classification method based on multi-view interactive representation described in this invention, the steps of identifying emotion features based on multi-view fusion using a temporal gating network and a multimodal recognition model, and generating matching emotion classification results and natural language responses in conjunction with business status, are as follows: A temporal gating network is used to extract features from a multi-view fused emotion feature sequence to obtain an emotion state vector. Emotional discrimination is performed on the emotional state vector to obtain the emotion determination result; A multimodal recognition model is used to identify and filter the emotion judgment results, obtain the target emotion category and corresponding response strategy information, and generate matching emotion classification results and natural language responses in combination with business status.

[0016] The beneficial effects of this invention are as follows: By using a temporal gating network to classify the emotion benchmark results, business state vectors and conflict perspective vectors are obtained, achieving synchronous characterization of emotional state and context. Based on the business state vectors and conflict perspective vectors, a multi-view interaction graph combined with emotion state classification is used to generate a multi-view fused emotion feature sequence. This sequence integrates and unifies the scheduling of voice, facial expression, text information, and context weights, driving the use of a temporal gating network and a multimodal recognition model for recognition and natural language response linkage, achieving clear discrimination and improving interaction accuracy and intelligence. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a multimodal sentiment classification method based on multi-view interactive representation.

[0019] Figure 2 A flowchart for generating multimodal data packets.

[0020] Figure 3 A flowchart generated from the sentiment benchmark results.

[0021] Figure 4 A flowchart for generating natural language responses. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a multimodal emotion classification method based on multi-view interactive representation, including the following steps: S1: Collect user's voice, facial expressions, and text data and preprocess them to generate multimodal data packets.

[0026] S1.1: By collecting the user's voice, facial expressions and text data, synchronously start and bind the identifiers to obtain a set of text fragments.

[0027] Furthermore, by collecting users' voice, facial expressions, and text data, synchronous activation and identification binding are performed. A session identifier (marking a user interaction process) and a time stamp are uniformly added to the voice data, facial expression data, and text data. Based on the unified time stamp (using the same time stamp on the voice data, facial expression data, and text data), the voice data is transcribed into text content through speech recognition and segmented into continuous text segments. At the same time, the text data is segmented into corresponding text segments according to the time stamp, and the start and end of the text segments are aligned and organized according to the time stamp of the facial expression data. The segments are then bound by the session identifier and aggregated into a set of text segments.

[0028] S1.2: Perform noise reduction, framing, segmentation, and normalization on the text fragment set to obtain a standardized sequence of speech, facial expression, and text data with uniform time stamps.

[0029] Furthermore, each text segment in the text fragment set is time-aligned with the corresponding timestamp and speech and facial expression data. Noise reduction processing is performed on the speech data, and it is framed according to a fixed time window. The facial expression part in the video data is segmented at the frame level and its size and lighting conditions are unified. Punctuation repair, stop word filtering, and length segmentation and normalization processing are performed on the text data. The processed speech frames, facial expression frames, and text fragments are reordered and aligned according to the unified timestamp. The speech, facial expression, and text data corresponding to each time point are strictly synchronized in time sequence to form a standardized sequence of speech, facial expression, and text data with unified timestamp.

[0030] Specifically, a timestamp refers to a unified time stamp assigned to each moment of speech, facial expression, and text data during the multimodal data acquisition process, marking the point in time when the data occurred on a unified timeline.

[0031] A fixed time window refers to a uniform time range (from 10 milliseconds to 40 milliseconds) used when continuously segmenting speech data, facial expression data, and text data on the time axis according to a consistent time length.

[0032] It should be noted that segmentation and normalization processing refers to the process of dividing continuous speech, facial expression, and text data into time segments on a unified time axis, and uniformly mapping the speech, facial expression, and text data so that each speech segment, facial expression segment, and text segment has a unified time stamp.

[0033] S1.3: Preprocess and integrate the standardized sequences to obtain integrated multi-source content identified by session and time, and generate multimodal data packets.

[0034] Furthermore, based on a unified time stamp, the speech, facial expression, and text data are aligned by timestamp and segmented according to the session identifier. All speech segments, facial expression frames, and text segments belonging to the same session are grouped into the same session unit. Within each session unit, the speech data, facial expression data, and text data are arranged sequentially according to time to form a time-continuous and modally corresponding data structure. The speech, facial expression, and text data in each session unit are encapsulated to obtain integrated multi-source content with complete time information and session context, generating a multimodal data packet.

[0035] Specifically, preprocessing integration refers to organizing and aligning speech, facial expressions, and text data in a standardized sequence according to session identifiers and chronological order.

[0036] S2: Based on multimodal data packets, a hierarchical attention mechanism is used for feature extraction to obtain aligned speech and facial expression text feature sequences and generate a multimodal feature stream.

[0037] S2.1: Synchronize and organize the audio segments, video frames, and text segments of the multimodal data packets to obtain the original multimodal sequence.

[0038] Furthermore, when synchronizing and organizing the audio segments, video frames, and text segments of the multimodal data packets, the audio segments, video frames, and text segments are matched and aligned one by one according to the unified timestamp carried by each modality in the multimodal data packets, ensuring that audio segments, video frames, and text segments at the same time point belong to the same session unit; the aligned audio segments, video frames, and text segments are arranged sequentially according to time order to form a temporally continuous and modally synchronized sequence structure, thus obtaining the original multimodal sequence.

[0039] S2.2: Based on the original multimodal sequence, feature extraction is used to perform frame segmentation and frequency domain analysis on the speech signal, face and facial region cropping on the video frame, and word segmentation and encoding on the text content to obtain a multi-channel sequence of initial speech features, initial facial expression features, and initial text features.

[0040] Furthermore, based on the original multimodal sequence, the speech signal is processed by frame segmentation, dividing continuous speech into fixed-length speech frames. Frequency domain analysis is performed on each speech frame to extract spectral features and form initial speech features. Face detection is performed on the video frames, and after locating the face region, the facial region is further cropped, while the expression region is preserved. Initial expression features are extracted based on the facial region. The text content is processed by word segmentation, splitting continuous text into word units. Each word unit is encoded and converted into a vector representation to form initial text features. The initial speech features, initial expression features, and initial text features are arranged in the corresponding time order to form a multi-channel sequence of initial speech features, initial expression features, and initial text features.

[0041] It should be noted that feature extraction is specifically processed using frame segmentation and frequency domain analysis, face and facial region cropping, word segmentation and encoding.

[0042] Framing and frequency domain analysis refers to the process of using FFT to continuously divide the speech signal along the time axis into a sequence of speech frames according to a fixed time length, and then extracting the frequency domain features of the amplitude and phase distribution of each speech frame.

[0043] FFT refers to converting a speech signal from the time domain to the frequency domain, calculating the speech frames of the speech signal, and analyzing and extracting frequency components and amplitude distribution.

[0044] Face and facial region cropping refers to the process of using MTCNN to detect and locate faces in video frames, determining the face and facial regions, cropping images within the face and facial regions, and spatially rearranging them to form a set of local facial images.

[0045] MTCNN is used for face detection and facial localization. It achieves fast face detection through a cascaded convolutional neural network (CNN) structure and simultaneously identifies the positions of the face (eyes, nose, and mouth).

[0046] Word segmentation and encoding refers to the process of segmenting and encoding text content according to semantics (text content features corresponding to emotional information in the text content), and using Word2Vec to aggregate and filter the initial text features to provide input.

[0047] Word2Vec refers to converting text content into vector representations and learning the semantic information of each word by analyzing the contextual relationships of words in large-scale texts.

[0048] S2.3: Based on multi-channel sequences, a hierarchical attention mechanism is used to aggregate and filter initial speech features, initial facial expression features, and initial text features to generate aligned speech feature sequences, facial expression feature sequences, and text feature sequences, thereby obtaining a multimodal feature flow.

[0049] Furthermore, the initial speech features, initial facial expression features, and initial text features are synchronized and aligned at the frame level in the temporal dimension. In the first layer of the hierarchical attention mechanism, self-attention is calculated within each modality, and the resulting attention weights are weighted and aggregated to generate discriminative speech feature sequences, facial expression feature sequences, and text feature sequences. In the second layer of the hierarchical attention mechanism, the speech feature sequences and facial expression feature sequences are fused using the text feature sequence as the query to achieve bidirectional interaction among the three modalities. The attention outputs of each layer are fused to generate time-aligned and semantically complementary speech feature sequences, facial expression feature sequences, and text feature sequences, thus obtaining a multimodal feature stream.

[0050] A better approach is to sequentially describe the complete process of frame-level synchronization alignment, self-attention computation, cross-modal fusion, and multi-layer attention output integration on a unified time dimension, making the generation path of multimodal features from initial features to multimodal feature flow clear and coherent.

[0051] Specifically, attention weight refers to the numerical value of attention obtained by calculating the initial features of speech, expression, and text at different time steps within the same modality in a hierarchical attention mechanism.

[0052] The formula for calculating attention is: ; in, Indicates the first The attention weights (real numbers between 0 and 1) corresponding to each time step. Indicates the first Attention score at each time step This represents the index of the time step traversed during the summation operation, used for normalizing the attention scores across all time steps. This represents the exponential operation function, used to map attention scores to positive numbers for normalization calculations.

[0053] It should be noted that the hierarchical attention mechanism refers to the use of existing attention calculation methods in multimodal feature streams to aggregate and filter initial speech features, initial facial expression features, and initial text features layer by layer in a hierarchical manner from local to global. This can generate aligned speech feature sequences, facial expression feature sequences, and text feature sequences from initial speech features, initial facial expression features, and initial text features at different levels.

[0054] Attention score This involves encoding the speech, facial expression, and text features at each time step to obtain query vectors and key vectors; calculating the similarity between the query vectors and key vectors, typically using the inner product to measure the degree of matching between them; and then normalizing the results to obtain an attention score.

[0055] The query vector represents the features of the current moment and the current input, and the information being queried. In multimodal information, the information that is related to the query is found from the multimodal feature stream; (e.g., in sentiment analysis tasks, the query vector represents a time step in the speech features, and the emotional information is found in facial expressions and text features).

[0056] The key vector is a representation in the multimodal feature stream, representing candidate information and the answer, thus obtaining the feature information of the candidate information and the query vector. Information about the query vector is obtained by calculating the similarity between the query vector and the key vector.

[0057] S3: Perform cross-modal fusion on the multimodal feature stream to obtain speech expression text and preliminary trimodal emotion vectors, and generate emotion benchmark results.

[0058] S3.1: Preliminary trimodal emotion vector refers to the emotion representation vector generated by cross-modal fusion of multimodal feature streams, which comprehensively reflects the emotional information of the three modalities.

[0059] S3.2: Align and organize the speech features, facial expression features, and text features in the multimodal feature stream to obtain the aligned speech feature segments, facial expression feature segments, and text feature segment sequences.

[0060] Furthermore, based on a unified time stamp, speech features, facial expression features, and text features are matched frame by frame in the time dimension. Missing and offset time slices are filled in by nearest neighbor padding to ensure that speech feature segments, facial expression feature segments, and text feature segments correspond to the same semantic unit under the same timestamp, forming a time-synchronized aligned sequence of speech feature segments, facial expression feature segments, and text feature segments.

[0061] Specifically, frame-by-frame matching refers to the process of aligning audio segments, facial expression segments, and text segments one by one by comparing timestamps on a unified timeline, using video frames as references, and matching audio segments, facial expression segments, and text segments within the same time range to the same frame position.

[0062] Nearest neighbor completion refers to a processing method that, when aligning and organizing speech segments, facial expression segments, and text segments according to timestamps, fills in missing time positions on the timeline by copying and filling in the speech segment, facial expression segment, and text segment with the nearest time position, thus forming a continuous and complete standardized sequence on a unified timeline.

[0063] S3.3: The emotion coding method is used to analyze and compress the sequences of speech feature segments, facial expression feature segments and text feature segments to obtain a set of single-modal emotion vectors.

[0064] Furthermore, emotion coding methods are applied to speech feature segments to extract emotional cues such as prosody, pitch, and speech rate, and then vector compression is performed to obtain single-modal emotion vectors representing speech emotions. For facial expression feature segments, emotion coding methods based on facial action unit recognition are applied to analyze eye, mouth, and facial muscle movement patterns and extract features to obtain single-modal emotion vectors representing facial expression emotions. For text feature segments, emotion coding methods based on sentiment dictionaries and contextual semantics are applied, combining part-of-speech, sentiment polarity, and contextual dependencies for semantic compression to obtain single-modal emotion vectors representing text emotions. These three types of single-modal emotion vectors are then combined and aligned temporally to form a single-modal emotion vector set.

[0065] It should be noted that emotion coding refers to the process of analyzing and compressing sequences of speech feature segments, facial expression feature segments, and text feature segments, and converting these segments into sets of single-modal emotion vectors that represent speech emotion, facial expression emotion, and text emotion, respectively.

[0066] The unimodal emotion vector set includes voice emotion, facial expression emotion, and text emotion.

[0067] S3.4: Perform cross-modal fusion on the set of single-modal emotion vectors, and jointly extract emotion information through a hierarchical attention mechanism to generate a preliminary trimodal emotion vector that simultaneously represents the overall emotional state of speech, facial expression and text, as well as the corresponding emotion benchmark results.

[0068] Furthermore, the sets of single-modal emotion vectors representing speech emotion, facial expression emotion, and text emotion are input into the cross-modal fusion structure. A hierarchical attention mechanism is used to jointly extract the single-modal emotion vectors based on temporal alignment. This allows the speech emotion vector, facial expression emotion vector, and text emotion vector to be integrated in the shared semantic space through the hierarchical attention mechanism, resulting in the emotion vectors being mutually aligned and consistent in the shared semantic space. The emotional states after concatenation, weighted summation, alignment, and consistency are then integrated to form a joint representation with a unified dimension. A preliminary trimodal emotion vector representing the overall emotional state of speech, facial expression, and text, along with the corresponding emotional baseline results, is generated.

[0069] Specifically, weighted interaction refers to the process of weighting the information interaction between the emotional perspective, business state perspective, and conflict perspective in a multi-perspective interaction graph structure based on the update results, so as to obtain the interaction results that reflect different perspectives.

[0070] Mutual calibration refers to a process in which, in a multi-perspective interactive graph structure, based on the emotional baseline results and the multi-perspective fused emotional feature sequence, the emotional representations corresponding to the emotional perspective, business state perspective, and conflict perspective are adjusted using alignment and weighted interaction to make the emotional classification results corresponding to different perspectives consistent on a unified time axis.

[0071] It should be noted that cross-modal fusion refers to a processing method that, based on the set of single-modal emotion vectors obtained by emotion coding methods, jointly extracts and aggregates the set of single-modal emotion vectors representing speech emotion, facial expression emotion, and text emotion to generate a preliminary trimodal emotion vector that simultaneously represents the overall emotional state of speech, facial expression, and text, and obtains an emotion benchmark result.

[0072] The emotion benchmark result refers to the emotion vector obtained by fusing multimodal data, which reflects the overall emotional state of the three modalities of speech, facial expression and text.

[0073] S4: Use a temporal gating network to classify the sentiment baseline results to obtain the business state vector and conflict perspective vector.

[0074] S4.1: Temporal gating network refers to the process of using a temporal gating network to classify and identify emotions based on emotional baseline results and multi-view fused emotional feature sequences, generating business state vectors, conflict perspective vectors, and emotion classification results.

[0075] S4.2: Perform time-dimension filtering and weighted fusion on the three-modal emotion vectors at different times of the emotion benchmark results to obtain the emotion representation sequence.

[0076] Furthermore, by extracting the trimodal emotion vectors corresponding to each moment from the emotion benchmark results and filtering them by time dimension, a time series arranged in chronological order is obtained. Based on the attention weight allocation method constructed in the hierarchical attention mechanism, the trimodal emotion vectors at each moment are weighted using attention weights. The attention weights are used to characterize the role of emotional information at different moments in the overall emotional evolution, resulting in a reintegrated emotion vector. The reintegrated emotion vectors are then weighted and fused according to the time dimension to obtain an emotion representation sequence.

[0077] Specifically, weighted fusion refers to a processing method in which attention weights are used to weight and sum the emotion vectors corresponding to voice emotion, facial expression emotion, and text emotion during the emotion representation processing, in order to generate an emotion representation that reflects the overall emotional state of voice emotion, facial expression emotion, and text emotion.

[0078] The weighted fusion emotion formula is: ; in, Indicates the first The emotion vectors for each modality are vectors of emotional features from speech, facial expressions, and text.

[0079] Time-dimensional filtering refers to the process of selecting and retaining emotional information at different time points based on the time sequence and changes in business scenarios from the trimodal emotion vector sequence at different times of the emotion benchmark results.

[0080] S4.3: Based on the emotion representation sequence, feature extraction is used to perform frame-by-frame and frequency domain analysis on the emotion features of each time slice to generate a business state vector.

[0081] Furthermore, based on the emotion representation sequence, feature extraction is performed on the emotion features of each time slice, and the continuous emotion representation sequence is extracted into fixed-length time segments; frequency domain analysis is performed on the emotion features of each time segment to extract frequency domain distribution features; combined with temporal context information, a hierarchical attention mechanism is used to weightedly fuse the frequency domain features and time dimension features to generate a business state vector.

[0082] It should be noted that frequency domain features refer to the characteristic representation of emotional information that reflects energy distribution, formant position, or frequency band intensity in the frequency dimension after frequency domain analysis processing of emotional features.

[0083] Temporal dimension features refer to the description of the temporal changes of an emotional expression sequence on a unified time axis, reflecting the characteristics of emotional information as it evolves over time.

[0084] S4.4: Based on the emotion representation sequence and business state vector, a temporal gating network is used for emotion classification to obtain a conflict perspective vector that reflects modal consistency.

[0085] Furthermore, the trimodal emotion vectors and corresponding business state vectors at each time step in the emotion representation sequence are concatenated sequentially on a unified time axis to form the input sequence. A temporal gating network is used to feed the input sequence (speech, facial expression, and text feature sequences) into the gating mechanism. The gating mechanism selectively remembers and forgets temporal information to update the joint representation of emotion and business in real time. The emotion and business information at each time step are weighted and adjusted according to the state of the previous step and the current input to obtain the joint representation of emotion and business at each time step, reflecting the changes in emotion and business state over time. At the output of the temporal gating network, the predicted distributions of speech emotion, facial expression emotion, and text emotion in the temporal dimension are extracted to obtain a vector representation reflecting the consistency of different modalities in emotion judgment, which is the conflict perspective vector.

[0086] It should be noted that the gating mechanism refers to the selective retention and suppression of emotional information in the emotional baseline results, multi-view fused emotional feature sequences, and emotional state vectors in a temporal gating network. This is achieved by assigning different weights to different emotional features to control the transmission and updating of emotional information over time.

[0087] S5: Based on business state vectors and conflict perspective vectors, multi-perspective interaction graphs are used in conjunction with emotional state and sentiment classification to generate a multi-perspective fused emotional feature sequence.

[0088] S5.1: Analyze the business state vector and conflict perspective vector to obtain the set of business context emotion perspective nodes, and determine the relationship between emotion perspective nodes and conflict perspectives to form a multi-perspective interaction graph structure.

[0089] Furthermore, the correlation between business state vectors and emotional state vectors is identified to determine the emotional state fragments in the current business context. These emotional state fragments are then combined with their corresponding business state vectors to obtain emotional perspective nodes labeled with business context features, forming a set of business context emotional perspective nodes. Based on the modal consistency information reflected by the conflict perspective vectors, the conflict perspective vectors are matched with each emotional perspective node in the business context emotional perspective node set. By using the business context emotional perspective node set as graph nodes and the determined correlations between emotional perspective nodes and conflict perspectives as graph edges, a multi-perspective interactive graph structure composed of emotional perspective, business state perspective, and conflict perspective is obtained.

[0090] A better approach is to fully express the generation and correlation relationships of business state vectors, emotional state vectors, and conflict perspective vectors in a multi-perspective interaction graph.

[0091] It should be noted that the association relationship refers to the connection formed by combining the emotional perspective, business state perspective, and conflict perspective when information is exchanged and weighted based on the business state vector and the conflict perspective vector between nodes in the emotional perspective and between conflict perspectives.

[0092] S5.2: Information interaction is carried out in a multi-view interactive graph structure, and node representations are aggregated by combining emotional state and sentiment classification to generate a multi-view fused emotional feature sequence.

[0093] Furthermore, based on the classification and aggregation of emotion state vectors, a set of emotion perspective nodes labeled with business context features is obtained, and a multi-view interaction graph structure is generated according to the association between emotion perspective nodes and conflict perspectives. In the multi-view interaction graph structure, each node (emotion perspective node, business state perspective node, and conflict perspective node) exchanges information with neighboring nodes, aggregates the information of neighboring nodes, and generates representations of emotion perspective nodes, business state perspective nodes, and conflict perspective nodes. This allows each emotion perspective node, business state perspective node, and conflict perspective node to update its features based on the representations of neighboring nodes, realizing information interaction between multiple perspectives. The weights of each node in information aggregation are dynamically adjusted based on the results of emotion state sentiment classification, and the updated node representations are aggregated by weighted summation to generate a multi-view fused emotion feature sequence.

[0094] Specifically, representation aggregation refers to the process of weighted fusion of node representations corresponding to the emotional perspective, business state perspective, and conflict perspective in a multi-perspective interaction graph structure, based on the emotional state classification results and weighted interaction results, to generate emotional feature representations in a multi-perspective fused emotional feature sequence.

[0095] S6: Based on the multi-view fusion of emotion feature sequences, a temporal gating network and a multimodal recognition model are used for identification, and the matching emotion classification results and natural language responses are generated in combination with the business status; the multimodal recognition model is constructed based on a data processing layer, a temporal processing layer and a discriminative output layer.

[0096] S6.1: Data processing layer, which synchronously collects the user's voice, facial expressions and text data, performs preprocessing and constructs aligned multimodal data packets.

[0097] Furthermore, based on the synchronous acquisition of user voice, facial expression, and text data, a set of text segments with unified time stamps is obtained through synchronous startup and binding with time stamps. According to the set of text segments, the voice signal is subjected to noise reduction and frame segmentation processing, the facial expression video is segmented and normalized, and the text content is segmented and standardized to obtain standardized sequences of voice data, facial expression data, and text data respectively. The standardized sequences are integrated using a time alignment strategy to generate integrated multi-source content organized by session and time stamp. Using the integrated multi-source content, aligned multimodal data packets are constructed as the output of the data processing layer and as the input of the timing processing layer.

[0098] S6.2: Temporal processing layer, which extracts and fuses features from various modalities through a hierarchical attention mechanism to form a unified multimodal feature stream.

[0099] Furthermore, the data processing layer synchronously collects users' voice, facial expressions, and text data, performs noise reduction, frame segmentation, and normalization processing to obtain standardized sequences with unified time stamps; it then integrates these standardized sequences to generate multimodal data packets; based on the voice segments, video frames, and text segments in the multimodal data packets, it uses feature extraction to obtain initial voice features, initial facial expression features, and initial text features, respectively; and it aggregates and filters the three types of initial features through a hierarchical attention mechanism to generate aligned voice feature sequences, facial expression feature sequences, and text feature sequences, thus obtaining a multimodal feature stream.

[0100] S6.3: The discriminant output layer performs emotion classification and semantic matching based on a temporal gating network and a multi-view interaction graph, resulting in a multimodal emotion recognition model.

[0101] Furthermore, based on the sentiment benchmark results, a temporal gating network is used for time-dimensional filtering and weighted fusion to generate business state vectors and conflict perspective vectors. Utilizing these vectors and combined with sentiment classification, the business state vectors and sentiment vectors are analyzed to obtain a set of sentiment perspective nodes within the business context. The relationships between these sentiment perspective nodes and conflict perspectives are determined, forming a multi-perspective interaction graph structure. Information interaction takes place within this multi-perspective interaction graph structure, and node representations are aggregated using sentiment classification to generate a multi-perspective fused sentiment feature sequence. Based on this sequence, a temporal gating network is used to extract feature sentiment state vectors for sentiment discrimination and confidence assessment, yielding the sentiment determination result. This completes the construction of a multimodal recognition model for sentiment recognition based on fused business states.

[0102] S6.4: A temporal gating network is used to extract features from the multi-view fused emotional feature sequence to obtain an emotional state vector.

[0103] Furthermore, the multi-view fused emotional feature sequence is sequentially input into a temporal gating network according to time steps. The gating mechanism in the temporal gating network selectively remembers and forgets the multi-view fused emotional features at each time step, retaining information relevant to the emotional state and suppressing irrelevant information. Emotional information across time steps is accumulated in the emotional state of the temporal gating network, and the emotional state is obtained as an emotional state vector.

[0104] S6.5: Perform emotion discrimination on the emotion state vector to obtain the emotion judgment result.

[0105] Furthermore, the emotion state vector is input into the discriminant output layer of the multimodal recognition model. The discriminant output layer matches and judges the emotion category information of the emotion state vector based on the emotion classification obtained by the temporal gating network and the multi-view interaction graph. It also performs discriminant analysis based on the distribution characteristics of the emotion state vector in different emotion categories and outputs the corresponding emotion category attribution result as the emotion determination result.

[0106] A better approach is to input the emotion state vector into the discriminant output layer of the multimodal recognition model, and then use the emotion classification obtained by the discriminant output layer based on the temporal gating network and the multi-view interaction graph to match and discriminate the distribution characteristics of the emotion state vector in different emotion categories, and obtain the corresponding emotion category attribution result as the emotion determination result.

[0107] S6.6: Employ a multimodal recognition model to identify and filter emotion judgment results, obtain target emotion categories and corresponding response strategy information, and generate matching emotion classification results and natural language responses in conjunction with business status.

[0108] Furthermore, the emotion determination result is input into the discriminant output layer of the multimodal recognition model. The discriminant output layer performs semantic matching and identification filtering on the candidate emotion tags in the emotion determination result based on the emotion state vector output by the temporal gating network and the contextual representation provided by the multi-view interaction graph. The target emotion category with the highest confidence is selected, and the response strategy information corresponding to the target emotion category is aligned and organized with the business state to generate an emotion classification result that conforms to the current business context. Based on the emotion classification result and response strategy information, a natural language response is generated.

[0109] Specifically, contextual representation refers to the joint encoding and aggregation of speech, facial expressions, and text data before and after emotional states in integrated multi-source content organized by session and time, forming a feature representation that reflects the current session business scenario and the pattern of emotional evolution.

[0110] In summary, this invention achieves simultaneous characterization of emotional state and context by: using a temporal gating network to classify the emotion benchmark results, obtaining business state vectors and conflict perspective vectors; and leveraging multi-view interaction graphs combined with emotion state classification to generate a multi-view fused emotion feature sequence. This sequence integrates and unifies the scheduling of speech, facial expression, text information, and contextual weights, driving the use of a temporal gating network and a multimodal recognition model for recognition and natural language response, resulting in clearer discrimination and improved interaction accuracy and intelligence.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal sentiment classification method based on multi-view interaction representation, characterized in that: Comprising, Collecting and preprocessing voice, facial expression and text data of a user to generate a multi-modal data package; Based on the multi-modal data package, using a hierarchical attention mechanism to extract features to obtain an aligned voice expression text feature sequence to generate a multi-modal feature flow; Cross-modal fusion is performed on the multi-modal feature flow to obtain a voice expression text and a preliminary three-modal emotion vector to generate an emotion benchmark result; Using a time gating network to perform emotion classification on the emotion benchmark result to obtain a business state vector and a conflict perspective vector; Based on the business state vector and the conflict perspective vector, using a multi-perspective interaction graph combined with emotion state emotion classification to generate a multi-perspective fusion emotion feature sequence; According to the multi-perspective fusion emotion feature sequence, using a time gating network and a multi-modal recognition model for recognition, and combining the business state to generate a matching emotion classification result and a natural language reply. 2.The multi-modal sentiment classification method based on multi-view interaction representation according to claim 1, wherein: The specific steps of collecting and preprocessing voice, facial expression and text data of a user to generate a multi-modal data package are as follows, By collecting voice, facial expression and text data of a user, synchronous starting and identification binding are performed to obtain a text segment set; The text segment set is subjected to noise reduction framing, cutting and normalization processing to obtain a standardized sequence of voice, facial expression and text data with unified time markers; The standardized sequence is preprocessed and integrated to obtain an integrated multi-source content according to the conversation and time identifier to generate a multi-modal data package. 3.The multi-modal sentiment classification method based on multi-view interaction representation according to claim 2, wherein: The specific steps of using a hierarchical attention mechanism to extract features based on the multi-modal data package to obtain an aligned voice expression text feature sequence to generate a multi-modal feature flow are as follows, The voice segment, video frame and text segment of the multi-modal data package are synchronously arranged to obtain a multi-modal original sequence; According to the multi-modal original sequence, feature extraction is used to perform framing and frequency domain analysis on the voice signal, face and facial region cropping on the video frame, and word segmentation and encoding processing on the text content to obtain a multi-channel sequence of voice initial features, expression initial features and text initial features; Based on the multi-channel sequence, a hierarchical attention mechanism is used to aggregate and filter the voice initial features, expression initial features and text initial features to generate aligned voice feature sequences, expression feature sequences and text feature sequences to obtain a multi-modal feature flow.

4. The multi-modal sentiment classification method based on multi-view interaction representation according to claim 3, characterized in that: The preliminary three-modal emotion vector refers to an emotion representation vector that comprehensively reflects three modal emotion information generated after cross-modal fusion of the multi-modal feature flow.

5. The multi-modal sentiment classification method based on multi-view interaction representation according to claim 4, characterized in that: The specific steps of cross-modal fusion of the multi-modal feature flow to obtain a voice expression text and a preliminary three-modal emotion vector to generate an emotion benchmark result are as follows, The voice features, expression features and text features in the multi-modal feature flow are aligned and arranged to obtain an aligned voice feature segment, expression feature segment and text feature segment sequence; Using an emotion coding method, the voice feature segment, expression feature segment and text feature segment sequence are analyzed and compressed to obtain a single-modal emotion vector set; The single-modal emotion vector set is fused cross-modally, emotion information is jointly extracted through a hierarchical attention mechanism, and a preliminary three-modal emotion vector and a corresponding emotion benchmark result are generated, which represent the overall emotion state of voice, expression and text. 6.The multi-modal sentiment classification method based on multi-view interaction representation according to claim 5, wherein: The time sequence gating network is used to perform emotion classification and recognition on the emotion benchmark result and the multi-perspective fusion emotion feature sequence, and generate a business state vector, a conflict perspective vector and an emotion classification result.

7. The multi-modal sentiment classification method based on multi-view interaction representation according to claim 6, characterized in that: The emotion benchmark result is classified by the time sequence gating network to obtain the business state vector and the conflict perspective vector, and the specific steps are as follows, The three-modal emotion vectors of the emotion benchmark result at different times are filtered and weighted in the time dimension to obtain an emotion representation sequence. According to the emotion representation sequence, the emotion features of each time slice are analyzed by frame and frequency domain analysis using feature extraction to generate a business state vector. Based on the emotion representation sequence and the business state vector, the time sequence gating network is used for emotion classification to obtain a conflict perspective vector reflecting modal consistency. 8.The multi-modal sentiment classification method based on multi-view interaction representation according to claim 7, wherein: Based on the business state vector and the conflict perspective vector, a multi-perspective fusion emotion feature sequence is generated by combining the emotion state emotion classification using a multi-perspective interaction graph, and the specific steps are as follows, The business context emotion perspective node set is obtained by analyzing the business state vector and the conflict perspective vector, and the emotion perspective node and the conflict perspective association relationship are determined to form a multi-perspective interaction graph structure. Information interaction is performed in the multi-perspective interaction graph structure, and node representation aggregation is performed in combination with emotion state emotion classification to generate a multi-perspective fusion emotion feature sequence.

9. The multi-modal sentiment classification method based on multi-view interaction representation according to claim 8, characterized in that: The specific construction process of the multi-modal recognition model is as follows, A multi-modal recognition model is constructed based on a data processing layer, a time sequence processing layer and a discriminative output layer. The data processing layer synchronously collects voice, facial expression and text data of a user, pre-processes and constructs aligned multi-modal data packets. The time sequence processing layer extracts and fuses each modal feature through a hierarchical attention mechanism to form a unified multi-modal feature stream. The discriminative output layer performs emotion classification and semantic matching based on the time sequence gating network and the multi-perspective interaction graph to obtain a multi-modal recognition model for emotion recognition.

10. The multi-modal sentiment classification method based on multi-view interaction representation according to claim 9, characterized in that: The specific steps of the multi-modal recognition model are as follows, The multi-modal recognition model is used to recognize the multi-perspective fusion emotion feature sequence using the time sequence gating network and the multi-modal recognition model, and the matching emotion classification result and natural language reply are generated in combination with the business state. The time sequence gating network is used to extract features from the multi-perspective fusion emotion feature sequence to obtain an emotion state vector. The emotion state vector is classified to obtain an emotion determination result. The multi-modal recognition model is used to recognize the emotion determination result to obtain a target emotion category and corresponding response strategy information, and the matching emotion classification result and natural language reply are generated in combination with the business state.

Images