User attitude tendency identification method and system and interaction system

Abstract

The invention relates to the field of attitude identification, and provides a user attitude tendency identification method and system and an interaction system in order to improve the accuracy of attitude tendency identification, the attitude of each mode is obtained through a self-adaptive attitude emotion polarity detection network, whether an attitude ambiguity problem exists between the modes is judged through an attitude polarity ambiguity diagnosis network, and the user attitude tendency identification accuracy is improved. If ambiguity exists, carrying out feature fusion by adopting a selective attention network, otherwise, carrying out feature fusion by adopting a common attention network; selecting an attention network, adopting a multi-head attention mechanism to learn contribution degrees of various semantics of various modes to tasks, and performing weighted connection on the features according to the contribution degrees to complete feature interactive fusion; the common attention network uses different emotional semantics of a text mode to guide non-text mode learning and connects the learned features to complete feature interaction fusion; and by adopting different feature fusion modes, the attitude tendency identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of attitude recognition, and in particular to a method, system and interactive system for identifying user attitude tendencies. Background Art

[0002] With the continuous development of information technology and intelligent technology, users tend to post videos online to comment on certain products and phenomena. Effectively collecting information on these user attitudes helps companies make timely adjustments to existing products and product directions. These user reviews primarily involve data such as images, videos, text, and audio. To effectively identify user tendencies based on this data, research has emerged on technologies such as multimodal data fusion and alignment.

[0003] A key focus of user attitude recognition is how to effectively represent the attitude relationships between different modalities. With the emergence of the Transformer architecture, its long-term temporal learning and dynamic interaction capabilities have been utilized in the research of various multimodal task analysis to effectively capture the relationships between different modalities.

[0004] While some multimodal analysis methods are currently available for learning the attitudinal and emotional relationships between users' different expressions, experiments have observed numerous instances of ambiguous attitudes and emotions when expressing personal opinions, a phenomenon known as sarcasm. For example, when a user says with a smile, "The rice here is really expensive," the visual modality indicates a positive attitude, while the textual modality indicates a negative attitude. This creates a semantic conflict between the visual and textual modalities, affecting the model's final judgment of the attitude. Such examples make it more difficult for neural networks to learn the relationships between multimodal expressions, resulting in poor model network performance. One possible solution to this problem is to collect and annotate a large number of ambiguous examples to increase the proportion of such examples in the dataset and achieve balanced learning examples. However, this approach is extremely labor-intensive and time-consuming.

[0005] Furthermore, existing multimodal learning methods suffer from modal imbalance during modal fusion, whereby different modalities contribute inconsistently to the final task, impacting the model's generalization and stability. Traditional multimodal fusion methods often employ fixed strategies (such as simple concatenation or weight averaging) to process different modalities, failing to optimize based on the data context and dynamic relationships between modalities. Furthermore, some methods tend to learn specific target semantics within the multimodal representation of the fusion layer, neglecting the semantic consistency between unimodal features and the semantic representation capabilities of unimodal feature extraction networks. Other studies exploit multimodal matching tasks to mine all cross-modal semantic relationships and then filter attitude-related semantic relationships through downstream tasks. However, the modalities pre-trained for such matching tasks contain a wide range of information, such as action and scene information. Relying solely on a single task to filter out redundant information not only increases model convergence time but also introduces more information redundancy into the multimodal representation. Summary of the Invention

[0006] In order to improve the accuracy of attitude tendency recognition, the present application provides a user attitude tendency recognition method, system and interactive system.

[0007] The technical solution adopted by the present invention to solve the above problems is: User attitude tendency identification methods include: Step 1: Obtain training samples. The training samples include multimodal data and labels. The modalities include video, audio, and the corresponding text of the audio. The labels include unimodal attitude labels and multimodal attitude labels. Step 2: Extract features from each modal data; Step 3: Build and train an attitude tendency recognition model. The attitude tendency recognition model includes: Modal encoder: Use the encoder corresponding to each modality to encode the extracted features to obtain single-modal features; Adaptive attitude and emotion polarity detection network: obtains the unimodal attitude label corresponding to each modality based on the data of each modality; Attitude Polarity Ambiguity Diagnosis Network: Based on unimodal features and unimodal attitude labels, it determines whether each modality has attitude polarity ambiguity. If so, it uses the selective attention network for feature interaction fusion; if not, it uses the common attention network for feature interaction fusion. Selective Attention Network: A multi-head attention mechanism is used to learn the contribution of various semantics of each modality to the task, and weighted connections are made based on the contribution to complete feature interaction fusion; Common Attention Network: Utilizes the different sentiment semantics of text modalities to guide the learning of non-text modalities and connects the learned features to achieve interactive feature fusion; Attitude tendency recognition network: outputs attitude tendency recognition results based on the fused features; Step 4: Obtain the multimodal data to be identified and identify the user's attitude tendency based on the trained attitude tendency recognition model.

[0008] Furthermore, the feature extraction step in step 2 is: For text modalities: Word Tokenization is used to divide sentences into words or subword units; the BERT model is used for encoding to obtain semantic features; For video modality: frame sampling of the video; face detection and alignment using MTCNN; and facial feature extraction using the DeepFace model; For audio modality: sample the audio; use Librosa or OpenSMILE to extract audio features.

[0009] Furthermore, an adaptive attitude emotion polarity detection network is trained based on each modal data and multimodal attitude labels. The specific steps are as follows: Based on the multimodal attitude labels, positive and negative reference values ​​are established to describe the unimodal sentiment polarity direction; Transform all modalities into the same feature space; The distances between each unimodal feature and the positive and negative reference values ​​are compared to determine the attitude polarity of each unimodal.

[0010] Furthermore, in the process of converting all modalities into the same feature space, a loss function is used to reduce the feature distribution distance before and after the modal enters the feature space.

[0011] Furthermore, the training sample also includes a second training sample, and the label of the second training sample only includes the multimodal attitude label.

[0012] Furthermore, the multimodal data to be recognized is obtained by using a camera to obtain video data, using a microphone to obtain audio data, and converting the audio data into text data.

[0013] User attitude tendency recognition system, including: Multimodal data acquisition module: used to obtain multimodal data, including video, audio and the text corresponding to the audio; Feature extraction module: extract features from each modal data acquired; Attitude tendency recognition module: completes attitude tendency recognition based on the extracted features and attitude tendency recognition model.

[0014] Furthermore, the multimodal data acquisition module uses a camera to acquire video data, uses a microphone to acquire audio data, and converts the audio data into text data.

[0015] Interactive systems, including: Data processing module: used to obtain video data and audio data input by the user, and convert the audio data into text data; perform feature extraction on the video data, audio data and text data; Algorithm embedded module: completes attitude tendency recognition based on the attitude tendency recognition model, and generates corresponding feedback based on text data and attitude tendency recognition results; Human-computer interaction module: used to interact with users; Data storage module: used to store interaction content and attitude tendency recognition results.

[0016] Furthermore, the interaction method with the user includes text and / or voice.

[0017] Compared with the prior art, the present invention has the following beneficial effects: the attitude of each modality is obtained through an adaptive attitude emotion polarity detection network, and whether there is an attitude ambiguity problem between the modalities is judged through an attitude polarity ambiguity diagnosis network. If there is ambiguity, a selective attention network is used for feature fusion, otherwise a common attention network is used for feature fusion; the selective attention network adopts a multi-head attention mechanism to learn the contribution of multiple semantics of each modality to the task, and weightedly connects the features according to the contribution to complete feature interaction fusion; the common attention network uses the different emotional semantics of the text modality to guide the learning of non-text modalities respectively and connects the learned features to complete feature interaction fusion; by adopting different feature fusion methods, the accuracy of attitude tendency recognition is improved, and the efficiency of single-modal attitude labeling is improved and the cost of manual labeling is reduced through the adaptive attitude emotion polarity detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart of the user attitude tendency identification method; Figure 2 A framework diagram for the attitude tendency identification model; Figure 3 Schematic diagram of the network framework for adaptive attitude emotion polarity detection; Figure 4 A comparison chart of the traditional selective attention mechanism and the selective attention mechanism proposed in this application; Figure 5 Schematic diagram of the attention selection network framework; Figure 6 Schematic diagram of the common attention network framework; Figure 7 Schematic diagram of the interactive system framework. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, the user attitude tendency identification method includes: Step 1: Obtain training samples. The training samples include multimodal data and labels. The modalities include video, audio, and the text corresponding to the audio. The labels include unimodal attitude labels and multimodal attitude labels. The multimodal attitude label refers to the label corresponding to the final attitude determined based on the video, audio, and text.

[0021] Step 2: Extract features from each modal data.

[0022] For text modalities, we first use Word Tokenization to divide sentences into words or subwords. We then use the BERT (Bidirectional Encoder Representations from Transformers) model to encode these word vectors to obtain context-sensitive, deep semantic feature representations. The BERT model's multi-layered Transformer architecture enables fine-grained semantic capture of text features at different levels of context, generating feature vectors with strong semantic expression capabilities.

[0023] For video, facial landmarks, eye gaze, and facial action units are commonly used visual features in existing multimodal attitude and emotion analysis research. The system model first samples the video, extracting consecutive image frames at a specific frame rate to reduce computational complexity and redundant information while preserving critical temporal information. By choosing a suitable frame rate, the video's representativeness is ensured, capturing important dynamic changes without causing information loss. Secondly, MTCNN (Multi-task Cascaded Convolutional Networks) is used for face detection and alignment. MTCNN efficiently identifies the location of faces in images and, through a three-stage cascade of convolutional networks, accurately locates key facial features (such as the eyes, nose, and mouth). This face detection and alignment step ensures more accurate subsequent facial feature extraction, eliminating the effects of angle or positional deviations. Finally, the DeepFace model is used for facial feature extraction. DeepFace encodes the aligned facial images using a deep neural network, extracting high-dimensional facial feature vectors that effectively represent the facial identity and attitude and emotion state. The DeepFace model has strong generalization capabilities and can accurately recognize faces under various lighting, posture, and expression conditions.

[0024] For audio modalities, various acoustic features have been shown to be effective for attitude and emotion recognition. Acoustic features can be divided into two categories: low-level descriptors (LLDs) and high-level statistical functions (HSFs). LLDs (including prosodic and spectral features) are computed on a frame-by-frame basis, while HSFs are computed on an entire sentence basis. The proposed model first samples the audio to extract high-quality audio signals. Then, libraries such as Librosa or OpenSMILE can be used for audio feature extraction. Librosa is a widely used audio processing library that provides a variety of audio analysis functions, including time-domain and frequency-domain feature extraction, such as Mel-Frequency Cepstral Coefficients (MFCCs), spectral centroids, and zero-crossing rates. These low-level descriptors capture detailed features in audio signals and are suitable for a variety of audio analysis tasks. OpenSMILE is a powerful open-source tool particularly suitable for attitude, emotion, and behavioral analysis tasks. It extracts a large number of hand-crafted features, covering time, frequency, and energy domains, as well as some advanced statistical features.

[0025] Step 3: Build and train the attitude tendency recognition model. Figure 2 Shown, including: Modal encoder: Use the encoder corresponding to each modality to encode the extracted features to obtain single-modal features; Adaptive attitude and emotion polarity detection network: obtains the unimodal attitude label corresponding to each modality based on the data of each modality; Attitude Polarity Ambiguity Diagnosis Network: Based on unimodal features and unimodal attitude labels, it determines whether each modality has attitude polarity ambiguity. If so, it uses the selective attention network for feature interaction fusion; if not, it uses the common attention network for feature interaction fusion. Selective Attention Network: A multi-head attention mechanism is used to learn the contribution of various semantics of each modality to the task, and weighted connections are made based on the contribution to complete feature interaction fusion; Common Attention Network: Utilizes the different sentiment semantics of text modalities to guide the learning of non-text modalities and connects the learned features to achieve interactive feature fusion; Attitude tendency recognition network: outputs attitude tendency recognition results based on the fused features.

[0026] The present invention constructs three different independent modality feature encoders for the feature learning process of different modalities, processing video, audio and text respectively.

[0027] Text modal input After word segmentation through WordPiece, it is input into the BERT model for semantic feature extraction. The calculation process is ,in ,and is the hidden layer size of BERT, R is the real number field, l represents the length of the input text, and then the self-attention module is used to Perform feature self-attention analysis, the formula is expressed as: ,in Represents multi-layer nonlinear calculations, used for the forward propagation process, Self-attention process of representative features.

[0028] In the forward propagation process of the unimodal learning network, each modality is first extracted through the bidirectional long short-term memory model (BiLSTM) and the pre-trained BERT model. Then, the self-attention mechanism and the forward propagation layer are used to further optimize the features. This includes a fully connected layer and a nonlinear activation layer with a ReLU function. In addition, to reduce the risk of overfitting, we used a dropout method before each linear transformation.

[0029] In order to effectively capture the temporal and spatial dynamic information in each modality and promote the learning of more semantic information, a bidirectional long short-term memory network (BiLSTM) is used to obtain temporal information, and a multi-stage progressive attention structure is designed based on Transformer to achieve feature encoding for non-text modalities.

[0030] Given non-text input features ,in Represents non-text feature input, l represents the length of input data, It is a non-text feature dimension; first, feature encoding is performed through a bidirectional long short-term memory network (BiLSTM) and combined with position encoding (PE), that is, . It is then passed to the encoder, which consists of three main computational modules: the self-attention module, the layer normalization module, and the nonlinear transformation module. Specifically, the non-text encoding process is as follows: , in represents the combination of residual network and layer regularization calculation, represents the forward propagation layer, which includes a fully connected layer and a nonlinear activation layer using the ReLU function. To reduce the risk of overfitting, the dropout method is applied before each forward propagation.

[0031] The key point of the present invention is to distinguish between attitude consistency and ambiguity in samples. Therefore, an adaptive modality attitude polarity detection algorithm is designed to adaptively generate unimodal attitude polarity relationship labels.

[0032] like Figure 3 As shown, the adaptive attitude and emotion polarity detection network is trained to label the attitude and emotion polarity relationship between different modalities in each sample based on the distance between unimodal and multimodal distributions.

[0033] Before obtaining the unimodal emotion polarity, it is necessary to establish a standard positive and negative reference to describe the unimodal emotion polarity direction. The present invention uses multimodal labels as the emotion polarity reference value. The calculation method of the positive and negative reference values ​​is as follows: , , in and are the reference values ​​of positive and negative sentiment polarity, is the total number of samples, represents the multimodal emotion label of the i-th sample, Is an indicator, if the condition in the brackets is met, it is '1', otherwise it is '0'. It is a multimodal feature representation.

[0034] The attitude polarity of each unimodal feature distribution is determined by comparing its distance to a reference feature distribution for positive attitude polarity and a reference feature distribution for negative attitude polarity. However, representations of different modalities exist in different feature spaces. Before calculating the distance between different modalities, all modalities need to be converted to the same feature space. To this end, a shared space is designed to map different modalities into the same space. The specific calculation is as follows: , in is the layer regularization calculation, It is a multimodal feature representation, that is, a multimodal feature representation after the fusion of the selective attention network and the common attention network. Represents unimodal feature representation in shared space, three unimodal feature representations and multimodal feature representation Having the same feature range, it is possible to effectively calculate the spatial distribution distance of single-modal and multi-modal features. represents the modality learning weight, indicates cross-attention, " represents the vector dot product operation, Represents text, audio, and video respectively.

[0035] After converting all unimodal features to the same shared space, the distance calculation formula between unimodal and multimodal feature representations is: , in is the unimodal sentiment polarity, ||.|| represents the L2 regularization function, It is a minimum value to prevent the denominator from being 0.

[0036] In order to preserve the original important semantic information of each modality during the shared space learning process and prevent the feature representations between modalities from being too similar, a loss function was added during the experiment to reduce the loss before the modalities enter the shared space ( ) and after entering the shared space ( )’s characteristic distribution distance, the formula is as follows: .

[0037] After obtaining the spatial distance relationship between all unimodal feature representations and multimodal feature representations, the attitude emotion polarity relationship label can be calculated as follows: .

[0038] in, , , Polarity labels for text, audio, and video respectively.

[0039] In order to learn stable attitude polarity relationship labels, the generated labels should be continuously updated as the model approaches the fitting level. When the model reaches the optimal level, the attitude polarity relationship labels are determined. Therefore, a label update strategy is designed by considering the generation results of multiple time periods of model training. The specific details are as follows: , in, It is the attitude, emotion and polarity relationship label at the latest moment after the nth round of model training. Generate labels for each epoch of model training.

[0040] As can be seen from the above formula, the update amplitude of the attitude polarity relationship label gradually decreases with the increase in the number of model training times. Through this formula, it can be achieved that when the model training gradually stabilizes, the attitude polarity relationship label also gradually stabilizes.

[0041] To facilitate the evaluation of attitude polarity relationships between different modalities in a sample, this paper proposes a multimodal attitude polarity ambiguity diagnosis network (SDT). This network introduces an additional label, called the attitude polarity relationship label, to represent multimodal attitude polarity relationships. When the attitude polarity of all unimodal expressions in a single sample is consistent (all positive or all negative attitude labels), the sample's attitude polarity relationship label is set to '1', indicating consistent attitude polarity. When the attitude polarity of one unimodal expression in a single sample is opposite to that of the other modalities, the sample's attitude polarity relationship label is set to '-1', indicating attitude polarity ambiguity.

[0042] The training task of the multimodal attitude ambiguity diagnosis network (SDT) is a regression task, using the mean absolute error as the loss function. The loss function can be expressed as: , in is the task loss value, Indicates attitude emotion polarity relationship label, the value is '1' or '-1', is the attitude-emotion polarity relationship prediction value, Indicates the number of training samples.

[0043] In the experiment, we use batch matrix-matrix product (BMM) to calculate the similarity between modalities and introduce multiple nonlinear transformations into the network (SDT). The calculation process of SDT is as follows: , , , in, As a nonlinear activation function used in the forward propagation process of the model, it can be expressed as , to normalize the feature value range to [-1, 1], '[,]' represents feature connection, represents the linear change layer, Indicates modality and modal The similarity calculation results between is the emotional polarity relationship label of the final three modes, Represent the batch size and the vector length of the mode respectively.

[0044] In this model, the method of multimodal fusion and interaction is determined by the results of the Attitude Ambiguity Diagnosis Network (SDT) evaluation, to reduce the mutual influence between the two types of samples during model training. For samples with ambiguous attitude polarity, a selective attention network is used for interactive fusion; for samples with consistent attitude polarity, a common attention network is used for feature interaction and fusion.

[0045] Select Attention Network: In the proposed model, the attitude selective attention mechanism is used to enable the network to learn the semantic expression characteristics of each modality from sample situations involving multimodal attitude polarity ambiguity, so that the neural network pays attention to such special cases, which makes it different from the existing multimodal selective attention method. Figure 4 As shown in Figure 2, traditional selective attention methods learn the different contributions of all modalities based on the different roles they play in different scenarios. Unlike these methods, the model proposed in this paper learns the specific semantics expressed by different modalities and assigns different levels of importance to the different semantics of all modalities, enabling the model to understand situations involving attitude ambiguity.

[0046] In multimodal attitude analysis, each modality contains various semantics that can provide in-depth understanding of the modality. The interaction of these different semantics from different modalities can enable the network to learn higher-level inter-modal interaction relationships, which is very helpful for making the network familiar with situations involving multimodal polarity ambiguity. Based on the above analysis, a multi-head attention mechanism is introduced to capture the multiple semantic information of each modality. In this network module, multi-class semantics of different modalities are learned through these attention heads. The framework architecture of the network is as follows Figure 5As shown in the figure, each modality is divided into multiple attention heads containing different semantics. These attention heads interact with themselves and other attention heads through cross-attention. In the final stage, self-attention is used to learn the contribution of these interacted semantic features to the task. Then, they are weighted and connected according to their learned contributions, and finally used for the multimodal attitude and sentiment analysis task.

[0047] Multi-head attention consists of multiple parallel attention heads, each of which contains different semantic information from the original modality, corresponding to an independent tensor scaling dot product attention operation, which can be specifically expressed as: , in, Indicates modality No. attention heads, [,] represents feature connection, represents the multimodal cross attention mechanism, j represents the modality The number of semantic heads, For modal The linear change process of Represents the number of attention heads. All modal attention heads are connected through features to form an overall multimodal feature representation. Then, the contribution of different attention heads is calculated under a task to achieve attitude semantic selection. The formula is as follows: , , in Represents feature connection, represents the Hadamard product, The result of selective attention interaction.

[0048] Common Attention Network: In order to better learn and capture the cross-modal feature interactions between different modalities in samples with consistent attitude polarity, an attitude commonality attention network is designed to learn the common attitude expressions of each modality and enhance the attitude expression ability of multimodal representation.

[0049] Many existing studies have shown that in multimodal analysis tasks, text is the dominant modality for task learning. In this context, paying more attention to text information or guiding network learning through text information can improve the model's understanding of the overall semantics. This approach is particularly beneficial in samples where the attitude of the text is consistent with the attitude expression of other modalities. Therefore, an attitude commonality learning network is designed, which uses text to help non-text modalities learn rich semantic information and prevent the suboptimal multimodal representation due to the dominance of text representation in the multimodal training process. The framework of the attitude commonality attention network is as follows: Figure 6 shown.

[0050] In the attitude-common attention network, the text modality representation is decomposed into two different semantic representations to assist the learning process of different non-text modalities. The specific calculation formula is shown below.

[0051] , , in, and It is the audio feature expression guided by text and the visual feature expression guided by text. , They represent the learning parameters set for learning different emotional semantics of text modalities.

[0052] The final network output is: ,in, This is the common cross-attention result.

[0053] This embodiment first uses training samples containing unimodal attitude labels and multimodal attitude labels to train the adaptive attitude emotion polarity detection network, attitude polarity ambiguity diagnosis network, selective attention network and common attention network; then uses second sample data containing only multimodal labels to generate unimodal attitude labels based on the adaptive attitude emotion polarity detection network, and then trains the attitude polarity ambiguity diagnosis network, selective attention network and common attention network to reduce the workload and cost of labeling unimodal attitude labels.

[0054] In order to verify the effectiveness of the proposed model, this paper uses two types of datasets. The first type is the CH-SIMS.v2 dataset containing multimodal and unimodal labels, and the second type is the MOSI and MOSEI datasets with only multimodal labels.

[0055] We first conduct experiments on the CH-SIMS.v2 dataset, which contains multimodal and unimodal labels, and compare the results with existing methods. The comparison results are shown in Table 1.

[0056] Table 1 Experimental results of the model on the CH-SIMS.v2 dataset

[0057] As can be seen in Table 1, the proposed model achieves state-of-the-art performance on the CH-SIMS.v2 dataset. Specifically, our model surpasses the current state-of-the-art method, AV-MC, in all metrics, particularly in binary classification and correlation. Furthermore, our approach significantly outperforms methods that focus on complex fusion techniques but ignore inconsistencies in multimodal attitude polarity. This demonstrates the effectiveness of our approach in designing different networks for samples with varying inter-modal attitude polarity relationships.

[0058] When conducting experiments on the MOSI and MOSEI datasets, which only have multimodal labels, an adaptive attitude and emotion polarity detection network is required. The results of multimodal attitude analysis for MOSI and MOSEI are shown in Table 2. As can be observed from the table, the proposed model achieves the best performance on most metrics for both datasets.

[0059] Table 2 Experimental results of the model on CMU-MOSI and CMU-MOSEI datasets

[0060] In Table 2, the proposed model outperforms the Hgraph-CL method, which uses graph learning to capture multimodal relationships. In addition, the proposed method also surpasses the TeTFN method. Similar to the proposed method, TeTFN recognizes that text modalities contain more information directly related to attitudes and emotions, and outperforms non-text modalities in attitude and emotion analysis performance, which demonstrates the effectiveness of the proposed model in attitude and emotion analysis. Furthermore, compared with methods that only consider complex fusion techniques and ignore attitude and emotion inconsistency, the proposed method shows a significant advantage in all datasets, which confirms the effectiveness of the adaptive polarity detection strategy introduced in the model. The proposed model outperforms all compared models in binary classification, demonstrating its ability to accurately distinguish between positive and negative samples after classifying attitude inconsistency.

[0061] Step 4: Obtain the multimodal data to be identified and identify the user's attitude tendency based on the trained attitude tendency recognition model. The multimodal data to be identified is obtained by using a camera to obtain video data, a microphone to obtain audio data, and converting the audio data into text data.

[0062] Correspondingly, the present invention also provides a user attitude tendency recognition system, comprising: Multimodal data acquisition module: used to obtain multimodal data, including video, audio and the text corresponding to the audio; Feature extraction module: extract features from each modal data acquired; Attitude tendency recognition module: completes attitude tendency recognition based on the extracted features and attitude tendency recognition model.

[0063] Specifically, the multimodal data acquisition module uses a camera to acquire video data, uses a microphone to acquire audio data, and converts the audio data into text data.

[0064] The present invention also provides an interactive system, such as Figure 7 Shown, including: Data processing module: used to obtain video data and audio data input by the user, and convert the audio data into text data; perform feature extraction on video data, audio data and text data.

[0065] Algorithm embedded module: Attitude tendency recognition is completed based on the attitude tendency recognition model, and corresponding feedback is generated based on text data and attitude tendency recognition results, thereby improving feedback accuracy. This part of the content can be implemented based on large models such as LLAMA3. A text-to-speech model (TTS) can also be set to provide real-time conversation capabilities between users and machines. An audio energy detection model (AEC) is introduced to realize user speech monitoring.

[0066] Human-Computer Interaction Module: This module is used for user interaction and primarily provides a dialog interface for real-time communication between users and intelligent machines. It also provides a status display interface for visually monitoring the performance of the attitude recognition model in real-world scenarios and the results of real-time attitude and sentiment analysis.

[0067] Data storage module: used to store interaction content and attitude tendency recognition results.

Claims

1. A method for identifying user attitude tendency, characterized in that: include: Step 1: Obtain training samples. The training samples include multimodal data and labels. The modalities include video, audio, and the corresponding text of the audio. The labels include unimodal attitude labels and multimodal attitude labels. Step 2: Extract features from each modal data; Step 3: Build and train an attitude tendency recognition model. The attitude tendency recognition model includes: Modal encoder: Use the encoder corresponding to each modality to encode the extracted features to obtain single-modal features; Adaptive attitude and emotion polarity detection network: obtains the unimodal attitude label corresponding to each modality based on the data of each modality; Attitude Polarity Ambiguity Diagnosis Network: Based on unimodal features and unimodal attitude labels, it determines whether each modality has attitude polarity ambiguity. If so, it uses the selective attention network for feature interaction fusion; if not, it uses the common attention network for feature interaction fusion. Selective Attention Network: A multi-head attention mechanism is used to learn the contribution of various semantics of each modality to the task, and weighted connections are made based on the contribution to complete feature interaction fusion; Common Attention Network: Utilizes the different sentiment semantics of text modalities to guide the learning of non-text modalities and connects the learned features to achieve interactive feature fusion; Attitude tendency recognition network: outputs attitude tendency recognition results based on the fused features; Step 4: Obtain the multimodal data to be identified and identify the user's attitude tendency based on the trained attitude tendency recognition model.

2. The user attitude tendency identification method according to claim 1, characterized in that: The feature extraction steps in step 2 are: For text modalities: Word Tokenization is used to divide sentences into words or subword units; the BERT model is used for encoding to obtain semantic features; For video mode: perform frame sampling on the video; Use MTCNN for face detection and alignment; Use DeepFace model to extract facial features; For audio mode: sample the audio; Librosa or OpenSMILE is used for audio feature extraction.

3. The user attitude tendency identification method according to claim 1, characterized in that: The adaptive attitude emotion polarity detection network is trained based on each modal data and multimodal attitude labels. The specific steps are as follows: Based on the multimodal attitude labels, positive and negative reference values ​​are established to describe the unimodal sentiment polarity direction; Transform all modalities into the same feature space; The distances between each unimodal feature and the positive and negative reference values ​​are compared to determine the attitude polarity of each unimodal.

4. The method for identifying user attitude tendency according to claim 3, characterized in that: In the process of converting all modes into the same feature space, a loss function is used to reduce the feature distribution distance before and after the mode enters the feature space.

5. The user attitude tendency identification method according to claim 1, characterized in that: The training samples also include a second training sample, and the labels of the second training sample only include multimodal attitude labels.

6. The method for identifying user attitude tendency according to any one of claims 1 to 5, characterized in that: The method of obtaining the multimodal data to be recognized is: using a camera to obtain video data, using a microphone to obtain audio data, and converting the audio data into text data.

7. User attitude tendency recognition system, characterized by: include: Multimodal data acquisition module: used to obtain multimodal data, including video, audio and the text corresponding to the audio; Feature extraction module: extract features from each modal data acquired; Attitude tendency recognition module: performs attitude tendency recognition based on the extracted features and the attitude tendency recognition model described in claim 1.

8. The user attitude tendency recognition system according to claim 7, characterized in that: The multimodal data acquisition module uses a camera to acquire video data, a microphone to acquire audio data, and converts the audio data into text data.

9. An interactive system, characterized in that include: Data processing module: used to obtain video data and audio data input by the user, and convert the audio data into text data; perform feature extraction on the video data, audio data and text data; Algorithm embedded module: performs attitude tendency recognition based on the attitude tendency recognition model described in claim 1, and generates corresponding feedback based on text data and attitude tendency recognition results; Human-computer interaction module: used to interact with users; Data storage module: used to store interaction content and attitude tendency recognition results.

10. The interactive system according to claim 9, characterized in that: The interaction methods with users include text and / or voice.

Images