Multi-modal opinion expression recognition system and method driven by large language model
Through a multimodal opinion expression recognition system driven by a large language model, combined with text and speech embedding, the problem of complex emotions and multimodal information integration is solved, and a more accurate opinion expression recognition effect is achieved.
Patent Information
- Application Number
- CN202410653682.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-24
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to deal with complex emotions in the recognition of opinions, especially ridicule or irony, and the lack of effective inter-modal alignment mechanisms in multimodal research, resulting in the neglect of speech data and the difficulty in integrating multimodal information.
The multimodal opinion expression recognition system driven by a large language model is realized through the large language model coding module, speech coding module, modal adaptation module and large language model fusion modal analysis module, combined with text and speech embedding, to realize speech and text Seamless docking and information fusion of features generate comprehensive vector representations for opinion expression recognition.
A more accurate and comprehensive emotional and opinion analysis is achieved, and the effect of multimodal opinion expression recognition is improved, especially when processing complex emotional and speech data, which significantly improves the accuracy and delicateness of recognition.
Smart Images

Figure CN118917407A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security technology, and in particular to a random password verification method based on smart devices. Background Art
[0002] With the rapid development of the Internet and social media, people can access social networks anytime and anywhere with just mobile devices.
[0003] The complexity of human communication lies in its multimodality, timing and asynchronous nature. It essentially integrates three main modes: language (such as vocabulary selection and sentence structure), vision (including facial expressions and body language) and sound (such as intonation and speech speed). These modes are not isolated in the communication process, but complement and coordinate with each other to form a complex asynchronous interaction sequence. With the exponential growth of the amount of text, pictures and video content on the Internet, social media has become a valuable data source in the field of natural language processing, especially in the field of opinion and emotion recognition. In the field of social media text processing, opinion expression recognition has become an increasingly important research topic. The core of its research is to accurately identify text paragraphs containing specific opinions and the emotional tendencies expressed by these texts. Through accurate opinion expression recognition, researchers can lay the foundation for more downstream application tasks involving sentiment analysis. For example, in the medical field, integrating opinion expression recognition technology can significantly enhance the patient experience in digital medical services; at the same time, this technology can also effectively monitor public sentiment and public opinion trends on social media platforms. In addition, companies can also use opinion expression recognition technology to more effectively collect and analyze customer feedback to optimize products and services. Therefore, opinion expression recognition has important practical research significance.
[0004] In the research and application process of opinion mining, the rapid development of generative large language models (such as T5 and GPT) marks a major leap in text processing technology. These large language models are trained on massive amounts of text data and have acquired powerful language understanding and generation capabilities, making them excellent in many natural language processing tasks. Especially in the fields of opinion mining and sentiment analysis, large language models can more accurately identify the opinions in the text through deep semantic analysis, and more carefully capture the implicit meaning and complex emotions in the text, such as sarcasm or pun, which are difficult to handle with traditional methods. For the above problems, existing research has proposed some solutions. The most direct solution is to use neural networks for text analysis. However, neural network-based methods will find it difficult to handle complex emotions in the text, such as sarcasm or irony. This is because most of the current opinion expression recognition data sets do not take this into account. For more complex emotion recognition in sentences, the focus is on using large language models and creating context-specific prompts for opinion mining tasks. The context is provided to the large language model as auxiliary information in the form of dialogue to assist the large language model in making judgments. However, the application of using large language models in opinion expression recognition has not been explored. In addition, the above methods all have a limitation that it is impossible to obtain sufficient emotional information through text modality alone. Relatively speaking, if other modalities, such as voice modality, are introduced, it will bring more delicate emotional content, because emotional color can significantly change the emotional polarity of the same text.
[0005] Previous studies have shown that there are still several challenges in the field of opinion expression recognition, including insufficient data sets, difficulties in traditional neural network models in processing complex emotions, and large differences from problems encountered in real-life situations. In addition, although the current multimodal research field has made significant progress, it mainly focuses on the combination of images and texts, while multimodal research on speech is relatively rare. This phenomenon shows the imbalance between the types of data and the scope of application in the multimodal field, especially in the processing and understanding of speech data. Moreover, although these methods have opened up new directions for multimodal research, there are still many challenges. First, the preliminary cascade method often lacks an effective inter-modal alignment mechanism, which may lead to the omission or misunderstanding of key task instructions when processing multimodal information. Second, although the introduction of lightweight adapters helps the model understand different modalities, most existing methods tend to treat the speech modality and text modality as modalities containing the same speech, which may cause the model to ignore the unique attributes of speech data, such as emotion and intonation information. Therefore, although current multimodal research has made some progress in the combination of speech and text, how to effectively integrate information from different modalities, especially how to retain and utilize the unique information in speech data, remains an important issue to be resolved.
[0006] In real life, opinions are usually expressed in multiple modalities, such as videos and audiobooks, which contain not only text information but also rich speech and visual elements. These multimodal information contain important emotions and opinion tendencies. For example, the pitch and speed of speech may reveal the emotional state of the speaker. However, current research on extracting opinion expressions from these multimodal data is still quite limited.
[0007] Although large language models have performed well in many natural language processing fields, their application in opinion expression recognition tasks has not been explored. Introducing large language models to realize opinion expression recognition, in view of the above multi-dimensional analysis and observation, a novel multimodal opinion expression recognition method, namely, a multimodal opinion expression recognition method (STOEI) and a novel system design to implement the technology are technical problems to be solved by the present invention. Summary of the invention
[0008] To solve the above problems, the present invention proposes a large language model driven multimodal opinion expression recognition system and method, which is based on a new adaptation architecture combined with a large language model, prompts to fine-tune the large language model and fuse text embedding and speech embedding, to more comprehensively understand and utilize the information in multimodal data for opinion expression recognition. The new task of multimodal opinion expression recognition is an emerging field that has not been fully explored in academia, and it aims to explore the alignment mechanism between modalities in more depth.
[0009] In order to achieve the above-mentioned object of the invention, the present invention proposes the following technical solutions:
[0010] A multimodal opinion expression recognition system driven by a large language model, comprising:
[0011] The large language model encoding module is used to encode the input text and prompts to be processed, specifically converting the original text data into a high-dimensional text embedding representation H text ;
[0012] The speech encoding module is used for detailed processing of speech input, specifically extracting key acoustic features from speech signals, including but not limited to pitch, volume intensity, and language rhythm, mapping the speech into corresponding speech embeddings, and thus extracting the corresponding speech feature embedding A;
[0013] The modality adaptation module is used to achieve seamless connection and information fusion between speech and text modality features, map speech features to the same high-dimensional vector space as text features, and obtain the speech embedding representation H speech ;
[0014] Large language model fusion modal analysis module: used to provide the system with a large language model analysis of comprehensive modal features, embedding the text into a representation H text With the speech embedding representation H speech The two embeddings are connected at the vector level to obtain a comprehensive vector representation H that integrates multimodal information. The expression is as follows:
[0015] H=H text ⊕H speech
[0016] Among them, ⊕ represents vector-level connection;
[0017] After successfully connecting and integrating the features of speech and text modalities, the data of the two modalities are combined to conduct a comprehensive sentiment and opinion analysis.
[0018] A multimodal opinion expression recognition system driven by a large language model A multimodal opinion expression recognition method driven by a large language model includes:
[0019] Initialization: First, define a piece of original speech as U and its corresponding text as S. A specific prompt learning template is added before the text S, and then a new text sequence X = (w1, w2, ..., w n ), where w represents the word in the sequence;
[0020] The speech encoding module is used as the text encoder of the large language model. The encoding mapping of text information is realized through the speech encoding module, that is, the text sequence X is mapped to a high-dimensional text embedding representation H text , and mapping the speech into the corresponding speech embedding, thereby extracting the corresponding speech feature embedding A, which is obtained from the last hidden layer of the speech encoding module;
[0021] The speech features are mapped to the same high-dimensional vector space as the text features through the modality adaptation module to obtain the speech embedding representation H speech ;
[0022] The adaptation process is expressed by the following equation:
[0023] H speech =Adapter(A)
[0024] After obtaining H text and H speech Afterwards, these two embeddings are concatenated at the vector level to obtain a comprehensive vector representation H that integrates multimodal information, expressed as follows:
[0025] H=H text ⊕H speech
[0026] Among them, ⊕ represents vector-level connection;
[0027] The comprehensive vector representation $\bm{H}$ that combines text features and speech features is input into the large language model, and the output probability distribution $\bm{O}$ is calculated to obtain the opinion expression recognition result.
[0028] Compared with the present invention, the characteristics and advantages are:
[0029] 1) For the first time, a solution for intelligent recognition of multimodal opinion expressions in real-world scenarios was proposed and implemented;
[0030] 2. Compared with other multimodal models, such as whispering-llama, the method proposed in the present invention considers multimodal samples more comprehensively in the field of multimodal opinion expression recognition, and the opinion expression recognition effect has achieved a leading level. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A module diagram of a multimodal opinion expression recognition system driven by a large language model of the present invention;
[0032] Figure 2 A flow chart of a multimodal opinion expression recognition method driven by a large language model of the present invention;
[0033] Figure 3 Example diagram of the learning template for prompts;
[0034] Figure 4 It is a schematic diagram of the model of the present invention. DETAILED DESCRIPTION
[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0036] The present invention first constructs a corresponding data set; then, in the model structure exploration part, the large language model is introduced into the traditional opinion expression recognition task. To address this gap in the field, the present invention introduces external knowledge, few-sample learning, and prompt fine-tuning of the large language model into opinion expression recognition in the form of dialogue guidance, and improves the performance of the large language model in the opinion expression recognition task by relying only on limited text resources; on the basis of opinion expression recognition driven by the large language model, the large language model and multimodal analysis technology are combined to perform opinion expression recognition in a multimodal context. This is to understand and identify the user's opinion expression at a deep level by analyzing the interaction and information complementarity between different modalities. Specifically, based on the opinion expression recognition method driven by the large language model, the present invention will conduct an in-depth study of the early fusion method of performing modal fusion of speech and text before the data is input into the model.
[0037] Example:
[0038] like Figure 1 As shown, a multimodal opinion expression recognition system driven by a large language model of the present invention mainly integrates the following five core components to enhance the model's ability to process complex emotions and opinion expressions:
[0039] Large Language Model Encoding Module (LLME): It is used to encode the input text and prompt to be processed. Specifically, it converts the original text data into a high-dimensional vector form that can be understood and processed by the large language model to capture the deep semantic features and sentiment tendencies in the text data, thereby generating a refined language feature representation.
[0040] Speech Encoding Module (VE): It is used for detailed processing of speech input, specifically extracting key acoustic features from speech signals, including but not limited to pitch, volume intensity, and language rhythm. These acoustic parameters are key clues to understanding human emotional states and are essential for improving the dimensionality and accuracy of emotion recognition. Generally, speech models are used for feature extraction.
[0041] Modal Adaptation Module (MA): This module serves as a bridge to achieve seamless connection and information fusion between the two modal features of speech and text. Specifically, the speech feature space is converted to a dimension that matches the text vector space through mapping, so that the data of the two different modalities can be effectively integrated on the same semantic plane. This process involves complex feature transformations, which aims to maximize the retention and utilization of cross-modal information. Among them, an adaptation structure is constructed in the modal adaptation module, which consists of two downsampling layers, a GELU activation function and an upsampling layer. The structure of the first downsampling layer is a one-dimensional convolutional network. This process is crucial for refining speech features to ensure that the size of the speech features can be reduced to a context size suitable for further analysis. It is followed by the subsequent downsampling layer, activation function and upsampling layer. These three layers form a standard small adapter framework, which significantly enhances the model's ability to capture and parse subtle emotions from speech data.
[0042] Large language model fusion modal analysis module (LLM): used to provide the system with large language model analysis with comprehensive modal features; specifically, after successfully connecting and integrating the voice and text modal features, the large language model has the ability to process voice information, enabling it to integrate the data of the two modalities for comprehensive sentiment and opinion analysis; the large language model uses the comprehensive features from both voice and text modalities for in-depth analysis to make more accurate and comprehensive sentiment and opinion judgments.
[0043] Furthermore, the system is equipped with a parameter fine-tuning module (LoRA): used to optimize and adjust the parameters of the large language model to improve the performance of the model on specific tasks; specifically, it introduces a low-rank matrix to adjust and adapt the original model parameters, making the model more flexible and efficient in the conversion and fusion process between different modalities. The LoRA module helps the model better understand and integrate data from different sources by fine-tuning the model weights, thereby improving the overall sentiment analysis performance.
[0044] like Figure 2 As shown, a multimodal opinion expression recognition method driven by a large language model of the present invention includes a total of 5 steps. The first step is to combine the prompt learning template and the text sequence to construct a new text sequence; the second step is to pass the text and speech through the large language model encoder and the speech encoder respectively to generate corresponding text embedding and speech features; the third step is the most critical step, which is to map the speech features to the text embedding space through the modality adapter; the fourth step is to merge the two embeddings; the fifth step is to send the merged embedding to the large language model for training and prediction. In this step, in order to quickly fine-tune the large language model, a parameter fine-tuning module (LoRA) is further set in the large language model. The specific description of each step is as follows:
[0045] Step 1: Initialization phase. First, define a piece of original speech as U and its corresponding text as S. Add a specific prompt learning template to the text S, and then construct a new text sequence X = (w1, w2, ..., w n ), where w represents the word in the sequence;
[0046] Step 2: Use the speech encoding module as the text encoder of the large language model to implement the encoding mapping of text information through the speech encoding module, that is, the text sequence X is mapped to a high-dimensional text embedding representation H text , and mapping speech into corresponding speech embedding, i.e., using the most advanced speech model, encoding the original speech U, and inputting it into an efficient speech encoder to extract the corresponding speech feature embedding A, which is obtained from the last hidden layer of the speech encoding module;
[0047] Step 3: Map the speech features to the same high-dimensional vector space as the text features through the modality adaptation module to obtain the speech embedding representation H speech. The adaptation structure is constructed in the modality adaptation module, which consists of two downsampling layers, a GELU activation function and an upsampling layer. The structure of the first downsampling layer is a one-dimensional convolutional network. This process is crucial for refining speech features to ensure that the size of speech features can be reduced to a context size suitable for further analysis, followed by the subsequent downsampling layer, activation function and upsampling layer. These three layers form a standard small adapter framework, which significantly enhances the model's ability to capture and parse subtle emotions from speech data.
[0048] The adaptation process is expressed by the following equation:
[0049] H speech =Adapter(A)
[0050] Step 4: After obtaining H text and H speech Afterwards, these two embeddings are concatenated at the vector level to obtain a comprehensive vector representation H that integrates multimodal information. The expression is as follows:
[0051] H=H text ⊕H speech
[0052] Among them, ⊕ represents vector-level connection;
[0053] Step 5: Input the comprehensive vector representation $\bm{H}$ that combines text features and speech features into the large language model, and calculate the output probability distribution $\bm{O}$ as the opinion expression recognition result. In the training stage, the parameter fine-tuning module (LoRA) is used to achieve low-rank decomposition of the large language model parameter matrix. This strategy can significantly reduce the number of parameters required in the model fine-tuning process. This study selects the cross entropy loss function as the training objective, which is defined as follows:
[0054]
[0055] Where yi, t, c is a one-hot encoding matrix. For sequence i at time step t, if the true next word is c, the value of the position is 1, otherwise it is 0.
[0056] Through this method, big languages can directly perform multimodal opinion expression recognition on speech and text.
[0057] In summary, the present invention has the following advantages:
[0058] First, this study is the first to propose a solution for multimodal opinion expression in real-world scenarios. This study proposes a new multimodal opinion expression recognition method based on early fusion. By mapping speech features to text feature space and then concatenating them with text, the large language model can directly process multimodal data, compared to other multimodal models, such as whispering-llama. The method proposed in this study has achieved a leading level in the field of multimodal opinion expression recognition.
[0059] Secondly, a prompt learning template for multimodal opinion expression recognition is designed. Through this template, not only the performance of the large language model in the opinion expression task can be greatly improved, but also the performance of the method proposed in this study in the multimodal opinion expression task can be improved. The experimental effect exceeds that of the large language model with zero-shot fine-tuning.
[0060] Third, compared with other multimodal methods, the method proposed in this study is also very sensitive to synthetic speech and can use synthetic speech as training data.
[0061] Fourthly, since this method uses LoRA fine-tuning, it does not require more computing resources in terms of training resources as previous methods.
[0062] Fifth, considering the asymmetry between speech and text in real life, a large number of asymmetric situations are added during model training, which can fit the real situation to the greatest extent.
[0063] Sixth, compared with other methods that require modification of the training paradigm or loss function, this method is data-driven, so there is no need to make any modifications to the pre-training paradigm and loss function of the large language model.
[0064]
[0065]
[0066] Among them, CIM-MOEI is a new multimodal opinion expression recognition dataset constructed in this study, and CIM-OEI means that it only contains the text part of the dataset. BERT-BiLSTTM represents the traditional deep learning model, and GPT4's zero-shot means directly testing on the dataset. Vicuna represents a large language model fine-tuned on the dataset. STOEI is the method proposed in this study. It can be seen that the method proposed in this study has shown the best results on the dataset of multimodal opinion expression recognition. At the same time, the large language model fine-tuned using the prompt learning template has also achieved very good results in the opinion expression recognition task.
[0068] This study first selected the open source model whisper-large-v2 with 1550M training parameters as the speech encoder and Vicuna 7b v1.5 as the large language model. The optimizer uses AdamW (β1=0.9, β2=0.95. The learning rate uses the cosine scaling strategy, the warm-up step ratio is set to 3%, and there is no weight decay. For the low-rank adaptation (LoRA) module, this study sets the adaptation rank to 8, the adjustment factor (Scaling Factor) to 16, and sets a dropout rate of 0.05 to prevent overfitting and promote the generalization ability of the model. During the training process, by default, the weights of the Whisper and LLaMA2 models are frozen, and only the weights of the modality adapter and LoRA modules can be trained.
Claims
1. A multimodal opinion expression recognition system driven by a large language model, characterized in that: include: The large language model encoding module is used to encode the input text and prompts to be processed, specifically converting the original text data into a high-dimensional text embedding representation H text ; The speech encoding module is used for detailed processing of speech input, specifically extracting key acoustic features from speech signals, including but not limited to pitch, volume intensity, and language rhythm, mapping the speech into corresponding speech embeddings, and thus extracting the corresponding speech feature embedding A; The modality adaptation module is used to achieve seamless connection and information fusion between speech and text modality features, map speech features to the same high-dimensional vector space as text features, and obtain the speech embedding representation H speech ; Large language model fusion modal analysis module: used to provide the system with a large language model analysis of comprehensive modal features, embedding the text into a representation H text With the speech embedding representation H speech The two embeddings are connected at the vector level to obtain a comprehensive vector representation H that integrates multimodal information. The expression is as follows: H=H text ⊕H speech Among them, ⊕ represents vector-level connection; After successfully connecting and integrating the features of speech and text modalities, the data of the two modalities are combined to conduct a comprehensive sentiment and opinion analysis.
2. The multimodal opinion expression recognition system driven by a large language model according to claim 1, characterized in that: The system is further provided with a parameter fine-tuning module for optimizing and adjusting the parameters of the large language model, specifically by introducing a low-rank matrix to adjust and adapt the original model parameters, and fine-tuning the model weights.
3. The multimodal opinion expression recognition system driven by a large language model according to claim 1, characterized in that: The adaptation structure in the modality adaptation module consists of two downsampling layers, a GELU activation function and an upsampling layer; the structure of the first downsampling layer is a one-dimensional convolutional network, so that the size of the speech feature is reduced to a context size suitable for further analysis.
4. The multimodal opinion expression recognition system driven by a large language model according to claim 2, characterized in that: The cross entropy loss function is selected as the training objective, which is defined as follows: Where yi, t, c is a one-hot encoding matrix. For sequence i at time step t, if the true next word is c, the value of the position is 1, otherwise it is 0.
5. A method for multimodal opinion expression recognition driven by a large language model in a multimodal opinion expression recognition system driven by a large language model according to any one of claims 1 to 4, characterized in that: include: Initialization: First, define a piece of original speech as U and its corresponding text as S. A specific prompt learning template is added before the text S, and then a new text sequence X = (w1, w2, ..., w n ), where w represents the word in the sequence; The speech encoding module is used as the text encoder of the large language model. The encoding mapping of text information is realized through the speech encoding module, that is, the text sequence X is mapped to a high-dimensional text embedding representation H text , and mapping the speech into the corresponding speech embedding, thereby extracting the corresponding speech feature embedding A, which is obtained from the last hidden layer of the speech encoding module; The speech features are mapped to the same high-dimensional vector space as the text features through the modality adaptation module to obtain the speech embedding representation H speech ; The adaptation process is expressed by the following equation: H speech =Adapter(A) After obtaining H text and H speech Afterwards, these two embeddings are concatenated at the vector level to obtain a comprehensive vector representation H that integrates multimodal information, expressed as follows: in, Represents vector level connection; The comprehensive vector representation $\bm{H}$ that combines text features and speech features is input into the large language model, and the output probability distribution $\bm{O}$ is calculated to obtain the opinion expression recognition result.
Citation Information
Patent Citations
Semantic segmentation method of attention mechanism based on deep learning
CN112287940A
Large language model training method and device for multi-modal content output and medium
CN117094419A
Emotion analysis method and system based on multi-modal prefix and cross-modal attention
CN117609882A
Emotion recognition method and system based on voice text cross-modal fusion
CN117765981A
Cited By
Low-resource language adaptive speech recognition method based on AdLoRA Plus
CN119943052A
Dialogue voice generation method and system based on emotion perception adapter and large model reasoning
CN120260539A
