Multi-modal sentiment analysis and interactive adjustment method and system based on large model

Abstract

The invention discloses a multi-modal sentiment analysis and interactive adjustment method and system based on a large model, and relates to the technical field of artificial intelligence and natural language processing. Comprising the steps of 1, collecting dialogue video data and preprocessing the video data, 2, respectively carrying out image feature extraction, voice feature extraction and text feature extraction according to video frames, audios and texts, 3, carrying out interactive fusion on every two modal features through a cross attention mechanism, and step 4, generating a corresponding dialogue strategy according to the emotion label in the recognition result and the requirement of appointed emotion adjustment, and carrying out interaction according to the dialogue strategy, so as to obtain the emotion recognition result of the multi-layer perceptron after the feature after the interaction fusion is subjected to average pooling processing and then is input into the multi-layer perceptron to carry out emotion classification, and obtaining a classification corresponding to the highest probability as the emotion label recognition result. Dynamically adjusting a dialogue strategy according to the change of the emotion label; the method is suitable for enhancing the emotion understanding and response ability of the intelligent dialogue system and improving the user experience.

Description

Technical Field

[0001] The present invention discloses a large-model-based multimodal sentiment analysis and interactive adaptation method and system, and relates to the technical field of artificial intelligence and natural language processing. Background Art

[0002] With the development of artificial intelligence technology, intelligent dialogue systems have been widely used in customer service, medical consultation, education and other fields. However, existing dialogue systems mainly rely on single text input for semantic understanding and response generation, and lack the ability to accurately identify and dynamically adapt to the user's emotional state. This limitation restricts the user experience of intelligent dialogue systems, especially in complex interactive application scenarios that require understanding of user multimodal information, such as language, expression, voice, etc., and emotional analysis and adaptation, such as smart medical care and psychological consultation.

[0003] Existing sentiment analysis methods usually use single-modal data, which can easily lead to information loss or misjudgment when processing multimodal data. In addition, existing sentiment analysis technologies are often directly applied to intelligent dialogue systems without an adaptation process, and cannot dynamically adapt to the user's real-time emotional state, especially in scenarios involving long-term interactions or emotional sensitivity, and cannot effectively cope with user emotional fluctuations, resulting in poor user experience. Summary of the invention

[0004] In view of the problems in the prior art, the present invention provides a large-model-based multimodal sentiment analysis and interactive adaptation method and system, which are suitable for enhancing the sentiment understanding and response capabilities of intelligent dialogue systems.

[0005] The specific scheme proposed by the present invention is: The present invention provides a multimodal sentiment analysis and interactive adaptation method based on a large model, comprising: Step 1: Collect conversation video data and pre-process the video data: extract and process video frames, audio and text respectively. Step 2: Perform image feature extraction, speech feature extraction, and text feature extraction based on video frames, audio, and text, respectively. Step 3: For each modality X, where X represents any one of the text feature T, speech feature A, and image feature I, generate the corresponding query vector , the key vector , value vector , expressed by the following formula: ; ; ; is a learnable weight matrix, is a bias term, which is used to fuse any two modalities based on the cross-attention mechanism, where the text feature T corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , the text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; The speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , the speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , image feature I corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ; The probability distribution of the output emotion category is obtained, and the category corresponding to the highest probability is the emotion label recognition result; Step 4: According to the emotion labels in the recognition results, specify the requirements for emotion adaptation, generate the corresponding dialogue strategy, interact according to the dialogue strategy, and dynamically adjust the dialogue strategy according to the changes in the emotion labels.

[0006] Furthermore, in step 1 of the multimodal sentiment analysis and interactive adaptation method based on a large model, the video data is preprocessed: Extract and process video frames: extract image frame sequences from the video at a preset frame rate. The extracted image frame sequence is processed, the resolution is adjusted to 224×224, and the image frame sequence is denoised and the contrast is enhanced. Extract and process audio: Use multimedia processing tools to separate the audio track from the video, separate the audio data based on the audio track, perform noise reduction and speech enhancement preprocessing on the audio data, Extract and process text: Convert the speech data in the audio data into text content, and remove noise words and repeated words from the text content.

[0007] Furthermore, in step 2 of the multimodal sentiment analysis and interactive adaptation method based on a large model, image features are extracted according to the image frame sequence extracted from the video frame: the preprocessed image frame sequence is input into the CLIP model, the image visual features are extracted using the visual encoder of the CLIP model, and global image visual features are generated by applying average pooling according to the time series.

[0008] Furthermore, in step 2 of the multimodal sentiment analysis and interactive adaptation method based on a large model, speech features are extracted based on audio: the audio data is first normalized and sampled at a sampling rate of 16 kHz, the samples are input into the HuBERT model, the feature averages of all time steps are calculated, and time-invariant global speech features are generated based on the feature averages. The global speech features include speech content information and speech emotional features. The speech emotional features include intonation, volume, and speaking speed.

[0009] Furthermore, in step 2 of the multimodal sentiment analysis and interactive adaptation method based on a large model, text features are extracted according to the text: the preprocessed text is input into the Baichuan-7B model, the text is segmented using the Baichuan-7B model, and the sentences are segmented to obtain the global feature vector of each segment of text, and then average pooling is applied to obtain the text features of the entire text.

[0010] Furthermore, in step 4 of the multimodal sentiment analysis and interactive adaptation method based on a large model, the emotional labels of the identification results involve happiness, sadness, anger, neutrality, fear, disgust, and surprise. The requirements for emotional adaptation are specified according to the emotional labels, and corresponding dialogue strategies are generated. The dialogue strategies are embedded in the command prompt of the large model, and interaction is performed according to the command prompt. At the same time, the dialogue strategies are dynamically adjusted according to changes in the emotional labels.

[0011] Furthermore, in step 4 of the multimodal sentiment analysis and interactive adaptation method based on a large model, the sentiment adaptation requirements specified by the sentiment tags are dynamically adjusted according to changes in the sentiment tags, the corresponding dialogue strategies are dynamically generated, the dialogue strategies are embedded in the instruction prompt of the large model, and interaction is performed according to the instruction prompt.

[0012] The present invention also provides a multimodal sentiment analysis and interactive adaptation system based on a large model, comprising a collection and processing module, a feature extraction module, a sentiment recognition module and an interactive adaptation module. The acquisition and processing module acquires conversation video data and pre-processes the video data: extracts and processes video frames, audio, and text respectively. The feature extraction module extracts image features, speech features and text features based on video frames, audio and text respectively. The emotion recognition module generates a corresponding query vector for each modality X, where X represents any one of the text feature T, speech feature A, and image feature I. , the key vector , value vector , expressed by the following formula: ; ; ; is a learnable weight matrix, is a bias term, which is used to fuse any two modalities based on the cross-attention mechanism, where the text feature T corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , the text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; The speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , the speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , image feature I corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ; The probability distribution of the output emotion category is obtained, and the category corresponding to the highest probability is the emotion label recognition result; The interaction adaptation module specifies the requirements for emotion adaptation based on the emotion labels in the recognition results, generates corresponding dialogue strategies, interacts according to the dialogue strategies, and dynamically adjusts the dialogue strategies according to changes in emotion labels.

[0013] The benefits of the present invention are: The present invention effectively combines multimodal data, such as text, voice, and images, with a large language model to construct an intelligent dialogue system with sentiment analysis and adaptation capabilities. It effectively integrates data from different modalities and performs dynamic sentiment adaptation, so that the intelligent dialogue interaction process can timely adjust the interaction strategy according to changes in user emotions, solve the problem of insufficient emotion understanding and interaction adaptation, and thus improve the naturalness and user experience in complex human-computer interaction scenarios during the intelligent dialogue process. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0016] Embodiment 1: The present invention provides a multimodal sentiment analysis and interactive adaptation method based on a large model, comprising: Step 1: Collect conversation video data and preprocess the video data: extract and process video frames, audio, and text separately.

[0017] The video data usually contains a video frame sequence and an audio track, which can capture the user's facial expressions and voice information. In step 1, the video data is preprocessed: Extract and process video frames: extract image frame sequences from the video at a preset frame rate. The extracted image frame sequence is processed, the resolution is adjusted to 224×224, and the image frame sequence is subjected to operations such as denoising and contrast enhancement for subsequent feature extraction.

[0018] Extract and process audio: Use the multimedia processing tool FFmpeg to separate the audio track from the video, separate the audio data based on the audio track, and perform pre-processing operations such as noise reduction and speech enhancement on the audio data to improve the clarity and intelligibility of the speech.

[0019] Extract and process text: Convert the speech data in the audio data into text content, and remove noise words and repeated words from the text content.

[0020] By temporally synchronizing the text, speech, and image sequences extracted from the video, aligning these data ensures that they reflect the user's overall emotional state at the same point in time.

[0021] Step 2: Perform image feature extraction, speech feature extraction, and text feature extraction based on video frames, audio, and text respectively.

[0022] Step 21: Extract image features based on the image frame sequence extracted from the video frame: Input the preprocessed image frame sequence into the CLIP model, use the visual encoder of the CLIP model to extract the image visual features, and apply average pooling based on the time series to generate global image visual features. CLIP (Contrastive Language-Image Pretraining) is a joint vision and language model. The CLIP model can effectively map images and texts into the same feature space, thereby achieving semantic understanding of images.

[0023] Step 22: Extract speech features based on audio: First, normalize the audio data and sample the audio at a sampling rate of 16kHz. Input the samples into the HuBERT model, calculate the feature averages of all time steps, and generate time-invariant global speech features based on the feature averages. The global speech features include speech content information and speech emotional features. The speech emotional features include intonation, volume, and speech speed. HuBERT (Hidden-Unit BERT) is a speech model based on the Transformer and BERT architectures. It generates hidden unit representations through self-supervised learning of speech signals, thereby extracting high-quality speech features.

[0024] Step 23: Extract text features based on the text: Input the preprocessed text into the Baichuan-7B model, use the Baichuan-7B model to segment the text, and perform word segmentation on the sentences to obtain the global feature vector of each text segment, and then apply average pooling to obtain the text features of the entire text. The Baichuan-7B model is a powerful pre-trained language model that can effectively capture the semantic and emotional information in the text through large-scale corpus training, and has strong context understanding and emotion recognition capabilities.

[0025] Step 3: For each modality X, where X represents any one of the text feature T, speech feature A, and image feature I, generate the corresponding query vector , the key vector , value vector , expressed by the following formula: ; ; ; is a learnable weight matrix, is a bias term, which is used to fuse any two modalities based on the cross-attention mechanism, where the text feature T corresponds to ( , , ) and the corresponding speech feature A ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get , The text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get ; Similarly, the speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get , The speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get , Image feature I corresponds to ( , , ) and the corresponding speech feature A ( , , ) performs cross-attention calculation between the two modalities. The formula is as follows, where d represents the vector dimension and the superscript T represents the transposition operation: , get , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ; The output is the probability distribution of emotion categories, and the category corresponding to the highest probability is the emotion label recognition result. The above process is that every two modal features are interactively fused through the cross attention mechanism, and the interactively fused features are input into the multi-layer perceptron for emotion classification after average pooling. The category corresponding to the highest probability is the emotion label recognition result; Step 4: According to the emotion labels in the recognition results, specify the requirements for emotion adaptation, generate the corresponding dialogue strategy, interact according to the dialogue strategy, and dynamically adjust the dialogue strategy according to the changes in the emotion labels.

[0026] The emotional labels of the recognition results may include happiness, sadness, anger, neutrality, fear, disgust, and surprise. According to the emotional labels, the requirements for emotional adaptation are specified, and the corresponding dialogue strategies are generated. The dialogue strategies are embedded in the instruction prompt of the large model, and the interaction is carried out according to the instruction prompt. For example, if the user's current emotional label is identified as "sadness", for the "sadness" emotion, it is necessary to generate comfort, understanding, or empathy content. Therefore, the requirements for emotional adaptation are specified to generate instruction prompts such as "comfort the user" and "express empathy", and dialogue strategies related to emotions are generated and embedded in the instruction prompt. That is, according to the emotional label "sadness", the instruction prompts "comfort the user" and "express empathy" are added to the prompt: "The user's emotional state is {sadness}, please generate a reply to comfort the user and express empathy", thereby explicitly requiring the large model to generate dialogue content with this emotional adaptation for interaction.

[0027] Furthermore, in step 4, according to the change of the emotional label, the emotional adaptation requirements specified by the emotional label are dynamically adjusted, the corresponding dialogue strategy is dynamically generated, the dialogue strategy is embedded in the instruction prompt of the large model, and the interaction is performed according to the instruction prompt. In multiple rounds of dialogue, the user's emotional state is continuously tracked, and the response generated during the interaction is dynamically adjusted according to the emotional changes. Specifically, after each round of dialogue, the user's emotional label is re-identified, and the emotional adaptation requirements for the next round of dialogue are adjusted according to the new emotional label. For example, when the emotion changes from "sadness" to "anger", the dialogue strategy that needs to be generated is a calmer and more rational response, which is embedded in the instruction, and the instruction prompt is dynamically adjusted: "The user's emotional state is {anger}, please generate a rational and calm response" to optimize the interaction, thereby realizing dynamic emotional adaptation and response generation.

[0028] In summary, the present invention can analyze the user's emotional state more comprehensively and accurately, and generate dialogue content that matches the user's emotion, thereby improving the user experience.

[0029] Embodiment 2: The present invention also provides a multimodal sentiment analysis and interactive adaptation system based on a large model, comprising a collection and processing module, a feature extraction module, a sentiment recognition module and an interactive adaptation module. The acquisition and processing module acquires conversation video data and pre-processes the video data: extracts and processes video frames, audio, and text respectively. The feature extraction module extracts image features, speech features and text features based on video frames, audio and text respectively. The emotion recognition module generates a corresponding query vector for each modality X, where X represents any one of the text feature T, speech feature A, and image feature I. , the key vector , value vector , expressed by the following formula: ; ; ; is a learnable weight matrix, is a bias term, which is used to fuse any two modalities based on the cross-attention mechanism, where the text feature T corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , the text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; The speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , the speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , image feature I corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ; is the output probability distribution of emotion categories, and the category corresponding to the highest probability is taken as the emotion label recognition result; The interaction adaptation module specifies the requirements for emotion adaptation based on the emotion labels in the recognition results, generates corresponding dialogue strategies, interacts according to the dialogue strategies, and dynamically adjusts the dialogue strategies according to changes in emotion labels.

[0030] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0031] Similarly, the system of the present invention effectively combines multimodal data, such as text, voice, images, etc. with a large language model to build an intelligent dialogue system with sentiment analysis and adaptation capabilities, effectively integrates data of different modalities, and performs dynamic sentiment adaptation, so that the intelligent dialogue interaction process can timely adjust the interaction strategy according to changes in user emotions, solve the problem of insufficient emotional understanding and interaction adaptation, thereby improving the naturalness and user experience in complex human-computer interaction scenarios during the intelligent dialogue process.

[0032] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0033] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A multimodal sentiment analysis and interactive adaptation method based on a large model, characterized by include: Step 1: Collect conversation video data and pre-process the video data: extract and process video frames, audio, and text respectively; Step 2: Perform image feature extraction, speech feature extraction and text feature extraction based on video frames, audio and text respectively; Step 3: For each modality X, where X represents any one of the text feature T, speech feature A, and image feature I, generate the corresponding query vector , the key vector , value vector , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ； The probability distribution of the output emotion category is obtained, and the category corresponding to the highest probability is the emotion label recognition result; Step 4: According to the emotion labels in the recognition results, specify the requirements for emotion adaptation, generate the corresponding dialogue strategy, interact according to the dialogue strategy, and dynamically adjust the dialogue strategy according to the changes in the emotion labels.

2. According to the multimodal sentiment analysis and interactive adaptation method based on a large model as described in claim 1, Its characteristic is that step 3 is expressed by the following formula: ； ； ； is a learnable weight matrix, is a bias term, and the fusion between any two modalities is performed based on the cross attention mechanism, where the text feature T corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculation between the two modalities to obtain , the text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain ; The speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain , the speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain , image feature I corresponds to ( , , ) and the corresponding speech feature A ( , , ) Perform cross-attention calculations between the two modalities to obtain .

3. According to the multimodal sentiment analysis and interactive adaptation method based on a large model according to claim 1, it is characterized by In step 1, the video data is preprocessed: Extract and process video frames: extract image frame sequences from the video at a preset frame rate. The extracted image frame sequence is processed, the resolution is adjusted to 224×224, and the image frame sequence is denoised and the contrast is enhanced. Extract and process audio: Use multimedia processing tools to separate the audio track from the video, separate the audio data based on the audio track, perform noise reduction and speech enhancement preprocessing on the audio data, Extract and process text: Convert the speech data in the audio data into text content, and remove noise words and repeated words from the text content.

4. According to the method of claim 1, the multimodal sentiment analysis and interactive adaptation method based on a large model is characterized by: In step 2, image features are extracted based on the image frame sequence extracted from the video frame: the preprocessed image frame sequence is input into the CLIP model, the image visual features are extracted using the visual encoder of the CLIP model, and average pooling is applied according to the time series to generate global image visual features.

5. According to the method of claim 1, the multimodal sentiment analysis and interactive adaptation method based on a large model is characterized by: In step 2, speech features are extracted based on the audio: the audio data is first normalized and sampled at a sampling rate of 16kHz. The samples are input into the HuBERT model, the feature averages of all time steps are calculated, and time-invariant global speech features are generated based on the feature averages. The global speech features include speech content information and speech emotional features. The speech emotional features include intonation, volume, and speaking rate.

6. The method of multimodal sentiment analysis and interactive adaptation based on a large model according to claim 1 is characterized by: In step 2, text features are extracted based on the text: the preprocessed text is input into the Baichuan-7B model, the Baichuan-7B model is used to segment the text, and the sentences are segmented to obtain the global feature vector of each paragraph of text, and then average pooling is applied to obtain the text features of the entire text.

7. According to the multimodal sentiment analysis and interactive adaptation method based on a large model as described in claim 1, it is characterized in that the emotional labels of the recognition results in step 4 involve happiness, sadness, anger, neutrality, fear, disgust, and surprise, and the requirements for emotional adaptation are specified according to the emotional labels, and the corresponding dialogue strategies are generated, and the dialogue strategies are embedded in the instruction prompt of the large model, and the interaction is performed according to the instruction prompt, and the dialogue strategies are dynamically adjusted according to the changes in the emotional labels.

8. The method of multimodal sentiment analysis and interactive adaptation based on a large model according to claim 7 is characterized by: In step 4, according to the change of the emotion label, the emotion adaptation requirements specified by the emotion label are dynamically adjusted, the corresponding dialogue strategy is dynamically generated, the dialogue strategy is embedded in the instruction prompt of the large model, and the interaction is performed according to the instruction prompt.

9. A multimodal sentiment analysis and interactive adaptation system based on a large model, characterized by It includes acquisition and processing module, feature extraction module, emotion recognition module and interactive adaptation module. The acquisition and processing module acquires the conversation video data and pre-processes the video data: extracting and processing the video frames, audio and text respectively; The feature extraction module performs image feature extraction, speech feature extraction and text feature extraction based on video frames, audio and text respectively; The emotion recognition module generates a corresponding query vector for each modality X, where X represents any one of the text feature T, speech feature A, and image feature I. , the key vector , value vector , Text features Concatenate the cross-attention results with other modalities to obtain , Concatenate the speech feature A with the cross-attention results of other modalities to obtain , Concatenate the image feature I with the cross-attention results of other modalities to obtain , right , and The average pooling is applied to obtain the fused multimodal feature representation, which is then input into the multi-layer perceptron MLP: ； is the output probability distribution of emotion categories, and the category corresponding to the highest probability is taken as the emotion label recognition result; The interaction adaptation module specifies the requirements for emotion adaptation based on the emotion labels in the recognition results, generates corresponding dialogue strategies, interacts according to the dialogue strategies, and dynamically adjusts the dialogue strategies according to changes in emotion labels.

10. The multimodal sentiment analysis and interactive adaptation system based on a large model according to claim 9, characterized in that The emotion recognition module is expressed by the following formula: ； ； ； is a learnable weight matrix, is a bias term, which is used to fuse any two modalities based on the cross-attention mechanism, where the text feature T corresponds to ( , , ) and the speech feature A corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , the text feature T corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain ; The speech feature A corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain , the speech feature A corresponds to ( , , ) and image feature I corresponding to ( , , ) Perform cross-attention calculations between the two modalities to obtain ; Image feature I corresponds to ( , , ) and the text feature T corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain , image feature I corresponds to ( , , ) and the speech feature A corresponding to ( , , ) Perform cross-attention calculation between the two modalities to obtain .

Images