Emotion analysis method, system and equipment based on multi-modal information and large language model and medium

Abstract

The invention discloses an emotion analysis method, system and device based on multi-modal information and a large language model and a medium, belongs to the technical field of natural language processing and multi-modal calculation, and aims at solving the technical problem of how to overcome the defects of data distribution difference, modal credibility deviation and low generated data quality in the prior art. According to the technical scheme, the method comprises the following steps: collecting multi-modal data; multi-modal data preprocessing and feature extraction: preprocessing the text data, the image data and the audio data respectively and extracting corresponding features; feature alignment: mapping features of different modes of texts, images and audios to a unified potential space by adopting a cross-modal alignment algorithm, and eliminating semantic gaps among the modes; weight distribution: adopting a dynamic weight distribution mechanism to adaptively and dynamically distribute weights of different modes; performing cross-domain attribute-level sentiment analysis; evaluating the credibility; and calibrating the confidence coefficient.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and multimodal computing technology, and specifically to a sentiment analysis method, system, device and medium based on multimodal information and a large language model. Background Art

[0002] Traditional sentiment analysis methods rely on annotated data from specific domains. However, in emerging or cross-domain scenarios, annotated data is scarce and its distribution varies significantly. Existing data augmentation methods (such as template-based generation and masked language models) suffer from semantic incoherence and insufficient diversity in the generated text. Although large language models (LLMs) have performed well in generating text, the cross-domain data they generate can still lack domain relevance, hindering model generalization. Multimodal sentiment analysis integrates information from text, speech, and vision, but the credibility of different modalities varies. For example, text may have low credibility due to subjective expression, while speech and vision modalities are more likely to reflect true emotions. Existing methods fail to fully account for modality credibility bias, leading to overreliance on low-credibility modalities (such as text), affecting prediction accuracy. The text generated by cross-domain data augmentation methods has a poor semantic match with the target domain. Multimodal fusion strategies fail to effectively distinguish modal credibility, resulting in underutilization of high-credibility modal information. A unified framework combining cross-domain data augmentation with multimodal credibility perception is lacking.

[0003] Therefore, how to overcome the defects of data distribution differences, modal credibility deviations and low quality of generated data in existing technologies and improve the accuracy and relevance of sentiment analysis enhanced by cross-domain data is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The technical task of the present invention is to provide a sentiment analysis method, system, device and medium based on multimodal information and a large language model to solve the problem of how to overcome the defects of data distribution differences, modal credibility deviations and low quality of generated data in the existing technology, and improve the accuracy and relevance of sentiment analysis enhanced by cross-domain data.

[0005] The technical task of the present invention is achieved in the following manner: a sentiment analysis method based on multimodal information and a large language model, the method is specifically as follows:

[0006] Collect multimodal data: Acquire multimodal data such as text, images, audio, and video through standardized interfaces;

[0007] Multimodal data preprocessing and feature extraction: Preprocess text, image, and audio data separately and extract corresponding features to obtain features of different modalities of text, image, and audio;

[0008] Feature alignment: A cross-modal alignment algorithm is used to map features from different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities.

[0009] Weight allocation: A dynamic weight allocation mechanism is used to adaptively and dynamically allocate weights to different modalities;

[0010] Cross-domain attribute-level sentiment analysis: This involves building a domain-adaptive attribute dictionary based on the domain knowledge graph. Named entity recognition and dependency parsing are used to locate attribute keywords in the text. A hierarchical classifier is then used to determine the overall sentiment. Fine-grained sentiment intensity scores are then assigned to the attributes. Multimodal sentiment features are then integrated through a gating mechanism to obtain the corresponding sentiment classification.

[0011] Credibility evaluation: After evaluating the integrity of multimodal data, the sentiment prediction results of different modalities are compared, the inter-modality conflict index is calculated, and the credibility is evaluated based on the matching degree of the domain knowledge base;

[0012] Calibrate confidence: Use temperature scaling or Bayesian uncertainty estimation to quantify the confidence of the model output. When the confidence falls below a preset threshold, manual review is triggered or the user is prompted to provide additional information.

[0013] As a preference, the text, image and audio data are processed separately and the corresponding features are extracted as follows:

[0014] For text data, we use pre-trained language models for word embedding, and combine them with domain dictionaries to enhance domain adaptability and obtain text features.

[0015] For image data, convolutional neural networks are used to extract visual features, and combined with posture estimation models to identify facial expressions and gestures to obtain image features;

[0016] For audio data, spectral analysis is used to extract voiceprint features, and the emotional speech recognition model is combined to analyze the acoustic features of intonation and pauses to obtain audio features.

[0017] As a preferred method, the cross-modal alignment algorithm optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; the details are as follows:

[0018] Cross-modal contrastive learning;

[0019] Mode-adaptive regularization.

[0020] More optimally, cross-modal contrastive learning is as follows:

[0021] By contrasting the loss function design, the InfoNCE loss function is used to maximize the inter-modality similarity while minimizing the intra-modality difference;

[0022] Adding intra-modal consistency loss through intra-modal difference constraint to ensure the compact distribution of features in the same mode;

[0023] Cross-modal contrastive learning is achieved through the InfoNCE loss function and intra-modal difference constraints.

[0024] More preferably, the mode-adaptive regularization is as follows:

[0025] The maximum mean difference is used to minimize the difference in feature distributions of different modalities through distribution matching method;

[0026] Dynamic weights are introduced through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements;

[0027] Regularize the feature distribution between modalities to ensure that the feature vectors of different modalities are comparable in the latent space.

[0028] As a preferred embodiment, the dynamic weight allocation mechanism is as follows:

[0029] Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task;

[0030] Adaptive weight update, specifically: first design the gating weights based on the gated fusion network and normalize them through the Sigmoid function; then add the weight parameters to the total loss function based on gradient optimization, optimize the weight distribution through gradient descent, and prevent weight degradation through entropy regularization.

[0031] A sentiment analysis system based on multimodal information and a large language model, the system comprising:

[0032] The acquisition module is used to obtain multimodal data such as text, images, audio and video through standardized interfaces;

[0033] The preprocessing and feature extraction module is used to preprocess text, image, and audio data and extract corresponding features to obtain features of different modalities of text, image, and audio. For text data, a pre-trained language model is used for word embedding, and domain dictionary is combined to enhance domain adaptability and obtain text features. For image data, a convolutional neural network is used to extract visual features, and a posture estimation model is used to recognize facial expressions and gestures to obtain image features. For audio data, spectral analysis is used to extract voiceprint features, and an emotional speech recognition model is used to analyze the acoustic features of intonation and pauses to obtain audio features.

[0034] The feature alignment module uses a cross-modal alignment algorithm to map features from different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities.

[0035] The weight allocation module is used to adaptively and dynamically allocate weights of different modalities using a dynamic weight allocation mechanism;

[0036] The sentiment analysis module is used to build a domain-adaptive attribute dictionary based on the domain knowledge graph. It uses named entity recognition and dependency parsing to locate attribute keywords in the text, and employs a hierarchical classifier to determine the overall sentiment tendency. It then performs fine-grained sentiment intensity scoring on the attributes and fuses multimodal sentiment features through a gating mechanism to obtain the corresponding sentiment classification.

[0037] The credibility assessment module is used to evaluate the integrity of multimodal data, compare the sentiment prediction results of different modalities, calculate the inter-modal conflict index, and evaluate the credibility based on the matching degree of the domain knowledge base;

[0038] The confidence calibration module is used to quantify the confidence of the model output using temperature scaling or Bayesian uncertainty estimation. When the confidence is lower than the preset threshold, manual review is triggered or the user is prompted to provide additional information.

[0039] As a preferred method, the cross-modal alignment algorithm optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; the details are as follows:

[0040] Cross-modal contrastive learning: Specifically, through contrastive loss function design, the InfoNCE loss function is used to maximize inter-modal similarity while minimizing intra-modal differences. Intra-modal consistency loss is added through intra-modal difference constraints to ensure the compact distribution of features in the same modality. Cross-modal contrastive learning is achieved through the InfoNCE loss function and intra-modal difference constraints.

[0041] Modality-adaptive regularization: Specifically, it minimizes the difference in feature distributions between different modalities using the maximum mean difference method through distribution matching; introduces dynamic weights through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements; and regularizes the feature distribution between modalities to ensure that feature vectors of different modalities are comparable in the latent space.

[0042] The dynamic weight allocation mechanism is as follows:

[0043] Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task;

[0044] Adaptive weight update, specifically: first design the gating weights based on the gated fusion network and normalize them through the Sigmoid function; then add the weight parameters to the total loss function based on gradient optimization, optimize the weight distribution through gradient descent, and prevent weight degradation through entropy regularization.

[0045] An electronic device comprising: a memory and at least one processor;

[0046] Wherein, the memory stores a computer program;

[0047] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the sentiment analysis method based on multimodal information and a large language model as described above.

[0048] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned sentiment analysis method based on multimodal information and a large language model.

[0049] The sentiment analysis method, system, device, and medium based on multimodal information and a large language model of the present invention have the following advantages:

[0050] (1) The present invention not only enhances the ability of sentiment analysis by utilizing multimodal data, but also improves the reliability of the analysis results by evaluating the credibility of sentiment expressions;

[0051] (2) This invention improves the accuracy and relevance of sentiment analysis enhanced by cross-domain data through multimodal models and large prediction models, significantly improving the accuracy and robustness in cross-domain and multimodal sentiment analysis tasks, and solving the problems of data distribution differences, modality credibility deviations, and low generated data quality in the prior art;

[0052] (3) This invention improves the accuracy and coverage of sentiment analysis, enabling a better understanding of complex social interaction scenarios;

[0053] (4) The present invention enhances the performance of the model when facing diverse inputs, especially showing superior performance in processing unstructured or multimodal data;

[0054] (5) The present invention introduces a credibility assessment mechanism, which helps to reduce the misjudgment rate and provides higher decision support value, especially in high-risk application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] Attachment Figure 1 Schematic diagram of the sentiment analysis method based on multimodal information and large language model. DETAILED DESCRIPTION

[0057] The sentiment analysis method, system, device and medium based on multimodal information and a large language model of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] As attached Figure 1 As shown, this embodiment provides a sentiment analysis method based on multimodal information and a large language model, and the method is specifically as follows:

[0060] S1. Collect multimodal data: Acquire multimodal data such as text, images, audio, and video through standardized interfaces;

[0061] S2. Multimodal data preprocessing and feature extraction: Preprocess text, image, and audio data separately and extract corresponding features to obtain features of different modalities of text, image, and audio;

[0062] S3. Feature alignment: A cross-modal alignment algorithm is used to map features of different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities.

[0063] S4. Weight allocation: A dynamic weight allocation mechanism is used to adaptively and dynamically allocate weights to different modalities;

[0064] S5. Cross-domain attribute-level sentiment analysis: This involves building a domain-adaptive attribute dictionary based on the domain knowledge graph. Named entity recognition and dependency parsing are used to locate attribute keywords in the text. A hierarchical classifier is then used to determine the overall sentiment. Fine-grained sentiment intensity scores are then assigned to the attributes. Multimodal sentiment features are then integrated through a gating mechanism to obtain the corresponding sentiment classification. For example, in a medical review, if the text mentions "side effects" but the patient's expression is calm, the model needs to comprehensively determine the true sentiment intensity.

[0065] S6. Credibility evaluation: After evaluating the integrity of the multimodal data, the intermodal conflict index is calculated by comparing the sentiment prediction results of different modalities, and the credibility is evaluated based on the matching degree of the domain knowledge base;

[0066] S7. Calibrate confidence: Use temperature scaling or Bayesian uncertainty estimation to quantify the confidence of the model output. When the confidence is lower than the preset threshold, it triggers manual review or prompts the user to provide additional information.

[0067] In this embodiment, the text, image and audio data are processed and their corresponding features are extracted as follows:

[0068] For text data, we use pre-trained language models for word embedding, and combine them with domain dictionaries to enhance domain adaptability and obtain text features.

[0069] For image data, convolutional neural networks are used to extract visual features, and combined with posture estimation models to identify facial expressions and gestures to obtain image features;

[0070] For audio data, spectral analysis is used to extract voiceprint features, and the emotional speech recognition model is combined to analyze the acoustic features of intonation and pauses to obtain audio features.

[0071] There are significant differences in the semantic spaces of multimodal data (such as text, images, and audio). For example, the semantics of text depends on the language structure, while the semantics of images depends on visual features. This difference leads to a semantic gap when directly fusing features of different modalities, which reduces model performance. In this embodiment, the cross-modal alignment algorithm optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; the details are as follows:

[0072] (1) Cross-modal contrastive learning;

[0073] (2) Modal adaptive regularization.

[0074] The cross-modal contrastive learning in step (1) of this embodiment is specifically as follows:

[0075] ①Through contrast loss function design, the InfoNCE loss function is used to maximize the similarity between modalities while minimizing the intra-modal differences;

[0076] ② Adding intra-modal consistency loss through intra-modal difference constraints to ensure that the feature distribution of the same mode is compact;

[0077] ③ Cross-modal contrastive learning is achieved through InfoNCE loss function and intra-modal difference constraints.

[0078] The modal adaptive regularization in step (2) of this embodiment is specifically as follows:

[0079] ① Minimize the feature distribution differences of different modalities using the maximum mean difference through the distribution matching method;

[0080] ②Introducing dynamic weights through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements;

[0081] ③ Regularize the feature distribution between modalities to ensure that the feature vectors of different modalities are comparable in the latent space.

[0082] The dynamic weight allocation mechanism in this embodiment is as follows:

[0083] ① Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task;

[0084] ② Adaptive weight update: Specifically, we first design gated weights based on a gated fusion network and normalize them using a sigmoid function. Then, we add the weight parameters to the total loss function using gradient optimization, optimize the weight distribution through gradient descent, and prevent weight degradation through entropy regularization. For example, in a customer service conversation, if the user's voice is clear, we increase the audio weight and decrease the text weight.

[0085] Example 2:

[0086] This embodiment provides a sentiment analysis system based on multimodal information and a large language model, the system comprising:

[0087] The acquisition module is used to obtain multimodal data such as text, images, audio and video through standardized interfaces;

[0088] The preprocessing and feature extraction module is used to preprocess text, image, and audio data and extract corresponding features to obtain features of different modalities of text, image, and audio. For text data, a pre-trained language model is used for word embedding, and domain dictionary is combined to enhance domain adaptability and obtain text features. For image data, a convolutional neural network is used to extract visual features, and a posture estimation model is used to recognize facial expressions and gestures to obtain image features. For audio data, spectral analysis is used to extract voiceprint features, and an emotional speech recognition model is used to analyze the acoustic features of intonation and pauses to obtain audio features.

[0089] The feature alignment module uses a cross-modal alignment algorithm to map features from different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities.

[0090] The weight allocation module is used to adaptively and dynamically allocate weights of different modalities using a dynamic weight allocation mechanism;

[0091] The sentiment analysis module is used to build a domain-adaptive attribute dictionary based on the domain knowledge graph. It uses named entity recognition and dependency parsing to locate attribute keywords in the text, and employs a hierarchical classifier to determine the overall sentiment tendency. It then performs fine-grained sentiment intensity scoring on the attributes and fuses multimodal sentiment features through a gating mechanism to obtain the corresponding sentiment classification.

[0092] The credibility assessment module is used to evaluate the integrity of multimodal data, compare the sentiment prediction results of different modalities, calculate the inter-modal conflict index, and evaluate the credibility based on the matching degree of the domain knowledge base;

[0093] The confidence calibration module is used to quantify the confidence of the model output using temperature scaling or Bayesian uncertainty estimation. When the confidence is lower than the preset threshold, manual review is triggered or the user is prompted to provide additional information.

[0094] There are significant differences in the semantic spaces of multimodal data (such as text, images, and audio). For example, the semantics of text depends on the language structure, while the semantics of images depend on visual features. This difference leads to a semantic gap when directly fusing features of different modalities, which reduces model performance. The cross-modal alignment algorithm in this embodiment optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; specifically as follows:

[0095] Cross-modal contrastive learning: Specifically, through contrastive loss function design, the InfoNCE loss function is used to maximize inter-modal similarity while minimizing intra-modal differences. Intra-modal consistency loss is added through intra-modal difference constraints to ensure the compact distribution of features in the same modality. Cross-modal contrastive learning is achieved through the InfoNCE loss function and intra-modal difference constraints.

[0096] Modality-adaptive regularization: Specifically, it minimizes the difference in feature distributions across modalities using the maximum mean difference through a distribution matching approach; introduces dynamic weights through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements; and regularizes the feature distributions across modalities to ensure that feature vectors across modalities are comparable in the latent space.

[0097] The dynamic weight allocation mechanism in this embodiment is as follows:

[0098] Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task;

[0099] Adaptive weight updating involves designing gating weights based on a gated fusion network and normalizing them using a sigmoid function. Then, based on gradient optimization, the weight parameters are added to the total loss function, weight distribution is optimized using gradient descent, and entropy regularization is used to prevent weight degradation. For example, in a customer service conversation, if the user's voice is clear, the audio weight is increased and the text weight is decreased.

[0100] Example 3: Typical application scenario implementation

[0101] (1) AI digital human emotional interaction system, technical implementation:

[0102] ① Multimodal input: The digital human camera captures facial expressions, the microphone captures voice, and the text input box receives user instructions.

[0103] ② Emotion-driven: Dynamically adjust the digital human response strategy based on analysis results.

[0104] ③ Credibility feedback: If the user's expression contradicts the text, the digital human will proactively ask: "You seem troubled. Do you need further help?"

[0105] (2) Emotion-driven OCR and document analysis, technical process:

[0106] ① Multimodal OCR enhancement: Combines the table structure and text content in the image to improve the recognition accuracy of complex documents.

[0107] ② Sentiment correlation analysis: In documents such as invoices and contracts, keywords are extracted through OCR and the risk level is determined based on the context.

[0108] (3) Intelligent Logo Design System, Technical Process:

[0109] ① Multimodal input analysis: Users input their needs through text descriptions, reference images, and voice feedback.

[0110] ②Generation and evaluation: Generate logo candidates based on the diffusion model, and use the sentiment analysis module to evaluate whether the design conforms to the emotional tendency described by the user.

[0111] ③ Credibility verification: Compare the generated logo with the brand’s historical data to ensure consistency.

[0112] (IV) System implementation and deployment:

[0113] Hardware architecture:

[0114] ① Edge computing deployment: Use lightweight models to process data in real time on terminal devices.

[0115] ② Cloud expansion: Achieve elastic expansion through microservice architecture and support high concurrency scenarios.

[0116] Software Architecture:

[0117] ① Modular design: Each functional module is encapsulated as an independent service, supporting API calls and flexible combinations.

[0118] ② Explainable interface: Provides visualization tools to explain the model decision-making process to meet compliance requirements in finance, healthcare and other fields.

[0119] Example 4:

[0120] This embodiment also provides an electronic device, including: a memory and a processor;

[0121] wherein the memory stores computer-executable instructions;

[0122] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the sentiment analysis method based on multimodal information and a large language model in any embodiment of the present invention.

[0123] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0124] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0125] Example 5:

[0126] This embodiment also provides a computer-readable storage medium having a plurality of instructions stored therein, which are loaded by a processor to cause the processor to execute the sentiment analysis method based on multimodal information and a large language model in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be caused to read and execute the program code stored in the storage medium.

[0127] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0128] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0129] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0130] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentiment analysis method based on multimodal information and a large language model, characterized in that: The method is as follows: Collect multimodal data: Acquire multimodal data such as text, images, audio, and video through standardized interfaces; Multimodal data preprocessing and feature extraction: Preprocess text, image, and audio data separately and extract corresponding features to obtain features of different modalities of text, image, and audio; feature Alignment: A cross-modal alignment algorithm is used to map features of different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities. Weight allocation: A dynamic weight allocation mechanism is used to adaptively and dynamically allocate weights to different modalities; Cross-domain attribute-level sentiment analysis: This involves building a domain-adaptive attribute dictionary based on the domain knowledge graph. Named entity recognition and dependency parsing are used to locate attribute keywords in the text. A hierarchical classifier is then used to determine the overall sentiment. Fine-grained sentiment intensity scores are then assigned to the attributes. Multimodal sentiment features are then integrated through a gating mechanism to obtain the corresponding sentiment classification. Credibility evaluation: After evaluating the integrity of multimodal data, the sentiment prediction results of different modalities are compared, the inter-modality conflict index is calculated, and the credibility is evaluated based on the matching degree of the domain knowledge base; Calibrate confidence: Use temperature scaling or Bayesian uncertainty estimation to quantify the confidence of the model output. When the confidence falls below a preset threshold, manual review is triggered or the user is prompted to provide additional information.

2. The sentiment analysis method based on multimodal information and large language model according to claim 1 is characterized in that The text, image and audio data are processed and the corresponding features are extracted as follows: For text data, we use pre-trained language models for word embedding, and combine them with domain dictionaries to enhance domain adaptability and obtain text features. For image data, convolutional neural networks are used to extract visual features, and combined with posture estimation models to identify facial expressions and gestures to obtain image features; For audio data, spectral analysis is used to extract voiceprint features, and the emotional speech recognition model is combined to analyze the acoustic features of intonation and pauses to obtain audio features.

3. The sentiment analysis method based on multimodal information and large language model according to claim 1, characterized in that The cross-modal alignment algorithm optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; the details are as follows: Cross-modal contrastive learning; Mode-adaptive regularization.

4. The sentiment analysis method based on multimodal information and large language model according to claim 3 is characterized in that Cross-modal contrastive learning is as follows: By contrasting the loss function design, the InfoNCE loss function is used to maximize the inter-modality similarity while minimizing the intra-modality difference; Adding intra-modal consistency loss through intra-modal difference constraint to ensure the compact distribution of features in the same mode; Cross-modal contrastive learning is achieved through the InfoNCE loss function and intra-modal difference constraints.

5. The sentiment analysis method based on multimodal information and large language model according to claim 3 is characterized in that The modal adaptive regularization is as follows: The maximum mean difference is used to minimize the difference in feature distributions of different modalities through distribution matching method; Dynamic weights are introduced through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements; Regularize the feature distribution between modalities to ensure that the feature vectors of different modalities are comparable in the latent space.

6. The sentiment analysis method based on multimodal information and large language model according to claim 1, characterized in that The dynamic weight allocation mechanism is as follows: Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task; Adaptive weight update, specifically: first design the gating weights based on the gated fusion network and normalize them through the Sigmoid function; then add the weight parameters to the total loss function based on gradient optimization, optimize the weight distribution through gradient descent, and prevent weight degradation through entropy regularization.

7. A sentiment analysis system based on multimodal information and a large language model, characterized by: The system includes: The acquisition module is used to obtain multimodal data such as text, images, audio and video through standardized interfaces; The preprocessing and feature extraction module is used to preprocess text, image, and audio data and extract corresponding features to obtain features of different modalities of text, image, and audio. For text data, a pre-trained language model is used for word embedding, and domain dictionary is combined to enhance domain adaptability and obtain text features. For image data, a convolutional neural network is used to extract visual features, and a posture estimation model is used to recognize facial expressions and gestures to obtain image features. For audio data, spectral analysis is used to extract voiceprint features, and an emotional speech recognition model is used to analyze the acoustic features of intonation and pauses to obtain audio features. The feature alignment module uses a cross-modal alignment algorithm to map features from different modalities, such as text, images, and audio, into a unified latent space, eliminating the semantic gap between modalities. The weight allocation module is used to adaptively and dynamically allocate weights of different modalities using a dynamic weight allocation mechanism; The sentiment analysis module is used to build a domain-adaptive attribute dictionary based on the domain knowledge graph. It uses named entity recognition and dependency parsing to locate attribute keywords in the text, and employs a hierarchical classifier to determine the overall sentiment tendency. It then performs fine-grained sentiment intensity scoring on the attributes and fuses multimodal sentiment features through a gating mechanism to obtain the corresponding sentiment classification. The credibility assessment module is used to evaluate the integrity of multimodal data, compare the sentiment prediction results of different modalities, calculate the inter-modal conflict index, and evaluate the credibility based on the matching degree of the domain knowledge base; The confidence calibration module is used to quantify the confidence of the model output using temperature scaling or Bayesian uncertainty estimation. When the confidence is lower than the preset threshold, manual review is triggered or the user is prompted to provide additional information.

8. The sentiment analysis system based on multimodal information and large language model according to claim 7, characterized in that: The cross-modal alignment algorithm optimizes cross-modal alignment through joint training to ensure that the features of different modalities are comparable and complementary in the latent space; the details are as follows: Cross-modal contrastive learning: Specifically, through contrastive loss function design, the InfoNCE loss function is used to maximize inter-modal similarity while minimizing intra-modal differences. Intra-modal consistency loss is added through intra-modal difference constraints to ensure the compact distribution of features in the same modality. Cross-modal contrastive learning is achieved through the InfoNCE loss function and intra-modal difference constraints. Modality-adaptive regularization: Specifically, it minimizes the difference in feature distributions between different modalities using the maximum mean difference method through distribution matching; introduces dynamic weights through adaptive regularization coefficients to balance contrast loss and distribution matching loss according to task requirements; and regularizes the feature distribution between modalities to ensure that feature vectors of different modalities are comparable in the latent space. The dynamic weight allocation mechanism is as follows: Modal correlation calculation: Calculate the amount of single-modal information based on information entropy evaluation and quantify the correlation between modalities using a mutual information neural estimator based on mutual information estimation to evaluate the contribution of each modality to the current task; Adaptive weight update, specifically: first design the gating weights based on the gated fusion network and normalize them through the Sigmoid function; then add the weight parameters to the total loss function based on gradient optimization, optimize the weight distribution through gradient descent, and prevent weight degradation through entropy regularization.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the sentiment analysis method based on multimodal information and a large language model as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the sentiment analysis method based on multimodal information and a large language model as described in any one of claims 1 to 6.

Images