Emotion-rhythm-vision triple dynamic alignment algorithm based on AI multi-mode large language model

Abstract

The invention discloses an'emotion-rhythm-vision 'triple dynamic alignment algorithm based on an AI multi-modal large language model, and relates to the technical field of cross-modal data processing. The method comprises the following steps: acquiring and preprocessing multi-modal data, synchronously acquiring audio, video and text data, and performing cleaning and timestamp standardization; cross-modal feature extraction: extracting text emotional semantics through an LLaMA-2 model, obtaining audio time sequence rhythm through a DTW algorithm, and extracting video visual features through a DenseNet model; based on dynamic space-time alignment of ST-CrossAttention, multi-modal features are fused, and weights are distributed; and generating an interpretable output and analysis report. The system comprises a multi-modal acquisition module, a feature extraction module, an alignment engine and an output module. According to the method, accurate alignment and emotion fusion of multi-modal data can be realized, the accuracy and interpretability of emotion analysis are improved, and the method is suitable for scenes such as movie and television analysis and human-computer interaction.

Description

Technical Field

[0001] This invention relates to the technical field of the intersection of artificial intelligence and computer vision, and in particular to a dynamic alignment algorithm for “emotion-rhythm-visual” triples based on an AI multimodal large language model. Background Technology

[0002] In scenarios such as film and television production, video recommendation, and mental health monitoring, sentiment analysis is of great significance and is a necessary step to achieve deep semantic understanding of video content.

[0003] Traditional emotion recognition technologies primarily rely on single-modality analysis, for example: Text sentiment analysis: Models like BERT can only analyze the emotions expressed in the text content and cannot combine information from other modalities. For example, when analyzing movie dialogue, it can only judge the emotional tendency based on the content of the dialogue, but cannot perceive factors such as the atmosphere of the scene in which the dialogue is set, the rhythm of the music, etc., resulting in incomplete and inaccurate sentiment recognition.

[0004] Speech tone detection: Tools like OpenSmile focus on features such as tone and speech rate to determine emotion, but neglect the influence of other modalities on emotional expression. For example, in a sad scene, even if a character speaks in a steady tone, visual elements such as low lighting and slow camera transitions, along with audio elements such as somber background music, collectively create a sad emotional atmosphere. Speech tone detection alone will miss this important information, thus failing to accurately capture the overall emotional expression.

[0005] Existing multimodal fusion methods have significant drawbacks: Early fusion: This involves simply concatenating or adding data from multiple modalities during the feature extraction stage without considering semantic relationships and temporal alignment between them. For example, in video sentiment analysis, feature extraction from text, audio, and visual modalities may be out of sync, causing the fused features to fail to accurately reflect the true state of emotion. Similarly, in dialogue scenarios, text content may not match facial expressions, actions, or the rhythm of background music; this temporal disconnect can lead to misjudgment of sentiment.

[0006] Late-stage fusion: After extracting features from each modality separately, only simple weighted summation and other fusion operations are performed during the decision-making stage, which also ignores the dynamic temporal alignment between modalities. For example, the IEEE ICASSP 2023 paper pointed out that in intense action scenarios, bimodal models suffer from an emotion recognition error rate as high as 32% because they cannot accurately align the features of different modalities. This fragmented fusion approach cannot fully utilize the synergistic relationships between multimodalities and cannot accurately capture the complex expressions of emotions.

[0007] In sentiment analysis of video content, rhythm is an important dimension, but current technologies generally do not model it as an independent dimension. Psychological research confirms that, according to Neuron 2022, video pacing (such as editing frequency and shot duration distribution) significantly affects the intensity of viewers' emotional resonance. For example, fast-paced editing and strong background music beats can enhance viewers' tension and excitement, while slow-paced shot transitions and soothing music create a tranquil or oppressive atmosphere, thus affecting viewers' emotional perception.

[0008] Limitations of existing technologies: Current technologies primarily focus on the fusion of textual and visual modalities, neglecting the crucial role of rhythmic features in emotional expression. For example, when analyzing a sad scene, even if both the visuals and textual content align with sadness, a fast-paced background music or overly frequent camera cuts can weaken the expression of sadness. Therefore, existing technologies, by not modeling rhythm as an independent dimension, cannot comprehensively and accurately analyze the emotional content of video content.

[0009] In summary, existing technologies for video sentiment analysis suffer from limitations such as unimodal constraints, multimodal fragmentation, and lack of rhythm perception, failing to meet the needs of industries such as film and television production, advertising optimization, and mental health monitoring for accurate sentiment analysis. Summary of the Invention

[0010] This invention proposes a dynamic alignment algorithm for "emotion-rhythm-visual" triples based on an AI multimodal large language model, solving the problems mentioned in the background art. The technical solution of this invention is implemented as follows: The dynamic alignment algorithm for "emotion-rhythm-visual" triples based on an AI multimodal large language model includes: Step S1: Multimodal feature extraction: S11: Sentiment semantic feature extraction: The LLaMA-2 model is used to parse video subtitles and speech-to-text, and output sentiment probability vectors; S12: Temporal rhythm feature extraction: The dynamic time warping (DTW) algorithm is used to quantify the frequency of shot switching, the intensity of background music beats and the intervals of dialogue pauses to generate rhythm encoding vectors. S13: Visual focus feature extraction: Based on DenseNet-161, extract the composition features of the image, including the proportion of close-up shots, color saturation variance, and motion vector amplitude; Step S2: Dynamic alignment of triples: S21: The ST-CrossAttention mechanism is used to align triplet features. The calculation formula is as follows:

[0011] in, For sentiment query vectors, For rhythm key vectors, A visual value vector; S22: Dynamically assign weights based on the ResNet-50 scene classification results, with the following assignment rules: Dialogue scene redistribution: emotional semantics 60%±5%, temporal rhythm 20%±5%, visual focus 20%±5%; Action scene weighting: emotional semantics 30%±5%, temporal rhythm 50%±5%, visual focus 20%±5%; Step S3: Joint Optimization Through the joint loss function: Optimize the model, where: The triplet contrast loss has a value range of (0,1). The rhythm-emotion consistency loss is defined as (0, 0.7). The interpretability regularization term is calculated as follows:

[0012] Wherein, Grad-CAM is the visual focus heatmap, and Attention_Heatmap is the spatiotemporal cross-attention weight distribution.

[0013] And hyperparameters =0.5, =0.3, =0.2.

[0014] Furthermore, the extraction of S12 temporal rhythm features also includes: The frequency of shot switching is detected by using the HSV histogram difference threshold of adjacent frames ≥ 0.35; The background music beat intensity is calculated using Fourier transform to determine the fundamental frequency energy peak, and is divided into three levels of encoding based on energy intensity: weak (0-0.3), medium (0.3-0.7), and strong (0.7-1.0).

[0015] Furthermore, step S13, visual focus feature extraction, further includes: The motion vector amplitude within 5 consecutive frames is calculated using optical flow. A motion amplitude greater than 15 pixels / frame is defined as a high dynamic range scene. Close-up shot proportion detection uses the area proportion of the YOLOv5 face detection box, and a proportion greater than 40% is judged as a close-up shot.

[0016] Furthermore, the dynamic weight allocation in step S22 adopts the entropy weight method, specifically including: Calculate the information entropy of each modality feature:

[0017] Weighting ratios are as follows: .

[0018] Furthermore, the construction of the rhythm-emotion consistency loss function Lrhythm-emotion includes:

[0019] Calculate the Pearson correlation coefficient between the rhythm encoding vector and the sentiment vector;

[0020] A quadratic penalty term (0.7) is applied when the correlation coefficient is <0.7. r) 2 ×10.

[0021] A dynamic alignment system for implementing the above method includes: Multimodal acquisition module: synchronously acquires video subtitles, audio waveforms, and frame-by-frame image streams; Feature extraction module: Parallel operation of LLaMA-2 sentiment analysis unit, DTW rhythm coding unit, and DenseNet-161 visual unit; Alignment engine: Built-in ST-CrossAttention processor and dynamic weight assigner; Interpretable output module: Generates Grad-CAM heatmaps and sentiment analysis reports with timestamps.

[0022] Furthermore, the multimodal acquisition module also includes: Audio-video synchronization device: By aligning the MFCC feature with the DTW of lip movement detection, the audio-visual synchronization error is less than 80ms; Redundant data processing unit: Automatically triggers feature reconstruction when the single-mode signal-to-noise ratio is <15dB.

[0023] Furthermore, the alignment engine's workflow includes: (1) Receive the three-way feature tensor output by the feature extraction module; (2) Select a preset weight template based on the scenario classification results; (3) Calculate the weighted fusion vector using ST-CrossAttention:

[0024] Where α, β, γ are dynamic weighting coefficients, and α + β + γ = 1.

[0025] Furthermore, the interpretable output module includes: Heatmap generator: Marks the correlation strength between visual focal areas and sentiment tags; Timestamp auditing unit: Records the triplet weight allocation log for each video segment; Anomaly Detector: An alert is triggered when the rhythm-emotion correlation coefficient is less than 0.5 for three consecutive segments.

[0026] Compared with existing technologies, this solution has the following advantages: 1. Significantly improves emotion recognition accuracy. By dynamically aligning the "emotion-rhythm-visual" triples, the problem of temporal asynchrony of multimodal features is solved, enabling emotion semantics, temporal rhythm and visual focus to work together. This results in an emotion recognition accuracy of up to 94.7% on the CMU-MOSEI dataset, breaking through the technical bottleneck of existing single-modal or dual-modal analysis.

[0027] 2. Enhanced Multimodal Collaborative Expression Capabilities. This innovative approach incorporates video rhythm as an independent dimension in sentiment computation, breaking through the traditional bimodal (text + visual) framework. By aligning multimodal features using the Dynamic Time Warping (DTW) algorithm, the asynchronous temporal nature of multimodal expressions is resolved, reducing alignment error by 68% (based on tests on the UCF101 dataset). This enables more accurate capture of sentiment information from multimodal collaborative expressions.

[0028] 3. Provide an interpretable alignment mechanism. A spatiotemporal cross-attention model is proposed to generate heatmaps demonstrating the correlation strength between emotion, rhythm, and vision, making the cross-modal alignment process visible. Simultaneously, Grad-CAM visualization technology ensures the logical association between visual focus areas and emotion tags, meeting the reliability and interpretability requirements of medical, legal, and other scenarios, and complying with the FDA's interpretable AI standard (21 CFR Part 11).

[0029] 4. Achieve adaptive dynamic weight allocation. Based on a scene classifier (ResNet-50 pre-trained model), the triplet weights are automatically adjusted according to different scene types (such as dialogue scenes and action scenes), enabling the model to flexibly adapt to the sentiment analysis needs of different scenarios. Compared with the fixed-weight model, the accuracy is improved by 12.3%, enhancing the model's generalization ability and adaptability.

[0030] 5. Broad Application Prospects. This technology can be widely applied in various fields such as film and television production, advertising optimization, and mental health monitoring. For example, it can help film and television production teams optimize the emotional curve of films, improve the accuracy of advertising matching, and provide early warnings of mental health problems through video content analysis, demonstrating high commercial value and social benefits. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a structural block diagram of the video content "emotion-rhythm-visual" triple dynamic alignment system based on a multimodal large language model, as described in this invention. Figure 2 This is a flowchart of the "emotion-rhythm-visual" triple dynamic alignment algorithm based on an AI multimodal large language model according to the present invention. Figure 3 This is a diagram of the three-level alignment framework of the present invention; Figure 4 This is a flowchart of the dynamic weight allocation process of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0034] Reference Figure 1 ,and Figure 4 This invention provides a dynamic alignment algorithm for "emotion-rhythm-visual" triples based on an AI multimodal large language model, including: Step S1: Multimodal feature extraction: S11: Sentiment semantic feature extraction: The LLaMA-2 model is used to parse video subtitles and speech-to-text, and output sentiment probability vectors; S12: Temporal rhythm feature extraction: The dynamic time warping (DTW) algorithm is used to quantify the frequency of shot switching, the intensity of background music beats and the intervals of dialogue pauses to generate rhythm encoding vectors. S13: Visual focus feature extraction: Based on DenseNet-161, extract the composition features of the image, including the proportion of close-up shots, color saturation variance, and motion vector amplitude; Step S2: Dynamic alignment of triples: S21: The ST-CrossAttention mechanism is used to align triplet features. The calculation formula is as follows:

[0035] in, For sentiment query vectors, For rhythm key vectors, A visual value vector; S22: Dynamically assign weights based on the ResNet-50 scene classification results, with the following assignment rules: Dialogue scene redistribution: emotional semantics 60%±5%, temporal rhythm 20%±5%, visual focus 20%±5%; Action scene weighting: emotional semantics 30%±5%, temporal rhythm 50%±5%, visual focus 20%±5%; Step S3: Joint Optimization Through the joint loss function: Optimize the model, where: The triplet contrast loss has a value range of (0,1). The rhythm-emotion consistency loss is defined as (0, 0.7). The interpretability regularization term is calculated as follows:

[0036] Wherein, Grad-CAM is the visual focus heatmap, and Attention_Heatmap is the spatiotemporal cross-attention weight distribution.

[0037] And hyperparameters =0.5, =0.3, =0.2.

[0038] Furthermore, the S12 temporal rhythm feature extraction also includes: the shot switching frequency is detected by the difference threshold of HSV histogram between adjacent frames ≥ 0.35; the background music beat intensity is calculated by Fourier transform to calculate the fundamental frequency energy peak, and divided into three levels of encoding according to energy intensity: weak (0-0.3), medium (0.3-0.7), and strong (0.7-1.0).

[0039] Step S13, visual focus feature extraction, further includes: calculating the motion vector amplitude within 5 consecutive frames using optical flow; a motion amplitude > 15 pixels / frame is defined as a high dynamic scene; close-up proportion detection uses the area ratio of the YOLOv5 face detection box, and > 40% is judged as a close-up.

[0040] Furthermore, in step S22, the dynamic weight allocation adopts the entropy weight method, specifically including: calculating the information entropy of each modal feature; the weight allocation ratio is based on: calculating the information entropy of each modal feature.

[0041] Weighting ratios are as follows: ...

[0042] Furthermore, the construction of the rhythm-emotion consistency loss function Lrhythm-emotion includes: calculating the Pearson correlation coefficient between the rhythm encoding vector and the emotion vector; and applying a quadratic penalty term (0.7) when the correlation coefficient < 0.7. r) 2 ×10.

[0043] See Figure 2 and Figure 3 Furthermore, a corresponding dynamic alignment system for implementing the above method is provided, including: Multimodal acquisition module: synchronously acquires video subtitles, audio waveforms, and frame-by-frame image streams; Feature extraction module: Parallel operation of LLaMA-2 sentiment analysis unit, DTW rhythm coding unit, and DenseNet-161 visual unit; Alignment engine: Built-in ST-CrossAttention processor and dynamic weight assigner; Interpretable output module: Generates Grad-CAM heatmaps and sentiment analysis reports with timestamps.

[0044] Furthermore, the multimodal acquisition module also includes: Audio-video synchronization device: By aligning the MFCC feature with the DTW of lip movement detection, the audio-visual synchronization error is less than 80ms; Redundant data processing unit: Automatically triggers feature reconstruction when the single-mode signal-to-noise ratio is <15dB.

[0045] Furthermore, the alignment engine's workflow includes: (1) Receive the three-way feature tensor output by the feature extraction module; (2) Select a preset weight template based on the scenario classification results; (3) Calculate the weighted fusion vector using ST-CrossAttention:

[0046] Where α, β, γ are dynamic weighting coefficients, and α + β + γ = 1.

[0047] Furthermore, the interpretable output module includes: Heatmap generator: Marks the correlation strength between visual focal areas and sentiment tags; Timestamp auditing unit: Records the triplet weight allocation log for each video segment; Anomaly Detector: An alert is triggered when the rhythm-emotion correlation coefficient is less than 0.5 for three consecutive segments.

[0048] The following analysis will use a specific case as an example: The following detailed explanation of the multimodal sentiment analysis and alignment method of the present invention is provided in conjunction with a specific application scenario. This embodiment uses a clip from a classic movie (timestamps 3 minutes 12 seconds to 5 minutes 34 seconds) as the analysis object to verify the sentiment recognition and cross-modal feature fusion effect of the present invention in complex scenes.

[0049] 1. Input data

[0050] This embodiment inputs a video clip from 3 minutes 12 seconds to 5 minutes 34 seconds of a classic movie. This clip contains the following multimodal information: Video stream: 2 minutes and 22 seconds long (out of 142 seconds), containing a scene where character A (the protagonist) talks to character B in the prison library. The main visual elements of the scene are the gray walls, iron bars, and facial expressions of the characters inside the prison. Audio stream: includes character dialogue (such as character A saying the line "Hope is a good thing, maybe the best of things, and good things never die") and background ambient sounds (footsteps in the distance of the prison, the sound of iron gates closing). Text information: The text of the dialogue segment generated by speech-to-text technology (ASR) consists of 128 Chinese characters.

[0051] 2. Multimodal Feature Extraction

[0052] The cross-modal feature extraction module described in this invention is used to extract features from the input text, audio, and video data respectively. (1) Extraction of text sentiment semantic features The dialogue excerpt is input into a pre-trained LLaMA-2 sentiment analysis model (7B parameters). The model performs semantic parsing on sentiment keywords (such as "hope," "good thing," and "perfection") in the text and outputs a sentiment dimension score. On the dimension of "hope", the emotional intensity score of the line "Hope is a good thing, maybe the best of things, and good things never die" is 0.88 (the value ranges from 0 to 1, and the higher the score, the stronger the sense of hope). At the same time, output the text sentiment tags: ["hope", "steadfast"].

[0053] (2) Extraction of audio temporal rhythm features

[0054] The rhythmic features of the audio stream are encoded using the Dynamic Time Warping (DTW) algorithm: Temporal analysis of the Mel spectrum features of the audio stream was performed to detect speech pause intervals, speech rate changes, and background noise rhythm. The average speech interval within the segment was calculated to be 1.2 seconds, and the background ambient noise rate was 0.3 times / second. Based on the video shot switching frequency (calculated using the frame difference method), the average shot dwell time is 8.2 seconds, which is determined to be "slow rhythm", and a rhythm code value R=0.32 is generated (the value range is 0-1, and R≤0.5 is defined as slow rhythm).

[0055] (3) Video visual focus feature extraction

[0056] Visual features were extracted from keyframes of the video (1 frame every 5 seconds, for a total of 29 frames) using the DenseNet-161 deep learning model. Color characteristics: By statistically analyzing the RGB value distribution of pixels within the frame, it was found that gray-blue tones (RGB value range [100, 120, 140]) accounted for 81%, while warm colors (such as yellow and red) accounted for only 9%. Visual emotion mapping: Based on the preset color-emotion association rules (such as gray-blue tones corresponding to "oppression" and "dullness"), output visual feature labels: ["oppression", "enclosed space"], with a feature intensity score of 0.76 (range of 0-1, the higher the score, the stronger the oppression).

[0057] 3. Cross-modal feature alignment and sentiment fusion

[0058] The ST-CrossAttention alignment engine described in this invention dynamically fuses the aforementioned multimodal features: (1) Feature alignment Based on timestamps, text sentiment features (corresponding to the time of the dialogue at 3 minutes and 45 seconds), audio rhythm features (time distribution of the entire segment), and visual color features (frame distribution of the entire segment) are mapped to a unified timeline (3 minutes and 12 seconds - 5 minutes and 34 seconds). The association weights of each modality feature were calculated using an attention mechanism: the text sentiment feature had the highest direct association with "hope" (weight 0.42), while the audio slow rhythm feature (weight 0.28) and the visual oppression feature (weight 0.30) served as auxiliary context features.

[0059] (2) Sentiment prediction

[0060] Based on the aligned features and weights, the final sentiment score is calculated: the overall hope score is 0.79 (calculation formula: text feature score × 0.42 + rhythm feature correction value × 0.28 + visual feature correction value × 0.30, where rhythm and visual features are converted into correction coefficients for "hope" through preset rules). Output prediction confidence: 94.7% (obtained through the model's evaluation of feature consistency; higher consistency results in higher confidence).

[0061] 4. Output Results

[0062] The analysis report generated by the system of this invention includes the following: Key conclusion: The dominant emotion in this passage is "hope" (intensity 0.79), with a confidence level of 94.7%. Multimodal interpretation: At the textual level: the line "Hope is a good thing" directly conveys a strong sense of hope; In terms of timing: the slow-paced shots (averaging 8.2 seconds each) enhanced the weight and impact of the theme of "hope"; Visually: The predominantly gray-blue color scheme (81%) creates a depressing environment that contrasts with the theme of "hope," highlighting the preciousness of hope in adversity. Visualization output includes a heatmap of emotional intensity over time (the "hope" level peaks at 0.88 at 3 minutes and 45 seconds) and a pie chart showing the weight distribution of each modality feature.

[0063] This embodiment verifies the effectiveness of the present invention in the film sentiment analysis scenario: through the accurate extraction and dynamic alignment of multimodal features, it can integrate text semantics, audio rhythm and visual information to achieve in-depth analysis of complex emotions, and the output results are interpretable and can be widely used in film and television content analysis, audience sentiment prediction and other fields.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic alignment algorithm for "emotion-rhythm-visual" triples based on an AI multimodal large language model, characterized in that... include: Step S1: Multimodal feature extraction: S11: Sentiment semantic feature extraction: The LLaMA-2 model is used to parse video subtitles and speech-to-text, and output sentiment probability vectors; S12: Temporal rhythm feature extraction: The dynamic time warping (DTW) algorithm is used to quantify the frequency of shot switching, the intensity of background music beats and the intervals of dialogue pauses to generate rhythm encoding vectors. S13: Visual focus feature extraction: Based on DenseNet-161, extract the composition features of the image, including the proportion of close-up shots, color saturation variance, and motion vector amplitude; Step S2: Dynamic alignment of triples: S21: The ST-CrossAttention mechanism is used to align triplet features. The calculation formula is as follows: in, For sentiment query vectors, For rhythm key vectors, A visual value vector; S22: Dynamically assign weights based on the ResNet-50 scene classification results, with the following assignment rules: Dialogue scene redistribution: emotional semantics 60%±5%, temporal rhythm 20%±5%, visual focus 20%±5%; Action scene weighting: emotional semantics 30%±5%, temporal rhythm 50%±5%, visual focus 20%±5%; Step S3: Joint Optimization Through the joint loss function: Optimize the model, where: The triplet contrast loss has a value range of (0,1). The rhythm-emotion consistency loss is defined as (0, 0.7). The interpretability regularization term is calculated as follows: Wherein, Grad-CAM is the visual focus heatmap, and Attention_Heatmap is the spatiotemporal cross-attention weight distribution. And hyperparameters =0.5, =0.3, =0.

2.

2. The "emotion-rhythm-visual" triple dynamic alignment algorithm based on an AI multimodal large language model according to claim 1, characterized in that, The S12 temporal rhythm feature extraction also includes: The frequency of shot switching is detected by using the HSV histogram difference threshold of adjacent frames ≥ 0.35; The background music beat intensity is calculated using Fourier transform to determine the fundamental frequency energy peak, and is divided into three levels of encoding based on energy intensity: weak (0-0.3), medium (0.3-0.7), and strong (0.7-1.0).

3. The "emotion-rhythm-visual" triple dynamic alignment algorithm based on an AI multimodal large language model according to claim 1, characterized in that, Step S13, visual focus feature extraction, further includes: The motion vector amplitude within 5 consecutive frames is calculated using optical flow. A motion amplitude greater than 15 pixels / frame is defined as a high dynamic range scene. Close-up shot proportion detection uses the area proportion of the YOLOv5 face detection box, and a proportion greater than 40% is judged as a close-up shot.

4. The "emotion-rhythm-visual" triple dynamic alignment algorithm based on an AI multimodal large language model according to claim 1, characterized in that, The dynamic weight allocation in step S22 adopts the entropy weight method, specifically including: Calculate the information entropy of each modality feature: Weighting ratios are as follows: .

5. The "emotion-rhythm-visual" triple dynamic alignment algorithm based on an AI multimodal large language model according to claim 1, characterized in that, The construction of the rhythm-emotion consistency loss function L rhythm-emotion includes: Calculate the Pearson correlation coefficient between the rhythm encoding vector and the sentiment vector; A quadratic penalty term (0.7) is applied when the correlation coefficient is <0.

7. r) 2 ×10.

Images