Cross-media data fusion analysis method and system based on multi-modal LLM

Abstract

The invention discloses a multi-modal LLM-based cross-media data fusion analysis method and system, and the method comprises the steps: receiving multi-modal data containing text data, image data and audio data from different sources, inputting the multi-modal data into a unified encoder of a multi-modal LLM model, and carrying out the joint encoding, semantic feature representation with cross-modal association is generated; inputting the semantic feature representation into a cross-modal alignment module of a multi-modal LLM model to generate cross-medium embedding representation with consistent space; performing multi-scale feature interaction on the cross-medium embedding representation by using a hierarchical fusion network in the multi-modal LLM model to generate semantic enhanced unified fusion features; and inputting the unified fusion features into a multi-task prediction header of the multi-modal LLM model, and outputting a comprehensive analysis result with cross-modal consistency. According to the embodiment of the invention, the accuracy, consistency and cognitive depth of cross-modal analysis can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data analysis, and particularly relates to a cross-medium data fusion analysis method and system based on a multi-modal LLM. BACKGROUND

[0002] In the current big data era, the forms of information are increasingly diverse, and cross-medium data such as text, images and audio constitute the main body of information. How to deeply fuse and jointly analyze heterogeneous multi-modal data from different sources has become a key challenge in the field of artificial intelligence. Existing analysis methods mostly adopt a divide-and-conquer strategy, that is, independent models are used to process data of different modalities, and then simple feature splicing or decision fusion is performed. Such methods are difficult to effectively capture the complex nonlinear correlation between modalities, resulting in prominent semantic gap problems and inconsistency of analysis results. In addition, for multi-dimensional analysis tasks such as sentiment, event and theme, traditional schemes usually need to deploy multiple special models, which has high system redundancy and is difficult to guarantee the unity of cross-task cognition. Although pre-trained large language models have made progress in single-modal understanding, they still have limitations such as insufficient alignment and single interaction level when dealing with deep fusion and collaborative reasoning tasks, which restricts the improvement of cross-modal overall cognitive ability. SUMMARY

[0003] The purpose of the present application is to provide a cross-medium data fusion analysis method and system based on a multi-modal LLM to solve the problems in the prior art and improve the accuracy, consistency and cognitive depth of cross-modal analysis.

[0004] One embodiment of the present application provides a cross-medium data fusion analysis method based on a multi-modal LLM, which comprises:

[0005] Receiving multi-modal data containing text data, image data and audio data from different sources, and inputting the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal correlation;

[0006] Inputting the semantic feature representations into a cross-modal alignment module of the multi-modal LLM model to realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism, and generating spatially consistent cross-medium embedding representations;

[0007] Performing multi-scale feature interaction on the cross-medium embedding representations by using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network comprises a self-attention layer and a cross-attention layer, and generating semantic-enhanced unified fusion features;

[0008] The unified fusion features are input into a multi-task prediction head of the multi-modal LLM model to simultaneously perform sentiment analysis, content summarization, and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

[0009] Optionally, the multi-modal data including text data, image data, and audio data from different sources are received and input into a unified encoder of the multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal correlation, including:

[0010] Text data, image data, and audio data are received through a multi-source data interface, a data cleaning algorithm is used to remove noise and invalid information, and the image data is normalized in size and the audio data is unified in sampling rate to generate preprocessed multi-modal data;

[0011] The preprocessed multi-modal data are input into a text encoding branch, an image encoding branch, and an audio encoding branch of the unified encoder, word embedding sequences are extracted through the text encoding branch, visual feature maps are extracted through the image encoding branch, and acoustic feature sequences are extracted through the audio encoding branch to generate initial modal features;

[0012] The initial modal features are input into a cross-modal interaction layer of the unified encoder, and the correlation weights of the text features and the visual features and the correlation weights of the text features and the audio features are calculated through an inter-modal attention mechanism to generate a cross-modal attention map;

[0013] Based on the cross-modal attention map, the initial modal features are weighted and fused, and dimension alignment is performed through a fully connected layer to finally generate semantic feature representations with cross-modal correlation.

[0014] Optionally, the semantic feature representations are input into a cross-modal alignment module of the multi-modal LLM model to achieve deep alignment of text semantics, visual content, and audio features through a dynamic routing mechanism to generate spatially consistent cross-media embedding representations, including:

[0015] The semantic feature representations are input into a dynamic routing initialization layer of the cross-modal alignment module, an initial routing path is assigned according to the modal type, and an initial alignment vector of each modal feature is calculated to generate an initial routing configuration;

[0016] Based on the initial routing configuration, an iterative routing algorithm is used to calculate a feature similarity matrix between the text modal, the visual modal, and the audio modal, and the routing weights are dynamically updated to generate an optimized routing weight matrix;

[0017] The semantic feature representations are transformed using the optimized routing weight matrix, and the text features, the visual features, and the audio features are mapped to a shared hidden space through feature projection to generate a hidden space feature mapping;

[0018] The hidden space feature mapping is normalized to ensure consistent distribution of different modal features in the hidden space, and finally a space-consistent cross-media embedding representation is generated.

[0019] Optionally, the multi-scale feature interaction of the cross-media embedding representation is performed by using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network includes a self-attention layer and a cross-attention layer, and a semantic-enhanced unified fusion feature is generated, including:

[0020] The cross-media embedding representation is input into the self-attention layer of the hierarchical fusion network, the feature dependency relationship within each modality is calculated through the self-attention mechanism, the semantic consistency of the features within the modality is enhanced, and the intra-modal enhanced feature is generated;

[0021] The intra-modal enhanced feature is input into the cross-attention layer of the hierarchical fusion network, the feature interaction between the text modality and the visual modality, and the feature interaction between the text modality and the audio modality are calculated through the cross-attention mechanism, and the inter-modal interaction feature is generated;

[0022] The inter-modal interaction feature is aggregated in multiple scales, the feature representation of different granularities is extracted by using the pooling operation, and the multi-scale information is fused by feature concatenation, and a multi-scale fusion feature is generated;

[0023] The multi-scale fusion feature is subjected to nonlinear transformation, the feature expression capability is enhanced by using the activation function, and the original information is preserved through the residual connection, and finally the semantic-enhanced unified fusion feature is generated.

[0024] Optionally, the unified fusion feature is input into a multi-task prediction head of the multi-modal LLM model, and the sentiment analysis, content summary and event reasoning tasks are simultaneously performed, and a comprehensive analysis result with cross-modal consistency is output, including:

[0025] The unified fusion feature is input into a shared bottom layer of the multi-task prediction head, feature compression and dimension reduction are performed through a fully connected layer, and a task-shared feature vector is generated;

[0026] The task-shared feature vector is input into a sentiment analysis subnetwork, a content summary subnetwork and an event reasoning subnetwork in parallel, wherein the sentiment analysis subnetwork outputs a sentiment polarity probability, the content summary subnetwork outputs a summary text, and the event reasoning subnetwork outputs an event logical chain, and an initial task output is generated;

[0027] The cross-modal consistency of the initial task output is checked, the semantic consistency between the sentiment analysis result and the content summary, and the logical consistency between the content summary and the event reasoning are calculated through a consistency loss function, and a task output after consistency optimization is generated;

[0028] The emotional analysis result, the content summary and the event reasoning result after the integration consistency optimization are integrated to generate a structured comprehensive analysis report as the final output of the comprehensive analysis result with cross-modal consistency.

[0029] Yet another embodiment of the present application provides a cross-media data fusion analysis system based on a multi-modal LLM, comprising:

[0030] A receiving module is configured to receive multi-modal data containing text data, image data and audio data from different sources, and input the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal association;

[0031] A generating module is configured to input the semantic feature representations into a cross-modal alignment module of the multi-modal LLM model to realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism, and generate spatially consistent cross-media embedding representations;

[0032] An interacting module is configured to perform multi-scale feature interaction on the cross-media embedding representations by using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network includes a self-attention layer and a cross-attention layer, to generate semantic-enhanced unified fusion features;

[0033] An executing module is configured to input the unified fusion features into a multi-task prediction head of the multi-modal LLM model to simultaneously perform sentiment analysis, content summarization and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

[0034] Yet another embodiment of the present application provides a storage medium having a computer program stored therein, wherein the computer program is configured to execute the method described in any of the above embodiments when running.

[0035] Yet another embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory has a computer program stored therein, and the processor is configured to run the computer program to execute the method described in any of the above embodiments.

[0036] Compared with the prior art, the multi-modal LLM-based cross-medium data fusion analysis method provided by the application receives multi-modal data containing text data, image data and audio data from different sources, inputs the multi-modal data into a unified encoder of a multi-modal LLM model for joint coding to generate semantic feature representations with cross-modal association; inputs the semantic feature representations into a cross-modal alignment module of the multi-modal LLM model to generate spatially consistent cross-medium embedding representations; utilizes a hierarchical fusion network in the multi-modal LLM model to perform multi-scale feature interaction on the cross-medium embedding representations to generate semantic-enhanced unified fusion features; inputs the unified fusion features into a multi-task prediction head of the multi-modal LLM model to output comprehensive analysis results with cross-modal consistency, thereby being capable of improving the accuracy, consistency and cognitive depth of cross-modal analysis. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A hardware structure block diagram of a computer terminal for the multi-modal LLM-based cross-medium data fusion analysis method provided by the embodiment of the application is provided.

[0038] Figure 2 A flowchart of the multi-modal LLM-based cross-medium data fusion analysis method provided by the embodiment of the application is provided.

[0039] Figure 3 A structure diagram of the multi-modal LLM-based cross-medium data fusion analysis system provided by the embodiment of the application is provided. DETAILED DESCRIPTION

[0040] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the application, and cannot be explained as a limitation of the application.

[0041] The embodiment of the application first provides a multi-modal LLM-based cross-medium data fusion analysis method, which can be applied to electronic devices such as computer terminals, specifically, ordinary computers and the like.

[0042] The following will be described in detail by taking the running on the computer terminal as an example. Figure 1 A hardware structure block diagram of a computer terminal for the multi-modal LLM-based cross-medium data fusion analysis method provided by the embodiment of the application is provided. As shown in Figure 1 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory can include a non-volatile storage medium and an internal memory.

[0043] Referring to Figure 2 The embodiment of the application provides a multi-modal LLM-based cross-medium data fusion analysis method, which can include the following steps:

[0044] S201, receiving multi-modal data including text data, image data and audio data from different sources, and inputting the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal associations;

[0045] Specifically, the text data, image data and audio data can be received through a multi-source data interface, a data cleaning algorithm is used to remove noise and invalid information, and the image data is size normalized and the audio data is sampling rate unified to generate preprocessed multi-modal data.

[0046] First of all, the source scene of multi-modal data is determined. Taking "social media user-generated content" as an example, the text data comes from the API interface of the social media platform (such as the Weibo open platform API), which receives the user's published text comments (such as "Today I went camping with friends in the countryside, the scenery is super beautiful, and I also recorded the bird calls!"). The image data comes from the object storage service of the platform (such as Aliyun OSS), which receives the user's uploaded camping scene pictures (format is JPEG, exists 512x512, 192x192 two sizes); the audio data comes from the user's voice comment upload interface, which receives the user's 10-second bird call audio (exists 44.1kHz, 8kHz two sampling rates, format is WAV).

[0047] In the data cleaning stage, dedicated algorithms are used for different modalities of noise characteristics:

[0048] Text data cleaning: use regular expression algorithm to remove special symbols, meaningless characters and redundant information, for example, match and delete "#", "@", "http: / / " topic tags, user mentions and links (regular expression example: r'#.*?#|@.*?\s|http.*?\s'), filter invalid words with length less than 2 characters (such as "oh", "ah"), and convert the cleaned text to UTF-8 encoding. The above example text is cleaned to "Today I went camping with friends in the countryside, the scenery is super beautiful, and I also recorded the bird calls!".

[0049] Image data cleaning: use median filter algorithm to remove salt and pepper noise in the image (such as white noise points in the camping photo), filter window size is set to 3x3 (large window size will blur details, small window size cannot effectively denoise); then perform size normalization, all images are scaled to 224x224 pixels through bilinear interpolation algorithm (adapt to the input size of the ResNet model in the subsequent image encoding branch), ensure that the aspect ratio is inconsistent (such as 512x512 images are cropped to 224x224, 192x192 images are enlarged to 224x224).

[0050] Audio data cleaning: remove background noise (such as wind in bird call audio) using spectral subtraction algorithm, first extract noise segment (0.5 seconds of audio at the beginning without bird call) to calculate noise spectrum, then subtract noise spectrum from original audio spectrum; then perform sampling rate uniform processing, resample all audio to 16kHz (this sampling rate takes into account the processing needs of speech and natural sound, which is the standard input sampling rate of audio encoding branch MFCC algorithm) through linear interpolation algorithm, and simultaneously unify the audio length to 10 seconds (the tail of less than 10 seconds is zero-padded, and the first 10 seconds of more than 10 seconds is truncated).

[0051] The final pre-processed multi-modal data is: 1 piece of 28-character UTF-8 encoded text, 1 224x224 pixel RGB image, and 1 10-second 16kHz sample rate mono-channel WAV audio.

[0052] The pre-processed multi-modal data is input into the text encoding branch, image encoding branch and audio encoding branch of the unified encoder respectively, the word embedding sequence is extracted through the text encoding branch, the visual feature map is extracted through the image encoding branch, and the acoustic feature sequence is extracted through the audio encoding branch, to generate initial modal features;

[0053] The three branches of the unified encoder are adapted to different modal data, and pre-trained models are used for initialization to ensure the effectiveness of feature extraction:

[0054] 1. Text encoding branch (extracting word embedding sequence)

[0055] The pre-trained BERT-base model (12-layer Transformer, hidden layer dimension 768, vocabulary size 30522) is selected as the text encoder, and the input is the pre-processed text data, the processing flow is as follows:

[0056] First, Tokenize the text to divide it into 11 Tokens (such as "today" "and" "friends" "go" "out" "camping" "," "wind" "scene" "super" "beautiful" "," "also" "record" "bird" "call" "!"), the actual Token number needs to be combined with the BERT tokenization rules, this example has 22 Tokens), and add [CLS] (beginning of sentence marker) and [SEP] (end of sentence marker), the final Token sequence length is 24;

[0057] The model maps each Token to a 768-dimensional vector through an embedding layer (word embedding + position embedding + segment embedding), and then extracts context-related features through a 12-layer Transformer encoder. The output is a word embedding sequence with a dimension of "sequence length x hidden layer dimension", i.e., a 24x768 matrix, where each row vector corresponds to the semantic features of a Token (e.g., the vector corresponding to [CLS] contains the semantic information of the entire sentence, and the vector corresponding to "camping" contains the exclusive semantic information of the word).

[0058] 2. Image encoding branch (extract visual feature map)

[0059] A pre-trained ResNet50 model (50-layer convolutional neural network, containing 4 residual block groups) is selected as the image encoder, and the input is a preprocessed image with a size of 224x224x3 (RGB three channels). The processing flow is as follows:

[0060] The image is first processed by a 7x7 convolutional layer (64 convolutional kernels, step size 2) and a 3x3 max pooling layer (step size 2), which compresses the size to 56x56x64.

[0061] Then, the feature dimension is gradually increased and the spatial size is compressed through 4 residual block groups (containing 3, 4, 6, and 3 residual units, respectively). Finally, after processing by the average pooling layer, the output is a visual feature map with a dimension of 2048x7x7 (2048 is the number of feature channels, and 7x7 is the spatial resolution), where each channel corresponds to a visual feature (e.g., the first channel corresponds to the "green vegetation" feature, and the 500th channel corresponds to the "sky blue" feature).

[0062] 3. Audio encoding branch (extract acoustic feature sequence)

[0063] The MFCC (Mel Frequency Cepstral Coefficient) algorithm is used to extract acoustic features in combination with a pre-trained AudioNet model. The input is a 10-second 16kHz monaural audio (total sample points 160000), and the processing flow is as follows:

[0064] First, the audio is processed by frame, with a frame length of 25ms (corresponding to 400 sample points) and a frame shift of 10ms (corresponding to 160 sample points), resulting in a total of 991 frames (calculation formula: frame number = (total sample points - frame length) / frame shift +1).

[0065] Hanning windowing is performed on each frame (to reduce spectral leakage), and then the time-domain signal is converted into a frequency-domain signal by a fast Fourier transform (FFT) (with an FFT point number of 512), followed by mapping the frequency-domain signal to a mel frequency domain by a mel filter bank (40 filters), calculating the energy of each filter, and finally extracting the first 40 coefficients by a discrete cosine transform (DCT) to obtain 40-dimensional MFCC features;

[0066] The MFCC features are input into an AudioNet model (a 6-layer CNN) to further extract high-level acoustic features, and finally an acoustic feature sequence with a dimension of 40x991 is output (40 is the feature dimension, and 991 is the frame sequence length), and each frame feature corresponds to the acoustic properties of the audio at a certain time (for example, the 500th frame feature contains "bird call high frequency" information).

[0067] The initial modal features output by the three branches are: a text word embedding sequence of 24x768, an image visual feature map of 2048x7x7, and an audio acoustic feature sequence of 40x991.

[0068] The initial modal features are input into the cross-modal interaction layer of the unified encoder, and the correlation weights between the text features and the visual features and the correlation weights between the text features and the audio features are calculated by the inter-modal attention mechanism to generate a cross-modal attention map.

[0069] The core of the cross-modal interaction layer is to establish the semantic correlation between different modal features, and "Scaled Dot-Product Attention" (scaled dot-product attention) is used as the inter-modal attention mechanism to calculate the correlation weights in two steps:

[0070] 1. Text-visual feature correlation weight calculation

[0071] First, the initial modal features are dimensionally adapted, the image visual feature map (2048x7x7) is converted into a 49x2048 vector sequence (49=7x7, each element is a 2048-dimensional visual feature vector) by flattening operation, and then the text word embedding sequence (24x768) and the flattened visual feature sequence (49x2048) are both mapped to 512 dimensions (the dimension of the unified attention calculation) by a linear projection layer to obtain text projection features (24x512) and visual projection features (49x512).

[0072] Subsequently, the attention weight is calculated: assuming that the text projection feature is Q (query, 24x512), the visual projection feature is K (key, 49x512) and V (value, 49x512), the attention weight is calculated by the formula "attention weight = softmax (QxK^T / sqrt(d_k))" (d_k=512, sqrt(d_k) is used for scaling to avoid gradient disappearance), and a 24x49 text-visual attention weight matrix is obtained. Each element a_ij in the matrix represents the association strength between the i-th Token of the text and the j-th visual feature of the image, for example, the weight value of the text "camping" (the 10th Token) and the image "tent" visual feature (the 20th element) is 0.85 (strong association), and the weight value of the "sky" visual feature (the 5th element) is 0.12 (weak association).

[0073] 2. Text-audio feature association weight calculation

[0074] Similarly, the audio acoustic feature sequence (40x991) is linearly projected and mapped to 512 dimensions to obtain the audio projection feature (991x512). Assuming that the text projection feature is Q (24x512), the audio projection feature is K (991x512) and V (991x512), the same attention formula is used for calculation, and a 24x991 text-audio attention weight matrix is obtained. For example, the weight value of the text "bird call" (the 20th Token) and the audio "high-frequency bird chirp" frame feature (the 500th frame) is 0.92 (strong association), and the weight value of the "wind sound" frame feature (the 100th frame) is 0.05 (weak association).

[0075] The final generated cross-modal attention map contains two weight matrices: a 24x49 text-visual attention map and a 24x991 text-audio attention map. When each map is visualized in the form of a heat map, the deeper the color, the higher the association strength.

[0076] Based on the cross-modal attention map, the initial modal features are weighted and fused, and the dimensions are aligned through a fully connected layer to finally generate a semantic feature representation with cross-modal association.

[0077] First, the weighted fusion is performed, taking the text feature as the core (because the text semantics are more explicit, it is used as the association reference), and the cross-modal attention weight is used to weight the visual and audio features:

[0078] Text-visual weighted fusion: matrix multiplication between the 24x49 attention weight matrix and the flattened visual feature sequence (49x2048) to obtain the 24x2048 weighted visual features (each text Token corresponds to a feature vector fused with visual information), and then element-wise addition with the original text word embedding sequence (24x768) to obtain the 24x768 text-visual fusion features (the visual features are mapped to 768 dimensions through a linear layer to ensure dimension matching);

[0079] Text-audio weighted fusion: matrix multiplication between the 24x991 attention weight matrix and the audio acoustic feature sequence (991x40) to obtain the 24x40 weighted audio features, which are also mapped to 768 dimensions through a linear layer and then element-wise added with the text-visual fusion features (24x768) to obtain the 24x768 preliminary fusion features.

[0080] Subsequent dimension alignment: the preliminary fusion features are processed through two fully connected layers: the first fully connected layer maps the 768-dimensional features to 1024-dimensional features with ReLU (enhanced nonlinear expression) as the activation function; the second fully connected layer keeps the 1024-dimensional dimension unchanged and only performs linear transformation to ensure more stable feature distribution. The final generated semantic features are represented as a 24x1024 matrix, and each row vector corresponds to a cross-modal semantic feature of a text Token (e.g., the vector corresponding to "camping" contains the text "camping" semantic feature, image "tent" visual feature, and audio "environmental sound" acoustic feature), realizing the semantic association and integration of text, image, and audio.

[0081] S202, input the semantic feature representation into the cross-modal alignment module of the multi-modal LLM model, realize the deep alignment of text semantics, visual content, and audio features through a dynamic routing mechanism, and generate a spatially consistent cross-media embedding representation;

[0082] Specifically, the semantic feature representation can be input into the dynamic routing initialization layer of the cross-modal alignment module, the initial routing path is allocated according to the modality type, and the initial alignment vector of each modality feature is calculated to generate an initial routing configuration;

[0083] First of all, the semantic feature of the input represents the scene. Still taking "social media camping content" as an example, the text semantic feature is a 24x1024 matrix (24 text tokens, each token contains 1024-dimensional cross-modal semantic information such as "camping", "scenery", and "bird call"), the image semantic feature is a 49x1024 matrix (obtained by flattening a 7x7 visual feature map, 49 visual feature points correspond to elements such as tents, trees, and sky), and the audio semantic feature is a 991x1024 matrix (991 audio frames correspond to acoustic information such as bird calls and wind sounds).

[0084] The core of the dynamic routing initialization layer is to establish an initial association channel for different modalities, which is implemented as follows:

[0085] Initial routing path allocation: According to the strength of semantic association between modalities, the path is divided, and the text is taken as the semantic core (carrying clear meaning), and the direct routing of "text→visual" and "text→audio" is established first, and the indirect routing of "visual→audio" is supplemented, forming three initial paths, which are represented by modality identifiers: Path 1 (T→V, text to visual), Path 2 (T→A, text to audio), and Path 3 (V→A, visual to audio). The path allocation is based on the pre-data statistics that the semantic correlation between text and visual (cosine similarity average 0.35) is higher than that between visual and audio (0.28), so the direct routing weight is preset higher.

[0086] Initial alignment vector calculation: The alignment vector is the reference anchor point of the modal feature, which is generated by "intra-modal mean pooling" —— taking the mean value of the text semantic feature matrix in the Token dimension, obtaining a 1x1024 text initial alignment vector V_T (such as [0.23, -0.15, 0.42,..., 0.08]); taking the mean value of the image feature matrix in the visual point dimension, obtaining a 1x1024 image initial alignment vector V_V (such as [0.18, 0.35, -0.12,..., 0.11]); taking the mean value of the audio feature matrix in the frame dimension, obtaining a 1x1024 audio initial alignment vector V_A (such as [0.52, -0.08, 0.19,..., 0.21]).

[0087] Initial routing configuration generation: integrate paths and alignment vectors, configuration content includes "path identifier, source modality, target modality, initial weight, alignment vector", initial weight is preset according to relevance: path 1 (T→V) 0.4, path 2 (T→A) 0.4, path 3 (V→A) 0.2, ensure the total weight is 1. Example configuration fragment: "path 1: T→V, initial weight 0.4, V_T=[0.23,...0.08], V_V=[0.18,...0.11]; path 2: T→A, initial weight 0.4, V_T=[0.23,...0.08], V_A=[0.52,...0.21]; path 3: V→A, initial weight 0.2, V_V=[0.18,...0.11], V_A=[0.52,...0.21]".

[0088] Based on the initial routing configuration, the feature similarity matrix between the text modality, visual modality and audio modality is calculated by the iterative routing algorithm, and the routing weight is dynamically updated to generate the optimized routing weight matrix;

[0089] The goal of the iterative routing algorithm is to make the routing weight more consistent with the actual correlation strength between modalities through multiple rounds of optimization. The core parameter is the number of iterations (set to 3 times, too few times will make the weight unstable, and too many times will increase the calculation amount). The similarity measure uses cosine similarity (formula: cosθ=(A B) / (|A|×|B|), A, B are modality feature vectors, value range [-1,1], the larger the correlation is stronger).

[0090] Specific implementation process:

[0091] Feature similarity matrix calculation: generate a matrix for each of the three initial paths:

[0092] Path 1 (T→V): calculate the cosine similarity between each text Token vector (24) and each visual point vector (49) of the image, generate a 24×49 text-visual similarity matrix M_TV. For example, the similarity between the text "camping" Token (10th) and the image "tent" visual point (20th) is 0.85, and the similarity with "sky" visual point (5th) is 0.12;

[0093] Path 2 (T→A): calculate the similarity between text Token vector and audio frame vector (991), generate a 24×991 text-audio similarity matrix M_TA, such as "bird call" Token (20th) and "bird chirp" frame (500th) similarity is 0.92;

[0094] Path 3 (V→A): Calculate the similarity between visual point vectors and audio frame vectors, generate a 49x991 visual-audio similarity matrix M_VA, such as the similarity between the "tent" visual point and the "wind" frame is 0.08.

[0095] Dynamic update routing weights: adjust the weights based on the similarity matrix in each iteration:

[0096] First round: take the average of all elements of the similarity matrix of each path (path 1 average 0.35, path 2 average 0.32, path 3 average 0.28), normalize by the softmax function (formula: softmax (x_i)=e^x_i / Σe^x_j), get the weights: path 1 0.36, path 2 0.33, path 3 0.31;

[0097] Second round: weight the similarity matrix with the current weight (e.g. M_TV element x 0.36), recalculate the average and normalize, the weight is updated to path 1 0.38, path 2 0.34, path 3 0.28;

[0098] Third round: repeat the weighting calculation, the weight stabilizes to path 1 0.39, path 2 0.35, path 3 0.26 (change < 0.01, iteration terminates).

[0099] Optimized routing weight matrix generation: combine the similarity matrix of each path with the maximum weight, such as M_TV optimized matrix element is "M_TV (i,j) x 0.39", M_TA is "M_TA (i,j) x 0.35", M_VA is "M_VA(i,j) x 0.26", forming an optimized matrix that reflects "similarity + weight".

[0100] Use the optimized routing weight matrix to transform the semantic feature representation, map the text features, visual features and audio features to the shared hidden space through feature projection, generate hidden space feature mapping;

[0101] The shared hidden space needs to ensure that the feature dimensions of different modalities are consistent and semantically comparable, with a dimension of 512 (too low will lose information, too high will increase computational complexity, 512 is the optimal value commonly used in multi-modal fusion), projection uses "linear layer + batch normalization" structure to avoid feature distribution deviation.

[0102] Specific implementation:

[0103] Feature projection module construction: design an independent linear layer for each modality (input dimension 1024, output dimension 512), weight initialization using He normal distribution (adapt to subsequent nonlinear transformation), batch normalization layer is used to stabilize the mean and variance of the projected features (mean tends to 0, variance tends to 1):

[0104] Text semantic features (24x1024) are projected into 24x512 text hidden space features T_h through linear layer, such as the T_h vector of the "camping" token is [0.18, -0.09, 0.35,..., 0.07];

[0105] Image semantic features (49x1024) are projected into 49x512 image hidden space features V_h, and the V_h vector of the "tent" visual point is [0.16, -0.11, 0.32,..., 0.06];

[0106] Audio semantic features (991x1024) are projected into 991x512 audio hidden space features A_h, and the A_h vector of the "bird chirping" frame is [0.82, -0.05, 0.15,..., 0.09].

[0107] Feature transformation and fusion: based on the optimized routing weight matrix, cross-modal fusion is realized through "source modality feature x weight + target modality feature x weight":

[0108] Text-visual fusion: calculate T_h x M_TV optimization matrix ^T + V_h x M_TV optimization matrix to get 24x512 text-visual fusion hidden features TV_h (matrix multiplication is adapted to the dimension through the broadcast mechanism to ensure element-level weighting);

[0109] Text-audio fusion: calculate T_h x M_TA optimization matrix ^T + A_h x M_TA optimization matrix to get 24x512 text-audio fusion hidden features TA_h;

[0110] Visual-audio fusion: calculate V_h x M_VA optimization matrix ^T + A_h x M_VA optimization matrix to get 49x512 visual-audio fusion hidden features VA_h.

[0111] Hidden space feature mapping generation: integrate TV_h, TA_h, VA_h, all features are 512-dimensional, can directly compare semantic association in hidden space, for example, the cosine similarity between the TA_h vector of the "bird call" Token and the TA_h vector of the "bird chirping" frame is 0.88, which is significantly higher than 0.15 with the "wind sound" frame.

[0112] Normalize the hidden space feature mapping to ensure consistent distribution of different modal features in the hidden space, and finally generate a consistent cross-media embedding representation in the space.

[0113] Normalization uses L2 normalization (Euclidean normalization), which normalizes the length of the feature vector to 1, eliminating the distribution deviation caused by the difference in value range of different modal factors. The formula is x_norm = x / ||x|| (||x|| is the L2 norm of the vector, i.e. √(x1²+x2²+...+x 512 ²)).

[0114] Specific implementation:

[0115] Vector-by-vector normalization: all vectors in the hidden space feature mapping are processed one by one:

[0116] Text-visual fusion hidden feature TV_h (24x512): take the TV_h vector of the i-th Token, calculate its length, such as a vector length √(0.18²+(-0.09)²+...+0.07²)=1.2, the normalized vector is TV_h(i) / 1.2, and the length becomes 1;

[0117] Similarly, L2 normalize TA_h (24x512), VA_h (49x512) to get TV_h_norm, TA_h_norm, VA_h_norm.

[0118] Distribution consistency verification: calculate the statistical indicators of the normalized features to ensure that the distribution between modalities is close:

[0119] TV_h_norm (text-visual): mean 0.02, variance 0.01;

[0120] TA_h_norm (text-audio): mean 0.025, variance 0.011;

[0121] VA_h_norm (visual-audio): mean 0.03, variance 0.012;

[0122] The mean difference between the three is < 0.01, and the variance difference is < 0.002, meeting the distribution consistency requirements.

[0123] Cross-media embedding representation generation: integrate three types of normalized features to form the final representation: 24x512 text-visual embedding (TV_h_norm), 24x512 text-audio embedding (TA_h_norm), and 49x512 visual-audio embedding (VA_h_norm). In this representation, the associated features of different modalities are close in the hidden space (e.g., the "tent" visual embedding has a similarity of 0.82 with the "camping" text embedding), and the irrelevant features are far apart (e.g., the "sky" visual embedding has a similarity of 0.09 with the "bird call" audio embedding), achieving spatially consistent cross-modal alignment.

[0124] In S203, the multi-scale feature interaction of the cross-media embedding representation is performed using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network includes a self-attention layer and a cross-attention layer, and a semantic-enhanced unified fusion feature is generated.

[0125] Specifically, the cross-media embedding representation can be input into the self-attention layer of the hierarchical fusion network, and the feature dependency within each modality is calculated through the self-attention mechanism to enhance the semantic consistency of the features within the modality and generate intra-modal enhanced features.

[0126] First, the input cross-media embedding representation is defined. Continuing the "social media camping content" case, the embedding representation includes three types of modality features: text cross-media embedding (24x512, 24 text tokens, each token containing 512-dimensional aligned semantic information such as "camping", "scenery", and "bird call" feature vectors), image cross-media embedding (49x512, 49 visual feature points corresponding to aligned features of elements such as tents, trees, and sky), and audio cross-media embedding (991x512, 991 audio frames corresponding to aligned features of acoustic signals such as bird calls and wind sounds).

[0127] The core of the self-attention layer is to capture the correlation and dependency of features within a single modality (such as semantic correlation in text, spatial correlation in images, and temporal correlation in audio). An 8-head Scaled Dot-Product self-attention mechanism is used (the number of heads is set to 8 to balance computational efficiency and feature capture ability; the processing dimension of each head is 512 / 8=64 to avoid attention dispersion caused by excessively high single-head dimension), and the specific implementation is as follows:

[0128] Dimension adaptation before attention calculation: Linear projection is performed on the three types of modal embeddings to generate the query (Q), key (K), and value (V) matrices required for self-attention, and the dimensions of Q, K, and V are consistent with the original embeddings (text Q=K=V=24×512, image Q=K=V=49×512, audio Q=K=V=991×512). The projection weights are initialized through pre-training to ensure matching with the distribution of cross-media embeddings.

[0129] Multi-head attention splitting and calculation: Q, K, and V are split by the number of heads (8), and each head corresponds to 64-dimensional features. Taking the text modality as an example, the Q, K, and V dimensions of each head after splitting are 24×64, and the attention weights are calculated through the Scaled Dot-Product formula:

[0130] Step 1: Calculate the transpose matrix product of Q and K (24×64 × 64×24 = 24×24) to obtain the original attention score;

[0131] Step 2: Divide by √d_k (d_k=64, √d_k=8) to avoid saturation of the softmax function due to excessively large scores;

[0132] Step 3: Normalize the weights through the softmax function so that the sum of the weights of each Token to other Tokens is 1.

[0133] For example, in the text modality, the "camping" Token (10th) has a normalized weight of 0.7 to the "scenery" Token (5th) (strong semantic association), a weight of 0.2 to the "bird call" Token (20th) (weak association), and a weight of 0.01 to the meaningless punctuation Token (24th) (extremely weak association).

[0134] Attention weighting and intra-modal enhanced feature generation: Multiply the normalized attention weights with the V matrix (e.g., text 24×24 × 24×64 = 24×64 per head) to obtain the attention weighted features of each head; then concatenate the 8 head features by dimension (24×(64×8)=24×512), integrate the head information through a linear layer (input 512-dimensional, output 512-dimensional), and finally generate the text intra-modal enhanced features (24×512).

[0135] Similarly, process the image and audio modalities:

[0136] In the image modality, the weight of the "tent" visual point (20th) to the "tree" visual point (30th) is 0.6 (spatial position is adjacent, and semantic correlation is strong), and the weight to the "sky" visual point (5th) is 0.15 (spatial distance is far, and correlation is weak), to generate the intra-image modality enhanced feature (49x512);

[0137] In the audio modality, the weight of the "bird chirping" frame (500th) to the adjacent frames (499th and 501th) is 0.8 (time sequence is continuous, and acoustic features are similar), and the weight to the "wind sound" frame (100th) is 0.05 (feature difference is large), to generate the intra-audio modality enhanced feature (991x512).

[0138] The intra-modal enhanced features are input into the cross-attention layer of the hierarchical fusion network, and the feature interaction between the text modality and the visual modality, and the feature interaction between the text modality and the audio modality are calculated through the cross-attention mechanism to generate the inter-modal interaction features;

[0139] The core of the cross-attention layer is to establish the semantic correlation between different modalities (text as the semantic core, and correlate visual and audio features), and similarly, 8 heads of Scaled Dot-Product attention are used, and the difference from self-attention "same modality Q=K=V" is that cross-attention uses "different modality Q and K / V", and the specific implementation is as follows:

[0140] Q, K, and V of cross-attention are allocated: taking "text-visual" and "text-audio" cross interactions as examples, the allocation rules are as follows:

[0141] Text-visual cross: taking the text intra-modal enhanced feature as Q (24x512, providing a semantic query benchmark), taking the image intra-modal enhanced feature as K (49x512, providing a visual feature index) and V (49x512, providing a visual feature content);

[0142] Text-audio cross: taking the text intra-modal enhanced feature as Q (24x512), taking the audio intra-modal enhanced feature as K (991x512) and V (991x512).

[0143] The allocation is based on the semantic explicitness of the text modality, which can accurately guide the correlation of visual and audio features.

[0144] Text-visual cross-attention calculation:

[0145] Dimension adaptation: Since the sequence length of Q (24x512) and K (49x512) is different, direct calculation of matrix multiplication needs to be adapted through the broadcast mechanism. First, Q, K, and V are split into 8 heads (64 dimensions per head), resulting in text Q heads (24x64), image K heads (49x64), and image V heads (49x64);

[0146] Weight calculation: Each head of Q is multiplied by the transpose of K (24x64 x 64x49 = 24x49). After scaling, it is normalized by softmax. For example, the normalized weight of the text "camping" Token (10th) to the image "tent" visual point (20th) is 0.85 (high semantic matching), and the weight to the "sky" visual point (5th) is 0.1 (low matching degree);

[0147] Weighted fusion: The weight is multiplied by the V head (24x49 x 49x64 = 24x64). After 8 heads are spliced, the linear layer is integrated to generate text-visual cross-interaction features (24x512). Each row of the feature corresponds to a semantic vector of the text Token after integrating visual information (e.g., the "camping" Token vector is integrated into the "tent" visual feature).

[0148] Text-audio cross-attention calculation:

[0149] Split and adaptation: text Q heads (24x64), audio K heads (991x64), and audio V heads (991x64);

[0150] Weight calculation: Q is multiplied by the transpose of K (24x64 x 64x991 = 24x991). After normalization, the weight of the text "bird call" Token (20th) to the audio "bird chirp" frame (500th) is 0.92 (high acoustic feature matching), and the weight to the "wind sound" frame (100th) is 0.03 (low matching degree);

[0151] Weighted fusion: generate text-audio cross-interaction features (24x512), such as the "bird call" Token vector integrated into the "bird chirp" acoustic feature.

[0152] Inter-modal interaction feature integration: The text-visual cross interaction feature (24x512) and the text-audio cross interaction feature (24x512) are spliced by dimension to obtain an inter-modal interaction feature of 24x1024. The feature contains cross-modal associations of text and vision, text and audio, such as the 1024-dimensional vector corresponding to the "camping" Token, the first 512 dimensions are text-tent visual association, and the last 512 dimensions are text-environment acoustic association.

[0153] Multi-scale aggregation of inter-modal interaction features: Pooling operation is used to extract feature representations of different granularities, and multi-scale information is fused by feature splicing to generate multi-scale fusion features.

[0154] The inter-modal interaction feature (24x1024) only reflects the cross-modal association of the text Token level, and needs to extract "local-global" and "fine-grained-coarse-grained" information through multi-scale aggregation, covering sentence-level features of text, region-level features of image, and segment-level features of audio. A three-level pooling strategy (Token / frame level, local block level, and global level) is adopted, and the specific implementation is as follows:

[0155] First level: Token / frame level pooling (fine-grained preservation): The original inter-modal interaction feature (24x1024) is directly preserved, corresponding to each Token level cross-modal association of the text, such as the features of individual Tokens "camping", "scenery", "bird call", etc. This granularity is used to preserve detailed semantics.

[0156] Second level: Local block level pooling (medium-grained integration): The feature sequence is divided into fixed windows, and the features within each window are averaged and pooled to integrate local associations.

[0157] Text inter-modal interaction feature (24x1024): The window size is set to 4 (every 4 Tokens as a local block), a total of 6 blocks (24 / 4=6), and the average feature of each block is calculated to obtain a local block level feature of 6x1024, such as the average feature of the first 4 Tokens ("today", "and", "friends", "go"), reflecting the local semantics of "traveling companions".

[0158] Simultaneously process the intra-modal enhanced features of image and audio (since there are no direct cross interaction features for image and audio, multi-scale information needs to be extracted from enhanced features): The window size of image (49x512) is set to 7 (every 7 visual points as a block), and a local block level feature of 7x512 is obtained. The window size of audio (991x512) is set to 100 (every 100 frames as a segment), and a local block level feature of 10x512 is obtained (991 / 100≈10).

[0159] Third level: global-level pooling (coarse-grained summary): global average pooling is performed on all sequence features to generate a single vector feature, reflecting the global semantics of the entire modality:

[0160] After global averaging of the text modality interaction features (24x1024), a text global feature of 1x1024 is obtained, containing the overall semantics of "camping scene + companions + beautiful scenery + bird calls";

[0161] After global averaging of the image modality internal enhanced features (49x512), an image global feature of 1x512 is obtained, reflecting the visual summary of "outdoor camping environment (tent + trees + sky)";

[0162] After global averaging of the audio modality internal enhanced features (991x512), an audio global feature of 1x512 is obtained, reflecting the acoustic summary of "birdsong as the main sound + a small amount of wind sound".

[0163] Multi-scale feature concatenation fusion: all multi-scale features are concatenated by dimension, and after dimension unification, multi-scale fusion features are generated:

[0164] Text-related feature concatenation: Token level (24x1024) + local block level (6x1024) + global level (1x1024) = 31x1024;

[0165] Image-related feature concatenation: local block level (7x512) + global level (1x512), mapped to 1024 dimensions to obtain 8x1024;

[0166] Audio-related feature concatenation: local block level (10x512) + global level (1x512), mapped to 1024 dimensions to obtain 11x1024;

[0167] Overall concatenation: 31x1024 + 8x1024 + 11x1024 = 50x1024 multi-scale fusion features, covering all cross-modal information from details to global.

[0168] Nonlinear transformation is performed on the multi-scale fusion features, and the activation function is used to enhance the feature expression ability, and the residual connection is used to preserve the original information, and finally the unified fusion features with enhanced semantics are generated.

[0169] Multi-scale fusion features (50x1024) need to be enhanced in expression ability through nonlinear transformation, while avoiding feature information loss through residual connection to ensure that the final features contain enhanced semantics and preserve the original cross-modal correlation. The specific implementation is as follows:

[0170] Nonlinear transformation module construction: adopt the structure of "two fully connected layers + GELU activation function", GELU activation function (formula: GELU (x)=x Φ(x), Φ(x) is the cumulative distribution function of the standard normal distribution) can alleviate the "neuron death" problem and better adapt to the complex distribution of multi-modal features:

[0171] First layer fully connected layer: input dimension 1024, output dimension 2048, dimension expansion for each row (50 row vectors) of multi-scale fusion features, enhance feature capacity, for example, a row vector from [0.18, -0.09,...,0.07] (1024 dimensions) is transformed into [0.25, -0.12, 0.31,..., 0.09] (2048 dimensions);

[0172] GELU activation: nonlinear transformation of 2048-dimensional features, suppresses invalid features (such as dimensions with values close to 0), and enhances valid features (such as the "camping - tent" correlation dimension value from 0.31 to 0.45);

[0173] Second layer fully connected layer: input dimension 2048, output dimension 1024, compress the expanded features back to the original dimension, ensure the uniformity of the subsequent processing dimension, and get 50x1024 nonlinear transformation features.

[0174] Residual connection preserves original information: the core of residual connection is "transformed features + original features", which avoids information loss caused by deep transformation:

[0175] Dimension adaptation: the original multi-scale fusion features are 50x1024, which are consistent with the nonlinear transformation feature dimension, without additional projection;

[0176] Residual addition: add the nonlinear transformation features and the original multi-scale fusion features element by element, for example, the transformed "camping - tent" correlation dimension value 0.45 + the original value 0.38 = 0.83, which not only preserves the original correlation, but also enhances the semantic expression;

[0177] Layer normalization: perform layer normalization (mean tends to 0, variance tends to 1) on the features after residual addition, stabilize the feature distribution, and get 50x1024 residual enhanced features.

[0178] Unified fusion feature generation: global average pooling is performed on the residual enhancement feature, compressing the 50x1024 sequence feature into a 1x1024 single vector. This vector is the semantic enhancement unified fusion feature. For example, the vector contains the comprehensive association of "text semantics (camping scene) + visual features (tent / tree) + audio features (bird chirping)", where the core semantic dimension value of "camping - tent - bird chirping" is 0.92, and the irrelevant dimension (such as "sky - wind") value is only 0.08, achieving high semantic focus.

[0179] S204, input the unified fusion feature into the multi-task prediction head of the multi-modal LLM model, simultaneously perform sentiment analysis, content summary and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

[0180] Specifically, the unified fusion feature can be input into the shared bottom layer of the multi-task prediction head, and feature compression and dimension reduction can be performed through the fully connected layer to generate a task-shared feature vector.

[0181] The unified fusion feature is a 1x1024 dimensional vector output by the early-stage hierarchical fusion network (e.g., in the camping scene, the vector contains the comprehensive association of "text semantics (camping, beautiful scenery) + visual features (tent, tree) + audio features (bird chirping)", and the core semantic dimension value is "camping - tent - bird chirping" with a value of 0.92). The shared bottom layer of the multi-task prediction head needs to remove redundant information through feature compression while preserving cross-modal core semantics to provide general feature support for the three sub-tasks (sentiment analysis, content summary, and event reasoning). The specific implementation is as follows:

[0182] The shared bottom layer adopts the structure of "single hidden layer fully connected network + batch normalization + ReLU activation", and the core parameter design is based on the balance between task demand and computational efficiency:

[0183] The input dimension of the fully connected layer is set to 1024 (consistent with the dimension of the unified fusion feature), and the output dimension is set to 512 (dimension reduction ratio 50%). This not only reduces the computational load of the subsequent sub-network, but also retains more than 90% of the core semantic information through experimental verification. When the output dimension is less than 512, the sentiment analysis accuracy decreases by more than 5%, and when the output dimension is higher than 512, the computational load increases by 30%. 512 is the optimal value.

[0184] The activation function is ReLU (formula: f(x) = max(0, x)), which is used to introduce non-linear representation and suppress invalid negative features (such as "sky - wind" and other weakly associated dimension values close to 0, which become 0 after ReLU, highlighting valid features).

[0185] The batch normalization layer (BatchNorm) is used to stabilize the feature distribution, with a momentum parameter of 0.9 (controls the update speed of the sliding average to avoid distribution fluctuations) and an epsilon parameter of 1e-5 (prevents the denominator from being 0) to ensure that the mean of the output features approaches 0 and the variance approaches 1.

[0186] Taking the unified fusion features of the camping scene as an example, the input vector is [0.92, 0.15, 0.88,..., 0.08] (1024 dimensions, the first three dimensions correspond to "camping - tent association" "sky - wind association" "bird chirping - text association"), after linear transformation by the fully connected layer (weight matrix is 512x1024, pre-training initialization), ReLU activation (weak association dimensions 0.15, 0.08 become 0), and batch normalization, a 512-dimensional task sharing feature vector is generated, with an example of [0.89, 0.85,..., 0.0], where the feature values of the core semantic dimensions "camping - tent" and "bird chirping - text" are preserved and enhanced, and the redundant dimensions are suppressed.

[0187] The task sharing feature vector is input into the sentiment analysis subnetwork, the content summary subnetwork, and the event reasoning subnetwork in parallel, where the sentiment analysis subnetwork outputs sentiment polarity probabilities, the content summary subnetwork outputs summary text, and the event reasoning subnetwork outputs event logical chains, generating initial task outputs.

[0188] The three subnetworks are parallelly calculated based on the task sharing feature vector, each adapting to the output requirements of different tasks, and all using pre-trained model initialization to improve performance, with the following specific implementations:

[0189] 1. Sentiment analysis subnetwork (output sentiment polarity probability)

[0190] The subnetwork structure is "two-layer fully connected + Softmax activation", which is used to judge the sentiment tendency of multi-modal content (divided into "positive", "neutral", and "negative" three categories):

[0191] First layer fully connected: input 512-dimensional task sharing features, output 256-dimensional features, activation function ReLU, used to further extract sentiment-related semantics (such as "beautiful scenery" "super beautiful" corresponding feature dimensions);

[0192] Second layer fully connected: input 256-dimensional features, output 3-dimensional features (corresponding to three categories of sentiment), activation function Softmax (formula: Softmax (x_i)=e^x_i / Σe^x_j), convert the output to probability values, and the sum of the three categories is 1;

[0193] Pre-training initialization: The weights of the fully connected layers are initialized with the parameters pre-trained on a multi-modal sentiment dataset (e.g., CMU-MOSEI), ensuring sensitivity to the sentiment association of "text adjectives + visual colors + audio tones".

[0194] Taking the camping scenario as an example, the subnetwork outputs the sentiment polarity probability as "positive: 0.85, neutral: 0.13, negative: 0.02", where the "positive" probability is the highest, resulting from the strong activation of positive semantic dimensions such as "beautiful scenery" and "bird calls" in the features.

[0195] 2. Content summary subnetwork (output summary text)

[0196] The subnetwork is built based on the pre-trained T5-small model (Encoder-Decoder structure, parameter size 60 million) and is suitable for text generation tasks while integrating multi-modal semantics:

[0197] Input processing: The 512-dimensional task-shared features are concatenated with the token sequence of the text modality (e.g., "Today I went camping with friends in the countryside, the scenery was super beautiful, and I recorded the bird calls!"), serving as the input of the Encoder of the T5 model;

[0198] Decoding generation: The Decoder uses the "autoregressive generation" strategy, with a length limit of 50 characters (to avoid excessive length or brevity, covering core information), and the beam search parameter is set to 3 (beam size=3, balancing generation diversity and accuracy);

[0199] Multi-modal semantic fusion: During the generation process, the model associates the features of the visual "tent" and audio "bird chirping" through attention mechanisms, ensuring that the summary contains cross-modal information rather than relying solely on text.

[0200] The output summary text for the camping scenario is "The user went camping with friends in the countryside, shared the beautiful scenery along the way, and recorded the live bird calls, conveying a positive experience overall", which not only contains the core information of the text but also mentions the visual (scenery) and audio (bird calls) elements, reflecting multi-modal fusion.

[0201] 3. Event reasoning subnetwork (output event logical chain)

[0202] The subnetwork uses the "BiLSTM + Conditional Random Field (CRF)" structure to extract the event causality and temporal relationships in multi-modal content, outputting a structured logical chain:

[0203] BiLSTM layer: input 512-dimensional task-shared features, hidden layer dimension 256, bidirectional calculation to capture the dependencies before and after the event (such as the timing relationship between "camping" and "recording bird calls");

[0204] CRF layer: globally optimal decoding of event label scores (such as "user action", "environmental feature", and "user behavior" labels) output by BiLSTM to determine the label of each event element;

[0205] Logical chain construction: sort the annotated event elements by "timing / causality" to form a logical chain, with each node in the chain containing "event type + event content + associated modality".

[0206] The output event logical chain for the camping scenario is "1. User action (text + vision): go camping with friends in the countryside → 2. Environmental feature (text + vision + audio): beautiful camping site with natural bird calls → 3. User behavior (text + audio): record on-site bird call audio for sharing", with each node explicitly associated with the modality source, reflecting cross-modal event association.

[0207] The final generated initial task output includes: sentiment polarity probability (positive 0.85, neutral 0.13, negative 0.02), summary text (the above camping summary), and event logical chain (the above three-step logical chain).

[0208] Cross-modal consistency check on the initial task output, calculate the semantic consistency between sentiment analysis results and content summary, and the logical consistency between content summary and event reasoning through a consistency loss function, and generate the task output after consistency optimization;

[0209] The initial task output may have cross-modal contradictions (such as sentiment analysis being "positive", but the summary has no positive expressions; or the summary and reasoning logic conflict), which need to be quantified and optimized through a consistency loss function, the specific implementation is as follows:

[0210] 1. Consistency loss function design

[0211] Adopt the "double loss function joint optimization" strategy, respectively for "sentiment - summary" semantic consistency and "summary - reasoning" logical consistency:

[0212] (1) Semantic consistency loss (Sentiment - Summary): Cosine Similarity Loss is used to compare the sentiment probability vector with the semantic vector of the summary text. First, the summary text is converted into a 768-dimensional semantic vector by pre-trained Sentence-BERT (e.g., the semantic vector of the camping summary contains dimensions such as "positive", "camping", "bird call", etc.), then the cosine similarity between the vector and the sentiment probability vector (after normalization) is calculated, loss = 1 - similarity (the higher the similarity, the lower the loss, the stronger the consistency).

[0213] (2) Logical consistency loss (Summary - Reasoning): Cross-Entropy Loss is used to match the event keywords of the summary text (such as "camping", "recording bird calls") with the event nodes of the reasoning logic chain, and calculate the cross-entropy of the matching probability. If the summary keywords have high overlap with the reasoning nodes (e.g., the summary "recording bird calls" matches the reasoning "recording bird call audio"), the loss is low; if the overlap is low (e.g., the summary "cooking" is unrelated to the reasoning "camping"), the loss is high.

[0214] 2. Consistency optimization process

[0215] Taking the initial output of the camping scenario as an example, the optimization steps are as follows:

[0216] Step 1: Calculate the semantic consistency loss. The sentiment probability vector (positive 0.85, neutral 0.13, negative 0.02) is normalized to [0.85, 0.13, 0.02], and the cosine similarity between the summary semantic vector (containing "positive experience" and "beautiful scenery") and the sentiment vector is 0.92, loss = 1-0.92=0.08 (low loss, strong consistency), no need to adjust the sentiment probability.

[0217] Step 2: Calculate the logical consistency loss. The summary keywords "camping", "recording bird calls", and "scenery" completely match the three nodes of the reasoning logic chain, with a matching probability of 0.95, and the cross-entropy loss = -0.95xln (0.95)≈0.048 (low loss, strong consistency), no need to adjust the reasoning logic chain.

[0218] Step 3: If there are inconsistent cases (e.g., sentiment positive 0.8, but no positive expression in the summary), update the shared bottom layer and subnetwork weights of the multi-task prediction head through backpropagation, adjust the sentiment probability (e.g., reduce to 0.6) or modify the summary text (add "super beautiful" and other positive words), until the loss value is lower than the preset threshold (e.g., 0.1).

[0219] The initial output of the camping scenario is consistent, and the optimized task output is consistent with the initial output. If there is inconsistency, the optimized output may be adjusted to "positive sentiment 0.82, with the addition of 'beautiful scenery' in the summary, and the reasoning logic chain remains unchanged."

[0220] Integrate the optimized sentiment analysis results, content summary, and event reasoning results to generate a structured comprehensive analysis report as the final output of the comprehensive analysis results with cross-modal consistency.

[0221] The integration must follow the principles of "structured, readability, and clear cross-modal correlation." The report contains four core modules, each corresponding to the optimized task output, with the addition of "cross-modal consistency explanation" to ensure users clearly understand the correlation logic of multi-modal data. The specific implementation is as follows:

[0222] 1. Report structure design

[0223] The report uses Markdown format for easy viewing and subsequent system integration. The core modules include:

[0224] (1) Multi-modal data source explanation: Clearly state the original sources of text, images, and audio (e.g., "Text: Microblog user published content; Image: User uploaded camping scene photos; Audio: User recorded live bird call audio").

[0225] (2) Sentiment analysis results: Include sentiment polarity, probability value, and associated modal basis (e.g., "Sentiment polarity: positive; positive probability: 0.85; associated modal basis: text 'beautiful scenery' (strong), image 'green vegetation and blue sky' (medium), audio 'clear bird chirping' (strong)").

[0226] (3) Content summary: Output the optimized multi-modal summary with associated modal labels (e.g., "

Text + Vision + Audio

[0227] (4) Event reasoning logic chain: Present in the form of "Step + Event Type + Content + Modal", such as "1. User action (text + vision): Go camping with friends in the countryside → 2. Environmental features (text + vision + audio): Beautiful scenery at the campsite (vision), accompanied by natural bird calls (audio) → 3. User behavior (text + audio): Record live bird call audio for sharing (text)".

[0228] (5) Cross-modal consistency explanation: summarize the consistency check results (e.g., "Sentiment analysis and content summary semantic consistency: high (loss value 0.08); content summary and event reasoning logic consistency: high (loss value 0.048); no cross-modal contradiction, the result is reliable").

[0229] The report integrates the outputs of the three types of tasks, clearly marks the cross-modal correlation basis and consistency results, meets the "cross-modal consistency" and "structured readability" requirements, and can be directly used for subsequent applications (such as social media content review and user preference analysis).

[0230] Another embodiment of the present application provides a multi-modal LLM-based cross-media data fusion analysis system, which is described with reference to Figure 3 , which can include:

[0231] The receiving module 301 is configured to receive multi-modal data containing text data, image data and audio data from different sources, and input the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal correlation;

[0232] The generating module 302 is configured to input the semantic feature representations into a cross-modal alignment module of the multi-modal LLM model, realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism, and generate spatially consistent cross-media embedding representations;

[0233] The interaction module 303 is configured to perform multi-scale feature interaction on the cross-media embedding representations by using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network includes a self-attention layer and a cross-attention layer, to generate semantic-enhanced unified fusion features;

[0234] The execution module 304 is configured to input the unified fusion features into a multi-task prediction head of the multi-modal LLM model, simultaneously perform sentiment analysis, content summary and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

[0235] The above embodiments according to the drawings detail the structure, features and effects of the present application. The above description is only a preferred embodiment of the present application, but the present application is not limited by the drawings shown. Any changes or modifications made in accordance with the concept of the present application, or equivalent embodiments with equivalent changes, are still within the scope of the present application.

Claims

1. A multi-modal LLM-based cross-media data fusion analysis method, characterized in that, The method comprises: receiving multi-modal data containing text data, image data and audio data from different sources, and inputting the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal correlation; inputting the semantic feature representations into a cross-modal alignment module of the multi-modal LLM model to realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism to generate spatially consistent cross-media embedding representations; using a hierarchical fusion network in the multi-modal LLM model to perform multi-scale feature interaction on the cross-media embedding representations, wherein the hierarchical fusion network comprises a self-attention layer and a cross-attention layer to generate semantic-enhanced unified fusion features; inputting the unified fusion features into a multi-task prediction head of the multi-modal LLM model to simultaneously perform sentiment analysis, content summarization and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

2. The method of claim 1, wherein, The receiving multi-modal data containing text data, image data and audio data from different sources, and inputting the multi-modal data into a unified encoder of a multi-modal LLM model for joint encoding to generate semantic feature representations with cross-modal correlation comprises: receiving text data, image data and audio data through a multi-source data interface, removing noise and invalid information using a data cleaning algorithm, and performing size normalization on the image data and sampling rate unification on the audio data to generate preprocessed multi-modal data; inputting the preprocessed multi-modal data into a text encoding branch, an image encoding branch and an audio encoding branch of the unified encoder, extracting word embedding sequences through the text encoding branch, extracting visual feature maps through the image encoding branch, and extracting acoustic feature sequences through the audio encoding branch to generate initial modal features; inputting the initial modal features into a cross-modal interaction layer of the unified encoder to calculate correlation weights of text features and visual features, and correlation weights of text features and audio features through an inter-modal attention mechanism to generate a cross-modal attention graph; based on the cross-modal attention graph, performing weighted fusion on the initial modal features, and performing dimension alignment through a fully connected layer to finally generate semantic feature representations with cross-modal correlation.

3. The method of claim 2, wherein, The inputting the semantic feature representations into a cross-modal alignment module of a multi-modal LLM model to realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism to generate spatially consistent cross-media embedding representations comprises: inputting the semantic feature representations into a dynamic routing initialization layer of the cross-modal alignment module to assign an initial routing path according to the modal type and calculate an initial alignment vector of each modal feature to generate an initial routing configuration; based on the initial routing configuration, calculating a feature similarity matrix between the text modal, visual modal and audio modal through an iterative routing algorithm, and dynamically updating routing weights to generate an optimized routing weight matrix; using the optimized routing weight matrix to transform the semantic feature representations, mapping the text features, visual features and audio features to a shared hidden space through feature projection to generate hidden space feature mapping; The hidden space feature mapping is normalized to ensure consistent distribution of different modal features in the hidden space, and finally a space-consistent cross-media embedding representation is generated.

4. The method of claim 3, wherein, The hierarchical fusion network in the multi-modal LLM model is used to perform multi-scale feature interaction on the cross-media embedding representation, wherein the hierarchical fusion network includes a self-attention layer and a cross-attention layer, and a semantic-enhanced unified fusion feature is generated, including: The self-attention layer of the hierarchical fusion network is input with the cross-media embedding representation, and the feature dependency relationship within each modality is calculated through the self-attention mechanism to enhance the semantic consistency of the features within the modality and generate intra-modal enhanced features; The cross-attention layer of the hierarchical fusion network is input with the intra-modal enhanced features, and the feature interaction between the text modality and the visual modality, as well as the feature interaction between the text modality and the audio modality, is calculated through the cross-attention mechanism to generate inter-modal interaction features; The inter-modal interaction features are aggregated in multiple scales, the features of different granularities are extracted through pooling operation, and the multi-scale information is fused through feature concatenation to generate multi-scale fusion features; The multi-scale fusion features are subjected to non-linear transformation, the feature expression capability is enhanced through the activation function, and the original information is preserved through residual connection to finally generate a semantic-enhanced unified fusion feature.

5. The method of claim 4, wherein, The unified fusion feature is input into the multi-task prediction head of the multi-modal LLM model, and the sentiment analysis, content summary and event reasoning tasks are simultaneously performed to output a comprehensive analysis result with cross-modal consistency, including: The unified fusion feature is input into the shared bottom layer of the multi-task prediction head, and the feature compression and dimension reduction are performed through the fully connected layer to generate a task-shared feature vector; The task-shared feature vector is input into the sentiment analysis subnetwork, the content summary subnetwork and the event reasoning subnetwork in parallel, wherein the sentiment analysis subnetwork outputs the sentiment polarity probability, the content summary subnetwork outputs the summary text, and the event reasoning subnetwork outputs the event logical chain to generate the initial task output; The cross-modal consistency of the initial task output is checked, and the semantic consistency between the sentiment analysis result and the content summary, as well as the logical consistency between the content summary and the event reasoning, is calculated through the consistency loss function to generate the task output after consistency optimization; The sentiment analysis result, the content summary and the event reasoning result after consistency optimization are integrated to generate a structured comprehensive analysis report as the final output of the comprehensive analysis result with cross-modal consistency.

6. A multi-modal LLM-based cross-media data fusion analysis system, characterized in that, The system comprises: A receiving module for receiving multi-modal data containing text data, image data and audio data from different sources and inputting the multi-modal data into the unified encoder of the multi-modal LLM model for joint encoding to generate semantic feature representation with cross-modal association; A generating module for inputting the semantic feature representation into the cross-modal alignment module of the multi-modal LLM model to realize deep alignment of text semantics, visual content and audio features through a dynamic routing mechanism to generate a space-consistent cross-media embedding representation; The interaction module is configured to perform multi-scale feature interaction on the cross-media embedding representation by using a hierarchical fusion network in the multi-modal LLM model, wherein the hierarchical fusion network comprises a self-attention layer and a cross-attention layer, and generate semantic-enhanced unified fusion features; The execution module is configured to input the unified fusion features into a multi-task prediction head of the multi-modal LLM model, and simultaneously perform sentiment analysis, content summarization and event reasoning tasks, and output comprehensive analysis results with cross-modal consistency.

7. The system of claim 6, wherein, The receiving module is specifically configured to: receive text data, image data and audio data through a multi-source data interface, remove noise and invalid information by using a data cleaning algorithm, and perform size normalization on the image data and sampling rate unification on the audio data, to generate preprocessed multi-modal data; input the preprocessed multi-modal data into a text encoding branch, an image encoding branch and an audio encoding branch of a unified encoder respectively, extract word embedding sequences through the text encoding branch, extract visual feature maps through the image encoding branch, and extract acoustic feature sequences through the audio encoding branch, to generate initial modal features; input the initial modal features into a cross-modal interaction layer of the unified encoder, calculate the correlation weights of the text features and the visual features, and the correlation weights of the text features and the audio features through an inter-modal attention mechanism, to generate a cross-modal attention map; based on the cross-modal attention map, perform weighted fusion on the initial modal features, and perform dimension alignment through a fully connected layer, to finally generate semantic feature representations with cross-modal correlation.

8. The system of claim 7, wherein, The generation module is specifically configured to: input the semantic feature representations into a dynamic routing initialization layer of the cross-modal alignment module, assign initial routing paths according to modal types, and calculate initial alignment vectors of the modal features, to generate an initial routing configuration; based on the initial routing configuration, calculate a feature similarity matrix among the text modal, the visual modal and the audio modal through an iterative routing algorithm, and dynamically update routing weights, to generate an optimized routing weight matrix; transform the semantic feature representations by using the optimized routing weight matrix, map the text features, the visual features and the audio features to a shared hidden space through feature projection, to generate a hidden space feature mapping; perform normalization processing on the hidden space feature mapping, to ensure consistent distribution of different modal features in the hidden space, and finally generate space-consistent cross-media embedding representations.

9. A storage medium, characterized by The storage medium has stored therein a computer program, wherein the computer program is configured to execute the method of any one of claims 1-5 when running.

10. An electronic device comprising a memory and a processor, characterized in that, The memory has stored therein a computer program, and the processor is configured to execute the computer program to execute the method of any one of claims 1-5.

Images