A Multimodal Semantic Alignment and Fusion System for Meta-Models

Abstract

This invention relates to the field of large-scale multimodal processing technology, specifically a multimodal semantic alignment and fusion system for a large-scale meta-model. The system includes: acquiring a raw multimodal data set composed of text, images, and audio; implementing an improved algorithm based on canonical correlation analysis, incorporating intermodal nonlinear mapping and local structure preservation constraints; performing cross-modal semantic alignment on the three types of heterogeneous modal data; generating semantically unified multimodal feature representations; inputting the aligned multimodal features into the meta-model fusion coding layer for deep feature fusion; generating a joint multimodal semantic vector with globally correlated semantics; and using this joint semantic vector to complete various downstream tasks and output results. This system can fit complex intermodal relationships, preserve local structural features of the data, reduce differences in semantic space between different modalities, and improve the semantic consistency of multimodal feature fusion.

Description

Technical Field

[0001] This invention relates to the field of large model multimodal processing technology, and in particular to a multimodal semantic alignment and fusion system for large meta-models. Background Technology

[0002] When processing multimodal data such as text, images, and audio, conventional processing methods for large-scale meta-models often extract features from each modality separately, lacking an effective cross-modal semantic association calibration mechanism. Traditional canonical correlation analysis only uses linear mapping, which can only capture simple linear relationships between modalities and lacks the ability to fit nonlinear associations between heterogeneous modalities. Furthermore, the computation process cannot apply local structural constraints, which can easily lead to the loss of internal structural information of the original data.

[0003] Multimodal data inherently exhibits semantic space offsets. Without a dedicated cross-modal semantic alignment mechanism, features from different modalities cannot be mapped to a unified semantic dimension. Directly inputting unaligned multimodal features into a large model for fusion encoding leads to semantic redundancy and feature conflicts, affecting the overall effectiveness of feature fusion. Downstream tasks lack well-regulated joint semantic representations as support, making it difficult to achieve the required processing accuracy for application standards.

[0004] The multimodal application of the metabody large model has strict standards for cross-modal nonlinear correlation fitting, preservation of local structure of original data, unified alignment of multimodal semantic space, and normalized fusion of deep features. Existing linear correlation analysis frameworks do not have the constraint ability of nonlinear mapping and local structure preservation, nor have they established a processing flow of pre-semantic alignment and then fusion encoding, making it difficult to meet the overall usage requirements of unified multimodal semantic representation and stable task inference of the metabody large model. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal semantic alignment and fusion system for large meta-models.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal semantic alignment and fusion system for a large meta-model, comprising:

[0007] The data acquisition module acquires a raw multimodal data set, which includes text modal data, image modal data, and audio modal data.

[0008] The semantic alignment processing module calls an improved canonical correlation analysis algorithm to perform cross-modal semantic alignment processing on the text modal data, the image modal data, and the audio modal data, generating a semantically aligned multimodal feature representation. The improved canonical correlation analysis algorithm is based on the basic principles of canonical correlation analysis and introduces intermodal nonlinear mapping and local structure preservation constraints.

[0009] The feature fusion encoding module inputs the semantically aligned multimodal feature representation into the fusion encoding layer of the meta-model for feature fusion processing to generate a multimodal joint semantic vector;

[0010] The downstream task execution module performs downstream task processing based on the multimodal joint semantic vector and generates task output results.

[0011] As a further aspect of the present invention, the acquisition of the original multimodal data set specifically includes:

[0012] The text modality data is segmented into a sequence of text segments according to sentence boundaries, with each text segment corresponding to a semantic unit;

[0013] The image modal data is divided into a set of image region blocks according to the object detection bounding box, and each image region block corresponds to a visual entity;

[0014] The audio modal data is divided into audio segment sequences according to silent segments, and each audio segment corresponds to a voice event;

[0015] For each text segment in the text segment sequence, perform word segmentation and word embedding encoding to generate a text embedding vector sequence;

[0016] Convolutional feature extraction is performed on each image region block in the set of image region blocks to generate an image embedding vector sequence;

[0017] Acoustic feature extraction is performed on each audio segment in the audio segment sequence to generate an audio embedding vector sequence;

[0018] The text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence are combined to form the original multimodal data set.

[0019] As a further aspect of the present invention, the step of invoking an improved canonical correlation analysis algorithm to perform cross-modal semantic alignment processing on the text modal data, the image modal data, and the audio modal data specifically includes:

[0020] The text embedding vector sequence of the text modal data forms a first modal feature matrix, the image embedding vector sequence of the image modal data forms a second modal feature matrix, and the audio embedding vector sequence of the audio modal data forms a third modal feature matrix.

[0021] Nonlinear kernel mapping is performed on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix respectively to generate the first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix;

[0022] Calculate the first canonical correlation projection direction between the first kernel feature matrix and the second kernel feature matrix, calculate the second canonical correlation projection direction between the second kernel feature matrix and the third kernel feature matrix, and calculate the third canonical correlation projection direction between the first kernel feature matrix and the third kernel feature matrix;

[0023] A three-modal joint projection space is constructed based on the first typical related projection direction, the second typical related projection direction, and the third typical related projection direction;

[0024] The first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix are projected onto the three-modal joint projection space to obtain the first alignment feature, the second alignment feature, and the third alignment feature;

[0025] A local structure preservation regularization term is applied to the first alignment feature, the second alignment feature, and the third alignment feature. The local structure preservation regularization term is used to constrain adjacent samples within the same modality to maintain their proximity relationship after projection.

[0026] The first alignment feature, the second alignment feature, and the third alignment feature, after applying the local structure preservation regularization term, are used as the multimodal feature representation after semantic alignment.

[0027] As a further aspect of the present invention, the semantically aligned multimodal feature representation is input into the fusion coding layer of the meta-model for feature fusion processing, specifically including:

[0028] The text alignment features, image alignment features, and audio alignment features in the semantically aligned multimodal feature representation are respectively input into the three independent modal coding sub-layers of the meta-model. Each modal coding sub-layer outputs a text modal hidden state sequence, an image modal hidden state sequence, and an audio modal hidden state sequence.

[0029] The text modal hidden state sequence, the image modal hidden state sequence, and the audio modal hidden state sequence are concatenated along the sequence dimension to generate a multimodal concatenated sequence;

[0030] The multimodal concatenation sequence is input into the multi-head self-attention sublayer of the fusion coding layer for cross-modal interaction processing to generate an interaction-enhanced multimodal feature sequence.

[0031] The enhanced multimodal feature sequence is input into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate the multimodal joint semantic vector.

[0032] As a further aspect of the present invention, downstream task processing is performed based on the multimodal joint semantic vector to generate task output results, specifically including:

[0033] Parse the type identifier of the downstream task, which includes any one of multimodal classification task, multimodal retrieval task, and multimodal generation task;

[0034] When the downstream task is identified as a multimodal classification task, the multimodal joint semantic vector is input into the classifier network, and the classifier network outputs the category probability distribution as the task output result.

[0035] When the downstream task is identified as a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and the candidate vectors in the preset retrieval library is calculated, and the data corresponding to the candidate vector with the highest similarity is taken as the task output result.

[0036] When the downstream task is identified as a multimodal generation task, the multimodal joint semantic vector is input into the decoder network, and the decoder network generates the target modality data sequence step by step as the task output.

[0037] As a further aspect of the present invention, nonlinear kernel mapping is performed on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix respectively to generate a first kernel feature matrix, a second kernel feature matrix, and a third kernel feature matrix, specifically including:

[0038] Constructing a kernel mapping function based on radial basis kernel functions;

[0039] For each row of eigenvectors in the first modality feature matrix, calculate the radial basis kernel function value between the eigenvector of that row and all row eigenvectors in the first modality feature matrix to form the kernel row vector corresponding to that row;

[0040] Stack the kernel row vectors corresponding to all rows in the first modality feature matrix row by row to generate the first kernel feature matrix;

[0041] The same radial basis kernel function is used to perform the same kernel mapping operation on the second modal feature matrix to generate the second kernel feature matrix;

[0042] The same radial basis kernel function is used to perform the same kernel mapping operation on the third modality feature matrix to generate the third kernel feature matrix.

[0043] As a further aspect of the present invention, a local structure preservation regularization term is applied to the first alignment feature, the second alignment feature, and the third alignment feature, specifically including:

[0044] For the first alignment feature, calculate the sum of squared Euclidean distances between each text sample and its K nearest neighbors in the modality, and use this sum of squared Euclidean distances as the first local structural loss;

[0045] For the second alignment feature, the sum of squared Euclidean distances between each image sample and its K nearest neighbor image samples in the modality is calculated, and this sum of squared Euclidean distances is used as the second local structure loss;

[0046] For the third alignment feature, the sum of squared Euclidean distances between each audio sample and its K nearest neighbors in the modality is calculated, and this sum of squared Euclidean distances is used as the third local structure loss.

[0047] The first local structural loss, the second local structural loss, and the third local structural loss are weighted and summed to generate the total local structural loss value.

[0048] The total local structure loss value is added as a regularization term to the objective function of the improved canonical correlation analysis algorithm to constrain the optimization process of the first alignment feature, the second alignment feature, and the third alignment feature.

[0049] As a further aspect of the present invention, the multimodal concatenation sequence is input into the multi-head self-attention sublayer of the fusion coding layer for cross-modal interaction processing to generate an interaction-enhanced multimodal feature sequence, specifically including:

[0050] The multimodal concatenation sequence is copied into a query matrix, a key matrix, and a value matrix;

[0051] The query matrix, the key matrix, and the value matrix are respectively divided into multiple query sub-matrices, key matrices, and value sub-matrices corresponding to multiple attention heads;

[0052] For each attention head, perform a dot product operation between the query submatrix and the transpose of the key submatrix of that head, and divide the dot product result by a preset scaling factor to obtain the attention score matrix.

[0053] Perform a normalized exponential function operation on the attention score matrix to obtain the attention weight matrix;

[0054] Multiply the value submatrix of the head by the attention weight matrix to obtain the output submatrix of the head;

[0055] The output sub-matrices of all attention heads are concatenated along the channel dimension, and then multiplied with the preset output weight matrix to generate the interactively enhanced multimodal feature sequence.

[0056] As a further aspect of the present invention, the enhanced multimodal feature sequence is input into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate the multimodal joint semantic vector, specifically including:

[0057] The feature vector at each position in the enhanced multimodal feature sequence is sequentially input into the first linear transformation layer of the feedforward neural network sublayer to generate the first linear transformation result.

[0058] The first linear transformation result is input into the activation function layer to generate the activated feature vector. The activation function layer uses Gaussian error linear units.

[0059] The activated feature vector is input into the second linear transformation layer of the feedforward neural network sublayer to generate the second linear transformation result.

[0060] The second linear transformation result is joined with the feature vector at the corresponding position in the interactively enhanced multimodal feature sequence to generate a residual joined feature vector.

[0061] Perform layer normalization on the feature vector after residual connection to generate normalized feature vector;

[0062] The normalized feature vectors at all positions are averaged and pooled along the sequence dimension to obtain a global feature vector, which is then used as the multimodal joint semantic vector.

[0063] As a further aspect of the present invention, when the downstream task is identified as a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and candidate vectors in a preset retrieval database is calculated, and the data corresponding to the candidate vector with the highest similarity is taken as the task output result, specifically including:

[0064] Obtain the multimodal joint semantic vector as the query vector;

[0065] Load a preset search library, which contains multiple candidate vectors and the original multimodal data identifier associated with each candidate vector;

[0066] Calculate the cosine similarity between the query vector and each candidate vector in the retrieval database, and generate a set of similarity scores. When calculating the cosine similarity, the two vectors are multiplied by the product of their respective norms.

[0067] Determine the maximum similarity score and its corresponding candidate vector index from the set of similarity scores;

[0068] Based on the candidate vector index, extract the corresponding original multimodal data identifier from the retrieval database;

[0069] The extracted raw multimodal data, identified by the identifier, is returned as the task output.

[0070] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0071] An improved canonical correlation analysis algorithm incorporates intermodal nonlinear mapping, enabling deep fitting of the correlations between heterogeneous modalities such as text, images, and audio, overcoming the limitations of traditional linear analysis. Local structure preservation constraints are added to constrain feature transformation during cross-modal semantic alignment, maintaining the inherent structural properties of the original multimodal data and preventing information loss during feature mapping. The algorithm achieves unified semantic alignment of multimodal data, correcting deviations in the semantic space distribution of different modalities and generating multimodal feature representations with well-defined semantic dimensions and matching correlations.

[0072] The semantically aligned multimodal feature representations are then fed into the meta-model fusion encoding layer for hierarchical feature fusion and integration. Effective semantic information from each modality is aggregated and reconstructed to generate a multimodal joint semantic vector with global semantic association. Pre-alignment reduces semantic fragmentation and feature redundancy caused by direct fusion of modal features, improving the semantic integrity and representational uniformity of the fused vector.

[0073] Multimodal joint semantic vectors can provide standardized basic representations for various downstream tasks, supporting the stable operation of the task inference process. The entire processing flow forms a complete closed loop from multimodal data input, cross-modal semantic correction, model fusion encoding to downstream task generation, standardizing the processing logic of multimodal data at each stage and adapting to the standardized operation requirements of the meta-model for multimodal semantic alignment, feature fusion, and task inference. Attached Figure Description

[0074] Figure 1 This is a state diagram of a multimodal semantic alignment and fusion system for a large meta-model as described in this invention.

[0075] Figure 2 A flowchart illustrating the process of obtaining the raw multimodal dataset;

[0076] Figure 3 This is a flowchart illustrating the operation of a multimodal semantic alignment and fusion system for a large-scale meta-model. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0078] See Figure 1A multimodal semantic alignment and fusion system for a large-scale meta-model includes a data acquisition module that acquires an original multimodal data set, comprising text modality data, image modality data, and audio modality data; a semantic alignment processing module that uses an improved canonical correlation analysis algorithm to perform cross-modal semantic alignment processing on the text modality data, image modality data, and audio modality data, generating semantically aligned multimodal feature representations. The improved canonical correlation analysis algorithm is based on the basic principles of canonical correlation analysis and incorporates intermodal nonlinear mapping and local structure preservation constraints; a feature fusion encoding module that inputs the semantically aligned multimodal feature representations into the fusion encoding layer of the large-scale meta-model for feature fusion processing, generating a multimodal joint semantic vector; and a downstream task execution module that executes downstream task processing based on the multimodal joint semantic vector, generating task output results.

[0079] In one embodiment of the present invention, the specific implementation method for obtaining the original multimodal data set is as follows, see reference. Figure 2 The text modal data is segmented into a sequence of text segments according to sentence boundaries, with each text segment corresponding to a semantic unit; the image modal data is segmented into a set of image region blocks according to object detection bounding boxes, with each image region block corresponding to a visual entity; the audio modal data is segmented into a sequence of audio segments according to silence segments, with each audio segment corresponding to a speech event; word segmentation and word embedding encoding are performed on each text segment in the text segment sequence to generate a text embedding vector sequence; convolutional feature extraction is performed on each image region block in the image region block set to generate an image embedding vector sequence; acoustic feature extraction is performed on each audio segment in the audio segment sequence to generate an audio embedding vector sequence; the text embedding vector sequence, image embedding vector sequence, and audio embedding vector sequence are combined into the original multimodal data set.

[0080] In a specific implementation, an original multimodal data sample is selected, which includes a text statement "A cat is sitting on a mat", a color image showing a cat and a mat, and a corresponding voice "The cat is on the mat". The text modal data is segmented into a sequence of text fragments according to sentence boundaries, and each text fragment corresponds to a semantic unit. For the above text statement, the sentence boundary is at the end of the statement, and after segmentation, a text fragment "A cat is sitting on a mat" is obtained; the image modal data is segmented into a set of image region blocks according to object detection bounding boxes, and each image region block corresponds to a visual entity. After performing object detection on the above color image, two bounding boxes are obtained, which enclose the cat region and the mat region respectively. After segmentation, a cat image region block and a mat image region block are obtained; the audio modal data is segmented into a sequence of audio fragments according to silence segments, and each audio fragment corresponds to a voice event. After performing silence detection on the above voice, there is no silence segment between "cat" and "on the mat", and the whole pronunciation is a continuous voice event. After segmentation, an audio fragment is obtained.

[0081] In some embodiments, word segmentation and word embedding encoding are performed on each text fragment in the text fragment sequence to generate a sequence of text embedding vectors. For the text fragment "A cat is sitting on a mat", the word segmentation operation is performed to obtain seven word segmentation units: "a", "cat", "is", "sitting", "on", "a", "mat". Each word segmentation unit is mapped to a 128-dimensional word embedding vector through a pre-trained word embedding matrix. The seven word embedding vectors are arranged in the order of word segmentation to generate a sequence of text embedding vectors; convolutional feature extraction is performed on each image region block in the set of image region blocks to generate a sequence of image embedding vectors. For the cat image region block, the region block is scaled to 224×224 pixels and then input into a pre-trained residual network. The 2048-dimensional feature vector output by the last pooling layer of the residual network is taken as the image embedding vector of the region block. The same operation is performed on the mat image region block to obtain another 2048-dimensional image embedding vector. The two image embedding vectors are arranged in descending order of detection confidence to generate a sequence of image embedding vectors; acoustic feature extraction is performed on each audio fragment in the audio fragment sequence to generate a sequence of audio embedding vectors. For the above audio fragment, Mel-frequency cepstral coefficients are extracted with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, and 39-dimensional Mel-frequency cepstral coefficient features are extracted. The mean value of all frame features of the entire audio fragment in the time dimension is taken to obtain a 39-dimensional audio embedding vector, and this vector is used as the only element of the audio fragment sequence to generate a sequence of audio embedding vectors.

[0082] Optionally, when the text segment sequence contains multiple text segments, each text segment independently performs word segmentation and word embedding encoding, and the word embedding vectors generated by all text segments are concatenated into a text embedding vector sequence according to the segment order; when the image region block set contains multiple image region blocks, each image region block independently performs convolutional feature extraction, and the image embedding vectors generated by all image region blocks are assembled into an image embedding vector sequence according to the region block arrangement order; when the audio segment sequence contains multiple audio segments, each audio segment independently performs acoustic feature extraction, and the audio embedding vectors generated by all audio segments are assembled into an audio embedding vector sequence according to the segment time order.

[0083] It can be understood that the text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence are combined into an original multimodal data set. The original multimodal data set contains three elements, namely the text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence. The text embedding vector sequence has a shape of [7, 128], the image embedding vector sequence has a shape of [2, 2048], and the audio embedding vector sequence has a shape of [1, 39].

[0084] In some embodiments, the word embedding encoding uses a learnable embedding matrix to map each word segmentation unit to a continuous vector, the convolutional feature extraction is based on a convolutional neural network pre-trained on an image classification task, and the acoustic feature extraction is based on short-time Fourier transform and a log-Mel filter bank. The text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence are combined without changing their respective data structures, maintaining modally independent feature dimensions.

[0085] Optionally, after the text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence are combined into an original multimodal dataset, a modality identifier is added to the embedding vector sequence of each modality. The modality identifier is used to distinguish the modality type to which the sequence belongs. The length of the embedding vector sequence of each modality in the original multimodal dataset is different, and the dimension of the embedding vector of each modality is different. It can be understood that the adjustable parameters involved in the above segmentation, encoding, and feature extraction operations are all preset with fixed values. The sentence boundary is marked by a period, question mark, or exclamation mark. The object detection bounding box is generated by a region proposal network. The silence segment is determined by a continuous frame sequence with energy below a preset threshold. The word segmentation vocabulary size is 30,000, the dimension of the word embedding vector is 128, the output dimension of the convolutional neural network is 2048, and the dimension of the Mel-frequency cepstral coefficients is 39.

[0086] In practical implementation, let the number of text segments in the original multimodal data sample be T, the number of image region blocks be I, and the number of audio segments be A. The text embedding vector sequence is represented as:

[0087]

[0088] in: The word embedding vector dimension is used to represent the image embedding vector sequence. ,in Given the convolutional feature vector dimension, the audio embedding vector sequence is represented as:

[0089]

[0090] in: The acoustic feature vector dimension is represented by the original multimodal data set as triples. .

[0091] In one embodiment of the present invention, the specific implementation of using an improved canonical correlation analysis algorithm to perform cross-modal semantic alignment processing on text modal data, image modal data, and audio modal data is as follows: (See [link to relevant documentation]). Figure 3 The first modality feature matrix is ​​constructed by using the text embedding vector sequence of text modality data, the second modality feature matrix is ​​constructed by using the image embedding vector sequence of image modality data, and the third modality feature matrix is ​​constructed by using the audio embedding vector sequence of audio modality data. Nonlinear kernel mapping is then performed on the first, second, and third modality feature matrices respectively to generate the first, second, and third kernel feature matrices. The first canonical correlation projection direction between the first and second kernel feature matrices, the second canonical correlation projection direction between the second and third kernel feature matrices, and the third canonical correlation projection direction between the first and third kernel feature matrices are calculated. The projection direction is determined; a three-modal joint projection space is constructed based on the first, second, and third canonical correlation projection directions; the first, second, and third kernel feature matrices are projected onto the three-modal joint projection space to obtain the first, second, and third alignment features; a local structure preservation regularization term is applied to the first, second, and third alignment features, which is used to constrain adjacent samples within the same modality to maintain their proximity relationship after projection; the first, second, and third alignment features after applying the local structure preservation regularization term are used as the multimodal feature representation after semantic alignment.

[0092] The specific implementation of inputting the semantically aligned multimodal feature representation into the fusion coding layer of the meta-model for feature fusion processing is as follows: The text alignment features, image alignment features, and audio alignment features in the semantically aligned multimodal feature representation are respectively input into three independent modality coding sub-layers of the meta-model. Each modality coding sub-layer outputs a text modality hidden state sequence, an image modality hidden state sequence, and an audio modality hidden state sequence. The text modality hidden state sequence, the image modality hidden state sequence, and the audio modality hidden state sequence are concatenated in the sequence dimension to generate a multimodal concatenated sequence. The multimodal concatenated sequence is input into the multi-head self-attention sub-layer of the fusion coding layer for cross-modal interaction processing to generate an interaction-enhanced multimodal feature sequence. The interaction-enhanced multimodal feature sequence is input into the feedforward neural network sub-layer of the fusion coding layer for nonlinear transformation to generate a multimodal joint semantic vector.

[0093] In the specific implementation, an original multimodal data sample containing text embedding vector sequences, image embedding vector sequences, and audio embedding vector sequences is selected. The text embedding vector sequence has a shape of [7, 128], the image embedding vector sequence has a shape of [2, 2048], and the audio embedding vector sequence has a shape of [1, 39]. All embedding vectors have been L2 normalized along the feature dimension, meaning that the Euclidean norm of each vector is 1. The text embedding vector sequences of the text modality data form the first modality feature matrix, which is a 7-row, 128-column matrix. The image embedding vector sequences of the image modality data form the second modality feature matrix, which is a 2-row, 2048-column matrix. The audio embedding vector sequences of the audio modality data form the third modality feature matrix, which is a 1-row, 39-column matrix.

[0094] In some embodiments, nonlinear kernel mapping is performed on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix to generate the first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix, respectively. For the first modality feature matrix, a radial basis function (RBF) is used to map the 128-dimensional original features to a high-dimensional kernel space. The RBF range is (0,1], which is dimensionless. After mapping, the first kernel feature matrix has a shape of 7 rows and 7 columns, where the element in the i-th row and j-th column is the kernel function value between the i-th row vector and the j-th row vector of the first modality feature matrix. For the second modality feature matrix, the same RBF is used for mapping, generating a second kernel feature matrix with a shape of 2 rows and 2 columns. For the third modality feature matrix, the same RBF is used for mapping, generating a third kernel feature matrix with a shape of 1 row and 1 column.

[0095] The first canonical correlation projection direction between the first and second kernel feature matrices is calculated by solving the generalized eigenvalue problem to obtain the first set of projection vectors, with the number of projection directions set to min(7,2)=2. The second canonical correlation projection direction between the second and third kernel feature matrices is then calculated, with the number of projection directions set to min(2,1)=1. Finally, the third canonical correlation projection direction between the first and third kernel feature matrices is calculated, with the number of projection directions set to min(7,1)=1. A three-modal joint projection space is constructed based on the first, second, and third canonical correlation projection directions. The dimension of this three-modal joint projection space is determined by the sum of the number of all projection directions, i.e., 2+1+1=4 dimensions. The canonical correlation projection directions of each modality are stacked column-wise to form a projection matrix, with each column of the projection matrix representing a projection direction. The first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix are projected onto the three-modal joint projection space to obtain the first alignment feature, the second alignment feature, and the third alignment feature. The projection operation is achieved by multiplying the kernel feature matrix with the corresponding projection matrix. The shape of the first alignment feature is 7 rows and 4 columns, the shape of the second alignment feature is 2 rows and 4 columns, and the shape of the third alignment feature is 1 row and 4 columns.

[0096] In specific implementation, the text alignment features, image alignment features, and audio alignment features in the semantically aligned multimodal feature representation are respectively input into three independent modal coding sub-layers of the meta-model. Each modal coding sub-layer outputs a text modal hidden state sequence, an image modal hidden state sequence, and an audio modal hidden state sequence. The text modal coding sub-layer is a Transformer-based encoder, with the input text alignment features having a shape of [7,4], and after positional encoding and multi-head self-attention calculation, the output text modal hidden state sequence has a shape of [7,512]. The image modal coding sub-layer is a Transformer-based encoder, with the input image alignment features having a shape of [2,4], and the output image modal hidden state sequence having a shape of [2,512]. The audio modal coding sub-layer is a Transformer-based encoder, with the input audio alignment features having a shape of [1,4], and the output audio modal hidden state sequence having a shape of [1,512].

[0097] It is understandable that after processing by the multi-head self-attention sub-layer, the feature vector at each position integrates information from all positions in the entire sequence, including cross-modal information. For example, the feature vector at the text position can focus on the features of the image and audio positions, thereby achieving interactive enhancement between the three modalities of text, image, and audio. The multimodal feature sequence with enhanced interaction is input into the feedforward neural network sub-layer of the fusion coding layer for nonlinear transformation to generate a multimodal joint semantic vector. The feedforward neural network sub-layer contains two linear transformation layers and one activation function layer. The input feature dimension is 512, the intermediate layer dimension is 2048, and the output dimension is 512. After processing by the feedforward neural network sub-layer, the feature vector at each position in the multimodal feature sequence with enhanced interaction is independently transformed into a new 512-dimensional feature vector. After all positions are transformed, a feature sequence with shape [10, 512] is formed.

[0098] In a specific implementation, let the dimension of the three-modal joint projection space be . Then the shape of the first alignment feature is The shape of the second alignment feature is The shape of the third alignment feature is ,in , , The objective function of the improved canonical correlation analysis algorithm is: where represents the number of text samples, the number of image samples, and the number of audio samples, respectively.

[0099]

[0100] in: , , These are the first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix, respectively. , , These are projection matrices formed by the first, second, and third canonical correlation projection directions, respectively. This represents the sum of the canonical correlation coefficients between two projected feature matrices. The weighting coefficients of the regularization term are used to preserve the local structure. , , These are the first local structure loss, the second local structure loss, and the third local structure loss, respectively, since the input features have been L2 normalized.

[0101] Optionally, the three independent modality coding sublayers of the meta-model share the same network structure but not the same parameters. The parameters of each modality coding sublayer are optimized and trained only for the data of the corresponding modality. The concatenation order of the multimodal concatenation sequence follows a fixed order of text, image, and audio. It can be understood that the multimodal joint semantic vector, as a global representation fusing information from the three modalities, has a dimension of 512 and can be directly used for subsequent downstream task processing.

[0102] In one embodiment of the present invention, the specific implementation of performing nonlinear kernel mapping on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix to generate the first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix is ​​as follows: a kernel mapping function is constructed based on the radial basis function kernel function; for each row feature vector in the first modality feature matrix, the radial basis function kernel function value between the row feature vector and all row feature vectors in the first modality feature matrix is ​​calculated to form the kernel row vector corresponding to the row; the kernel row vectors corresponding to all rows in the first modality feature matrix are stacked row by row to generate the first kernel feature matrix; the same radial basis function kernel function is used to perform the same kernel mapping operation on the second modality feature matrix to generate the second kernel feature matrix; the same radial basis function kernel function kernel function kernel function kernel function kernel is used to perform the same kernel mapping operation on the third modality feature matrix to generate the third kernel feature matrix.

[0103] The specific implementation of applying local structure preservation regularization terms to the first alignment feature, second alignment feature, and third alignment feature is as follows: For the first alignment feature, the sum of squared Euclidean distances between each text sample and its K nearest neighbors in the modality is calculated, and this sum of squared Euclidean distances is used as the first local structure loss; for the second alignment feature, the sum of squared Euclidean distances between each image sample and its K nearest neighbors in the modality is calculated, and this sum of squared Euclidean distances is used as the second local structure loss; for the third alignment feature, the sum of squared Euclidean distances between each audio sample and its K nearest neighbors in the modality is calculated, and this sum of squared Euclidean distances is used as the third local structure loss; the first local structure loss, the second local structure loss, and the third local structure loss are weighted and summed to generate a total local structure loss value; the total local structure loss value is added as a regularization term to the objective function of the improved canonical correlation analysis algorithm to constrain the optimization process of the first alignment feature, the second alignment feature, and the third alignment feature.

[0104] In the specific implementation, a multimodal data sample containing a first modality feature matrix, a second modality feature matrix, and a third modality feature matrix is ​​selected. The first modality feature matrix is ​​a 7-row, 128-column matrix, the second modality feature matrix is ​​a 2-row, 2048-column matrix, and the third modality feature matrix is ​​a 1-row, 39-column matrix. Nonlinear kernel mapping is performed on the first, second, and third modality feature matrices respectively. A kernel mapping function is constructed based on the radial basis function (RBF) kernel function, the expression of which is:

[0105]

[0106] in: and These are the two rows of eigenvectors in the eigenmatrix. The square of the Euclidean distance between the two eigenvectors. is the bandwidth parameter of the radial basis kernel function.

[0107] In some embodiments, for each row of eigenvectors in the first modal feature matrix, the radial basis function (RBF) kernel value between that row of eigenvectors and all rows of eigenvectors in the first modal feature matrix is ​​calculated to form the corresponding kernel row vector. The first modal feature matrix has 7 rows. The RBF kernel values ​​are calculated for the eigenvectors in the first row and the eigenvectors in rows 1 to 7 respectively, resulting in 7 kernel function values. These kernel function values ​​are then arranged column-wise to form a kernel row vector of length 7. The same operation is performed on the eigenvectors in rows 2 to 7 to obtain 7 kernel row vectors. The kernel row vectors corresponding to all rows in the first modal feature matrix are stacked row-wise to generate the first kernel feature matrix. The first kernel feature matrix has a shape of 7 rows and 7 columns, and the element in the i-th row and j-th column is equal to... ,in and These are the eigenvectors of the i-th and j-th rows of the first modality eigenma matrix, respectively. The same radial basis function (RBF) kernel is applied to the second modality eigenma matrix, generating a second kernel eigenma matrix. This second kernel eigenma matrix has two rows. The RBF kernel value is calculated for each row's eigenvector and all row eigenvectors, resulting in two kernel row vectors. Stacked row by row, the second kernel eigenma matrix has a shape of 2 rows and 2 columns. The same RBF kernel is applied to the third modality eigenma matrix, generating a third kernel eigenma matrix. This third kernel eigenma matrix has one row. The RBF kernel value is calculated for each row's eigenvector and itself, resulting in one kernel row vector. The third kernel eigenma matrix has a shape of 1 row and 1 column, with a unique element of 1.

[0108] Optional, the bandwidth parameter of the radial basis kernel function Set to the median of the Euclidean distance between all sample pairs divided by For the first modality feature matrix, the Euclidean distance between each pair of eigenvectors in the 7 rows is pre-calculated, and the median of all distances is taken as the benchmark value. For the second and third modality feature matrices, their respective bandwidth parameters are calculated independently. The radial basis function kernel function has a range of (0,1]. When two eigenvectors are identical, the kernel function value is 1, and when the distance between two eigenvectors approaches infinity, the kernel function value approaches 0.

[0109] It can be understood that nonlinear kernel mapping maps the original linearly inseparable feature space to a high-dimensional kernel space, making the correlation between modes in the kernel space so that it can be solved using linear canonical correlation analysis. The first, second, and third kernel feature matrices are all symmetric positive semi-definite matrices, with all diagonal elements being 1.

[0110] In the specific implementation, local structure preservation regularization terms are applied to the first alignment feature, second alignment feature, and third alignment feature obtained after projection. The first alignment feature is a 7x4 matrix, the second alignment feature is a 2x4 matrix, and the third alignment feature is a 1x4 matrix. For the first alignment feature, the sum of squared Euclidean distances between each text sample and its K nearest neighbors in the modality is calculated. This sum of squared Euclidean distances is used as the first local structure loss. The value of K is preset to 3. For the first row of samples in the first alignment feature (corresponding to the first text sample), the Euclidean distance between this row vector and the other 6 rows of vectors in the first alignment feature is calculated. The 3 samples with the smallest distance (excluding itself) are selected, and the squared Euclidean distances between the first row vector and these 3 nearest neighbors are summed to obtain the local structure contribution value of the first sample. The same operation is performed on the samples from the second to the seventh row, and the local structure contribution values ​​of all 7 samples are added together to obtain the first local structure loss.

[0111] For the second alignment feature, the sum of squared Euclidean distances between each image sample and its K nearest neighbors in the modality is calculated. This sum of squared Euclidean distances is used as the second local structure loss. The second alignment feature has two rows of samples, and the K value is preset to 1 (because the maximum number of nearest neighbors between two samples is 1). For the first row of image samples, the Euclidean distance between them and the second row of image samples is calculated. The squared distance is used as the local structure contribution value of the first sample. For the second row of image samples, the Euclidean distance between them and the first row of image samples is calculated. The squared distance is used as the local structure contribution value of the second sample. The two contribution values ​​are added together to obtain the second local structure loss.

[0112] For the third alignment feature, the sum of squared Euclidean distances between each audio sample and its K nearest neighbors in the modality is calculated. This sum of squared Euclidean distances is used as the third local structure loss. The third alignment feature has a total of 1 row of samples and there are no other samples in the modality, so the third local structure loss is set to 0.

[0113] In some embodiments, the first local structural loss, the second local structural loss, and the third local structural loss are weighted and summed to generate the total local structural loss value. The weight coefficients of the three modes are denoted as follows: , , Total local structural loss value The calculation formula is:

[0114]

[0115] in: , , These represent the first local structure loss, the second local structure loss, and the third local structure loss, respectively. Text modal weight coefficients. Set to 1, image modality weighting coefficient Set to 1, audio modal weighting coefficient Set it to 0.5.

[0116] Optionally, the weighting coefficients can be adaptively adjusted based on the number of samples in each modality, with modalities having more samples receiving larger weighting coefficients. When the number of samples within a modality is less than K, the actual number of nearest neighbor samples used in the calculation is the total number of samples in that modality minus 1. If the total number of samples in a modality is 1, the local structure loss is directly set to 0. The total local structure loss value is added as a regularization term to the objective function of the improved canonical correlation analysis algorithm to constrain the optimization process of the first alignment feature, the second alignment feature, and the third alignment feature. The overall objective function of the improved canonical correlation analysis algorithm is a combined optimization problem of maximizing the canonical correlation coefficient between modalities and minimizing the total local structure loss value. It can be understood that the local structure preservation regularization term ensures that semantically similar samples within the same modality maintain a close positional relationship after being projected into the joint space, thereby preserving the local geometric structure information within the modality.

[0117] In practice, for the nearest neighbor search of the first alignment feature, Euclidean distance is used as the distance metric. The formula for calculating Euclidean distance is: ,in and These are the two row vectors in the first alignment feature. and These are the m-th components of the two vectors. The specific value of the first local structural loss depends on the distribution of the alignment features after projection, and this loss value changes dynamically with the changes in the alignment features.

[0118] Optionally, in each iteration of optimization, the K nearest neighbors of each sample within the modality are recalculated, and a linear scanning method based on Euclidean distance is used for nearest neighbor search. For modalities with a large number of samples, KD-trees or ball trees are used to accelerate the nearest neighbor search process. It can be understood that the local structure preservation regularization term is jointly optimized with the objective function of canonical correlation analysis, and the strength of local structure preservation is controlled by adjusting the regularization coefficient, which ranges from 0 to 1.

[0119] Table 1: Parameter configuration of the three modes in the local structure preservation regularization term

[0120] Text modality 7 3 1.0 Image modality 2 1 1.0 audio modality 1 1 0.5

[0121] Referring to Table 1, the parameter configurations for the three modalities in the local structure preservation regularization term are listed: the text modality has 7 samples, the nearest neighbor number K is set to 3, and the weight coefficient is 1.0; the image modality has 2 samples, the nearest neighbor number K is set to 1, and the weight coefficient is 1.0; and the audio modality has 1 sample, the nearest neighbor number K is set to 1, and the weight coefficient is 0.5.

[0122] In one embodiment of the present invention, the specific implementation of inputting the multimodal concatenation sequence into the multi-head self-attention sublayer of the fusion coding layer for cross-modal interaction processing to generate the interaction-enhanced multimodal feature sequence is as follows: the multimodal concatenation sequence is copied into a query matrix, a key matrix, and a value matrix; the query matrix, key matrix, and value matrix are respectively divided into query submatrices, key submatrices, and value submatrices corresponding to multiple attention heads; for each attention head, the query submatrices of that head are multiplied by the transpose of the key submatrices, and the dot product result is divided by a preset scaling factor to obtain an attention score matrix; a normalized exponential function operation is performed on the attention score matrix to obtain an attention weight matrix; the value submatrices of that head are multiplied by the attention weight matrix to obtain the output submatrices of that head; the output submatrices of all attention heads are concatenated along the channel dimension, and the concatenated result is multiplied by a preset output weight matrix to generate the interaction-enhanced multimodal feature sequence.

[0123] The specific implementation of inputting the multimodal feature sequence enhanced by interaction into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate the multimodal joint semantic vector is as follows: The feature vector at each position in the multimodal feature sequence enhanced by interaction is sequentially input into the first linear transformation layer of the feedforward neural network sublayer to generate the first linear transformation result; the first linear transformation result is input into the activation function layer, which uses Gaussian error linear units, to generate the activated feature vector; the activated feature vector is input into the second linear transformation layer of the feedforward neural network sublayer to generate the second linear transformation result; the second linear transformation result and the corresponding feature vector in the multimodal feature sequence are subjected to a residual concatenation operation to generate the residual concatenated feature vector; layer normalization is performed on the residual concatenated feature vector to generate the normalized feature vector; the normalized feature vectors at all positions are average pooled along the sequence dimension to obtain the global feature vector, which is used as the multimodal joint semantic vector.

[0124] In the specific implementation, a multimodal concatenation sequence with a shape of [10, 512] is selected as input, where the sequence length is 10 and the feature vector dimension of each position is 512. The multimodal concatenation sequence is copied into a query matrix, a key matrix, and a value matrix, all of which have a shape of [10, 512]. The query matrix, key matrix, and value matrix are respectively divided into query submatrices, key submatrices, and value submatrices corresponding to multiple attention heads. The number of attention heads is set to 8, and the dimension of each attention head is 512 divided by 8, which equals 64. After the query matrix is ​​divided, 8 query submatrices are obtained, each with a shape of [10, 64]. After the key matrix is ​​divided, 8 key submatrices are obtained, each with a shape of [10, 64]. After the value matrix is ​​divided, 8 value submatrices are obtained, each with a shape of [10, 64].

[0125] In some embodiments, for each attention head, the query submatrix and the transpose of the key submatrix are multiplied by a dot product, and the dot product result is divided by a preset scaling factor to obtain the attention score matrix. For the h-th attention head, the query submatrix is ​​denoted as... The key matrix is ​​denoted as The scaling factor is set to ,in The formula for calculating the attention score matrix is: The dot product operation yields a matrix of shape [10, 10]. A normalized exponential function operation is then performed on the attention score matrix to obtain the attention weight matrix. The normalized exponential function is applied row-by-row to the attention score matrix, converting the 10 scores in each row into a probability distribution. The shape is [10, 10], and the sum of the elements in each row is 1. The value submatrix... With attention weight matrix Multiplying them together yields the output submatrix of the h-th attention head. The shape of the output submatrix is ​​[10, 64].

[0126] Optionally, before performing the dot product operation, the query submatrix and the key submatrix are multiplied by a position encoding matrix, which is generated using sine and cosine functions and has dimensions [10, 64]. When performing the normalized exponential function operation, the temperature coefficient is set to 1.0, and no additional scaling parameters are introduced. The output submatrixes of all attention heads are concatenated along the channel dimension to obtain a concatenated matrix with a shape of [10, 512] (8 attention heads, each outputting 64 dimensions, concatenated to 512 dimensions). The concatenated matrix is ​​then multiplied by a preset output weight matrix to generate an interactively enhanced multimodal feature sequence. The preset output weight matrix... The shape is [512, 512]. After multiplication, the output is an enhanced multimodal feature sequence with the shape remaining [10, 512]. It can be understood that the output of the multi-head self-attention sub-layer is a weighted transformation of the input multimodal concatenation sequence. The attention weights reflect the degree of attention each position in the sequence receives from other positions. Since the multimodal concatenation sequence contains positional information from text, image, and audio modalities, the cross-attention values ​​between different modalities in the attention weight matrix enable cross-modal information interaction.

[0127] In the specific implementation, the enhanced multimodal feature sequence is input into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate a multimodal joint semantic vector. The enhanced multimodal feature sequence has a shape of [10, 512] and contains feature vectors at 10 positions. The feature vector at each position in the enhanced multimodal feature sequence is then sequentially input into the first linear transformation layer of the feedforward neural network sublayer to generate the first linear transformation result. The weight matrix of the first linear transformation layer... The shape is [512, 2048], and the bias vector is... The shape is

[2048] , and the feature vector at a single location is... The result of the first linear transformation is The output dimension is 2048.

[0128] The result of the first linear transformation is input into the activation function layer to generate the activated feature vector. The activation function layer uses Gaussian error linear units, and the activation function expression of the Gaussian error linear unit is:

[0129]

[0130] in: The cumulative distribution function of the standard Gaussian distribution. erf is the error function. The activation value is calculated independently for each component of the first linear transformation result, generating a 2048-dimensional feature vector after activation.

[0131] In some embodiments, the activated feature vector is input into the second linear transformation layer of the feedforward neural network sublayer to generate the second linear transformation result, and the weight matrix of the second linear transformation layer is... Shape [2048, 512], bias vector The shape is

[512] , for the activated feature vector The result of the second linear transformation is The output dimension is restored to 512.

[0132] The result of the second linear transformation is joined with the feature vectors at corresponding positions in the interactively enhanced multimodal feature sequence using a residual concatenation operation to generate a residual concatenated feature vector. For the i-th position, the residual concatenated feature vector is... ,in Let be the feature vector at the i-th position in the input interactively enhanced multimodal feature sequence. Let be the eigenvector at the i-th position in the result of the second linear transformation. The shape of the eigenvector after residual connection is [10, 512].

[0133] Layer normalization is performed on the feature vectors after residual concatenation to generate normalized feature vectors. Layer normalization calculates the mean and variance independently for each sample's feature dimension. For a single feature vector... The mean is calculated as The variance is The normalized vector is ,in The constant takes the value of Then multiply by a learnable scaling parameter. And add translation parameters The final normalized feature vector shape is

[512] .

[0134] Table 2: Input / output dimensions and number of trainable parameters for each component of a feedforward neural network sublayer

[0135] First linear transformation layer 512 2048 1,049,088 Gaussian error linear unit activation layer 2048 2048 0 Second linear transformation layer 2048 512 1,049,088 Residual connection layer 512 512 0 Layer normalization layer 512 512 1,024

[0136] Referring to Table 2, the input and output dimensions and the number of trainable parameters of each component of the feedforward neural network sublayer are listed. The first and second linear transformation layers each have 1,049,088 parameters (of which the weight matrix parameters are 512×2048=1,048,576 and the bias vector parameters are 2048). The Gaussian error linear unit activation layer and the residual connection layer do not contain trainable parameters. The layer normalization layer contains 512 scaling parameters and 512 translation parameters, for a total of 1,024 parameters.

[0137] It can be understood that the residual connection operation and layer normalization operation described above are performed independently for each position in the multimodal concatenation sequence. After the feature vectors at the 10 positions undergo the same transformation, 10 normalized feature vectors are obtained. These 10 normalized feature vectors are then averaged along the sequence dimension to obtain the global feature vector. The formula for the average pooling operation is as follows:

[0138]

[0139] in: Let be the normalized feature vector at position i. The global feature vector has the shape

[512] and is used as a multimodal joint semantic vector.

[0140] Optionally, average pooling can be replaced by weighted average pooling, with the weight parameters adaptively learned from the feature sequence through an attention mechanism. The feedforward neural network sublayer is repeatedly stacked 12 times throughout the fusion coding layer, with the dimension of the feature sequence remaining unchanged after each stack. It can be understood that starting from the interactively enhanced multimodal feature sequence of shape [10, 512], after passing through the feedforward neural network sublayer, residual connections, layer normalization, and average pooling, a multimodal joint semantic vector of shape

[512] is finally obtained, which integrates information from all positions in the multimodal concatenation sequence.

[0141] In one embodiment of the present invention, the specific implementation of performing downstream task processing based on the multimodal joint semantic vector to generate task output results is as follows: The type identifier of the downstream task is parsed, which includes any one of a multimodal classification task, a multimodal retrieval task, and a multimodal generation task; when the type identifier of the downstream task is a multimodal classification task, the multimodal joint semantic vector is input into a classifier network, and the classifier network outputs a category probability distribution as the task output result; when the type identifier of the downstream task is a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and candidate vectors in a preset retrieval database is calculated, and the data corresponding to the candidate vector with the highest similarity is used as the task output result; when the type identifier of the downstream task is a multimodal generation task, the multimodal joint semantic vector is input into a decoder network, and the decoder network generates a target modality data sequence step by step as the task output result.

[0142] When the downstream task is identified as a multimodal retrieval task, the specific implementation of calculating the cosine similarity between the multimodal joint semantic vector and candidate vectors in a preset retrieval library, and using the data corresponding to the candidate vector with the highest similarity as the task output, is as follows: Obtain the multimodal joint semantic vector as the query vector; load the preset retrieval library, which contains multiple candidate vectors and the original multimodal data identifier associated with each candidate vector; calculate the cosine similarity between the query vector and each candidate vector in the retrieval library, generating a similarity score set. When calculating the cosine similarity, the dot product of the two vectors is divided by the product of their respective norms; determine the maximum similarity score and its corresponding candidate vector index in the similarity score set; extract the corresponding original multimodal data identifier from the retrieval library based on the candidate vector index; and return the original multimodal data identified by the extracted original multimodal data identifier as the task output.

[0143] In practical implementation, when the downstream task is identified as a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and candidate vectors in the preset retrieval database is calculated. The data corresponding to the candidate vector with the highest similarity is taken as the task output. Specifically, the multimodal joint semantic vector is obtained as the query vector, denoted as [the query vector is not specified in the original text]. Load a preset search library, which contains multiple candidate vectors and the original multimodal data identifier associated with each candidate vector. The number of candidate vectors in the search library is denoted as . , No. The candidate vectors are denoted as The original multimodal data identifier associated with this candidate vector is denoted as Calculate the cosine similarity between the query vector and each candidate vector in the search database, generating a set of similarity scores. The formula for calculating the cosine similarity is:

[0144]

[0145] in: For query vector and the first The dot product of candidate vectors, To find the Euclidean norm of a vector, For the first The Euclidean norm of the candidate vectors and the cosine similarity range from [-1, 1], with a larger value indicating greater similarity between the two vectors.

[0146] Determine the maximum similarity score and its corresponding candidate vector index from the set of similarity scores. Let the maximum similarity score be . The corresponding candidate vector index is Based on candidate vector index Extract the corresponding original multimodal data identifiers from the retrieval database. The extracted raw multimodal data identifier is returned as the task output. The raw multimodal data includes at least one modality data among text modality data, image modality data, and audio modality data.

[0147] It is understandable that the candidate vectors in the retrieval database are pre-calculated using the same feature extraction method as the multimodal joint semantic vectors, and each candidate vector is mapped and stored to its corresponding original multimodal data identifier. When the retrieval database is large, an approximate nearest neighbor search method based on product quantization is used to accelerate the cosine similarity calculation process. Optionally, the cosine similarity calculation can be replaced by the negative value of Euclidean distance or the Pearson correlation coefficient. When using Euclidean distance, the candidate vector with the smallest distance is taken as the retrieval result. The retrieval database supports incremental updates; when new data is added, the corresponding candidate vector is calculated and inserted into the retrieval database.

[0148] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multi-modal semantic alignment and fusion system for meta-body models, the system comprising: The system includes: The data acquisition module acquires a raw multimodal data set, which includes text modal data, image modal data, and audio modal data. The semantic alignment processing module calls an improved canonical correlation analysis algorithm to perform cross-modal semantic alignment processing on the text modal data, the image modal data, and the audio modal data, generating a semantically aligned multimodal feature representation. The improved canonical correlation analysis algorithm is based on the basic principles of canonical correlation analysis and introduces intermodal nonlinear mapping and local structure preservation constraints. The feature fusion encoding module inputs the semantically aligned multimodal feature representation into the fusion encoding layer of the meta-model for feature fusion processing to generate a multimodal joint semantic vector; The downstream task execution module performs downstream task processing based on the multimodal joint semantic vector and generates task output results.

2. The multimodal semantic alignment and fusion system for a large meta-model according to claim 1, characterized in that, The acquisition of the original multimodal data set specifically includes: The text modality data is segmented into a sequence of text segments according to sentence boundaries, with each text segment corresponding to a semantic unit; The image modal data is divided into a set of image region blocks according to the object detection bounding box, and each image region block corresponds to a visual entity; The audio modal data is divided into audio segment sequences according to silent segments, and each audio segment corresponds to a voice event; For each text segment in the text segment sequence, perform word segmentation and word embedding encoding to generate a text embedding vector sequence; Convolutional feature extraction is performed on each image region block in the set of image region blocks to generate an image embedding vector sequence; Acoustic feature extraction is performed on each audio segment in the audio segment sequence to generate an audio embedding vector sequence; The text embedding vector sequence, the image embedding vector sequence, and the audio embedding vector sequence are combined to form the original multimodal data set.

3. The multimodal semantic alignment and fusion system for a large meta-model according to claim 1, characterized in that, The improved canonical correlation analysis algorithm is invoked to perform cross-modal semantic alignment processing on the text modal data, the image modal data, and the audio modal data, specifically including: The text embedding vector sequence of the text modal data forms a first modal feature matrix, the image embedding vector sequence of the image modal data forms a second modal feature matrix, and the audio embedding vector sequence of the audio modal data forms a third modal feature matrix. Nonlinear kernel mapping is performed on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix respectively to generate the first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix; Calculate the first canonical correlation projection direction between the first kernel feature matrix and the second kernel feature matrix, calculate the second canonical correlation projection direction between the second kernel feature matrix and the third kernel feature matrix, and calculate the third canonical correlation projection direction between the first kernel feature matrix and the third kernel feature matrix; A three-modal joint projection space is constructed based on the first typical related projection direction, the second typical related projection direction, and the third typical related projection direction; The first kernel feature matrix, the second kernel feature matrix, and the third kernel feature matrix are projected onto the three-modal joint projection space to obtain the first alignment feature, the second alignment feature, and the third alignment feature; A local structure preservation regularization term is applied to the first alignment feature, the second alignment feature, and the third alignment feature. The local structure preservation regularization term is used to constrain adjacent samples within the same modality to maintain their proximity relationship after projection. The first alignment feature, the second alignment feature, and the third alignment feature, after applying the local structure preservation regularization term, are used as the multimodal feature representation after semantic alignment.

4. The multimodal semantic alignment and fusion system for a large meta-model according to claim 1, characterized in that, The semantically aligned multimodal feature representations are input into the fusion encoding layer of the meta-model for feature fusion processing, specifically including: The text alignment features, image alignment features, and audio alignment features in the semantically aligned multimodal feature representation are respectively input into the three independent modal coding sub-layers of the meta-model. Each modal coding sub-layer outputs a text modal hidden state sequence, an image modal hidden state sequence, and an audio modal hidden state sequence. The text modal hidden state sequence, the image modal hidden state sequence, and the audio modal hidden state sequence are concatenated along the sequence dimension to generate a multimodal concatenated sequence; The multimodal concatenation sequence is input into the multi-head self-attention sublayer of the fusion coding layer for cross-modal interaction processing to generate an interaction-enhanced multimodal feature sequence. The enhanced multimodal feature sequence is input into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate the multimodal joint semantic vector.

5. The multimodal semantic alignment and fusion system for a large meta-model according to claim 1, characterized in that, Downstream task processing is performed based on the multimodal joint semantic vector to generate task output results, specifically including: Parse the type identifier of the downstream task, which includes any one of multimodal classification task, multimodal retrieval task, and multimodal generation task; When the downstream task is identified as a multimodal classification task, the multimodal joint semantic vector is input into the classifier network, and the classifier network outputs the category probability distribution as the task output result. When the downstream task is identified as a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and the candidate vectors in the preset retrieval library is calculated, and the data corresponding to the candidate vector with the highest similarity is taken as the task output result. When the downstream task is identified as a multimodal generation task, the multimodal joint semantic vector is input into the decoder network, and the decoder network generates the target modality data sequence step by step as the task output.

6. The multimodal semantic alignment and fusion system for a large meta-model according to claim 3, characterized in that, Nonlinear kernel mapping is performed on the first modality feature matrix, the second modality feature matrix, and the third modality feature matrix respectively to generate a first kernel feature matrix, a second kernel feature matrix, and a third kernel feature matrix, specifically including: Constructing a kernel mapping function based on radial basis kernel functions; For each row of eigenvectors in the first modality feature matrix, calculate the radial basis kernel function value between the eigenvector of that row and all row eigenvectors in the first modality feature matrix to form the kernel row vector corresponding to that row; Stack the kernel row vectors corresponding to all rows in the first modality feature matrix row by row to generate the first kernel feature matrix; The same radial basis kernel function is used to perform the same kernel mapping operation on the second modal feature matrix to generate the second kernel feature matrix; The same radial basis kernel function is used to perform the same kernel mapping operation on the third modality feature matrix to generate the third kernel feature matrix.

7. The multimodal semantic alignment and fusion system for a large meta-model according to claim 3, characterized in that, Applying local structure preservation regularization terms to the first alignment feature, the second alignment feature, and the third alignment feature specifically includes: For the first alignment feature, calculate the sum of squared Euclidean distances between each text sample and its K nearest neighbors in the modality, and use this sum of squared Euclidean distances as the first local structural loss; For the second alignment feature, the sum of squared Euclidean distances between each image sample and its K nearest neighbor image samples in the modality is calculated, and this sum of squared Euclidean distances is used as the second local structure loss; For the third alignment feature, the sum of squared Euclidean distances between each audio sample and its K nearest neighbors in the modality is calculated, and this sum of squared Euclidean distances is used as the third local structure loss. The first local structural loss, the second local structural loss, and the third local structural loss are weighted and summed to generate the total local structural loss value. The total local structure loss value is added as a regularization term to the objective function of the improved canonical correlation analysis algorithm to constrain the optimization process of the first alignment feature, the second alignment feature, and the third alignment feature.

8. The multimodal semantic alignment and fusion system for a large meta-model according to claim 4, characterized in that, The multimodal concatenation sequence is input into the multi-head self-attention sublayer of the fusion coding layer for cross-modal interaction processing to generate an interaction-enhanced multimodal feature sequence, specifically including: The multimodal concatenation sequence is copied into a query matrix, a key matrix, and a value matrix; The query matrix, the key matrix, and the value matrix are respectively divided into multiple query sub-matrices, key matrices, and value sub-matrices corresponding to multiple attention heads; For each attention head, perform a dot product operation between the query submatrix and the transpose of the key submatrix of that head, and divide the dot product result by a preset scaling factor to obtain the attention score matrix. Perform a normalized exponential function operation on the attention score matrix to obtain the attention weight matrix; Multiply the value submatrix of the head by the attention weight matrix to obtain the output submatrix of the head; The output sub-matrices of all attention heads are concatenated along the channel dimension, and then multiplied with the preset output weight matrix to generate the interactively enhanced multimodal feature sequence.

9. A multimodal semantic alignment and fusion system for a large meta-model according to claim 4, characterized in that, The enhanced multimodal feature sequence is input into the feedforward neural network sublayer of the fusion coding layer for nonlinear transformation to generate the multimodal joint semantic vector, specifically including: The feature vector at each position in the enhanced multimodal feature sequence is sequentially input into the first linear transformation layer of the feedforward neural network sublayer to generate the first linear transformation result. The first linear transformation result is input into the activation function layer to generate the activated feature vector. The activation function layer uses Gaussian error linear units. The activated feature vector is input into the second linear transformation layer of the feedforward neural network sublayer to generate the second linear transformation result. The second linear transformation result is joined with the feature vector at the corresponding position in the interactively enhanced multimodal feature sequence to generate a residual joined feature vector. Perform layer normalization on the feature vector after residual connection to generate normalized feature vector; The normalized feature vectors at all positions are averaged and pooled along the sequence dimension to obtain a global feature vector, which is then used as the multimodal joint semantic vector.

10. A multimodal semantic alignment and fusion system for a large meta-model according to claim 5, characterized in that, When the downstream task is identified as a multimodal retrieval task, the cosine similarity between the multimodal joint semantic vector and candidate vectors in a preset retrieval database is calculated. The data corresponding to the candidate vector with the highest similarity is taken as the task output result, specifically including: Obtain the multimodal joint semantic vector as the query vector; Load a preset search library, which contains multiple candidate vectors and the original multimodal data identifier associated with each candidate vector; Calculate the cosine similarity between the query vector and each candidate vector in the retrieval database, and generate a set of similarity scores. When calculating the cosine similarity, the two vectors are multiplied by the product of their respective norms. Determine the maximum similarity score and its corresponding candidate vector index from the set of similarity scores; Based on the candidate vector index, extract the corresponding original multimodal data identifier from the retrieval database; The extracted raw multimodal data, identified by the identifier, is returned as the task output.

Images