Cross-modal semantic alignment methods, systems, and storage media

Abstract

This application provides a cross-modal semantic alignment method, system, and storage medium. The method extracts multimodal carbon features and performs standardized preprocessing, then uses a projection algorithm to map them to a low-dimensional common semantic space. It combines matrix factorization and association learning models to construct cross-modal semantic associations, and finally generates alignment results through large-scale model fusion inference. This solves the problems of insufficient accuracy and poor semantic coherence in cross-modal semantic alignment of traditional methods.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a cross-modal semantic alignment method, system, and storage medium. Background Technology

[0002] With the rapid growth of multimodal carbon-related data, establishing semantically consistent representations across different modalities has become a core challenge for intelligent systems. Traditional methods often rely on a unified embedding space, but they suffer from insufficient accuracy in maintaining semantic consistency, especially in weakly supervised or unsupervised scenarios. Therefore, there is an urgent need to propose a cross-modal semantic alignment method that integrates multiple techniques to improve semantic alignment accuracy. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a cross-modal semantic alignment method, system and storage medium, which achieves high-precision semantic alignment of different modal data by constructing a cross-modal feature space, performing projection transformation, performing matrix decomposition and introducing an association learning mechanism, thereby improving the performance of multimodal understanding systems.

[0004] In a first aspect, embodiments of this application provide a cross-modal semantic alignment method, including:

[0005] Text features, image features, and sound features are extracted from the input multimodal carbon-related data, and standardized preprocessing is performed to obtain multimodal carbon features represented in a unified format.

[0006] The text features, image features, and sound features are semantically mapped using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space.

[0007] The projection representation is factored using a matrix factorization algorithm to obtain a semantic factor matrix and its coefficient matrix.

[0008] Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and coefficient matrix, and the aligned cross-modal semantic vector is output.

[0009] The cross-modal semantic vectors are fused and inferred using a large model and AI framework to generate the final cross-modal semantic alignment result.

[0010] In another feasible implementation, the step of extracting text features, image features, and sound features from the input multimodal carbon-related data, and performing standardized preprocessing to obtain multimodal carbon features represented in a unified format, includes:

[0011] Text features, image features, and sound features are extracted from the input multimodal carbon-related data using a feature extractor. The text features are then segmented and part-of-speech tagged. Context-sensitive word embedding features are extracted using a pre-trained large model.

[0012] After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain the mapping result.

[0013] Batch normalization is performed on the mapping result to obtain a normalized result, wherein the normalization is a norm that balances the features of each modality;

[0014] The normalization result is input into the encoder to generate multimodal carbon features in a unified format.

[0015] In another feasible implementation, after aligning the word embedding features with the image features, the image features are mapped to the word embedding features using bilinear interpolation to obtain the mapping result, including:

[0016] The image features and word embedding features are mapped one-to-one according to spatial coordinates and semantic tags;

[0017] Based on the correspondence, the coordinates of four adjacent pixels are identified in the image region of the image feature;

[0018] Calculate the horizontal and vertical distance weights between the coordinates of the four pixels and the target word embedding position in the image features;

[0019] The feature values ​​of the four pixels are weighted proportionally by the horizontal and vertical distance weights.

[0020] The weighted results are summed to form the single-point image features after bilinear interpolation;

[0021] The single-point image features are normalized using the L2 norm to obtain the mapping result.

[0022] In another feasible implementation, the step of semantically mapping the text features, image features, and sound features using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space includes:

[0023] A linear transformation is performed on the text features, the image features, and the sound features to generate an initial projection vector;

[0024] The initial projection vector is nonlinearly mapped using the ReLU activation function to obtain the ReLU vector;

[0025] By using batch normalization, the ReLU vector is scaled and translated to obtain the projection vector;

[0026] The global dependencies in the projection vector are extracted using a self-attention mechanism to obtain the projection representation in the low-dimensional common semantic space.

[0027] In another feasible implementation, the step of extracting global dependencies from the projection vector using a self-attention mechanism to obtain a projection representation in a low-dimensional common semantic space includes:

[0028] Extract the query vector, key vector, and value vector from the projection vector;

[0029] The attention weights of the query vector, the key vector, and the value vector are calculated by scaling the dot product.

[0030] The attention weights are subjected to softmax normalization, and the normalized weights are multiplied and accumulated with the query vector, the key vector and the value vector to obtain the projection representation in the low-dimensional common semantic space.

[0031] In another feasible implementation, the step of factoring the projected representation using a matrix factorization algorithm to obtain a semantic factor matrix and its coefficient matrix includes:

[0032] Arrange the projection vectors in the projection representation into a semantic matrix according to time order or spatial position;

[0033] An L2 regularization term is added to the semantic matrix, and singular values ​​and singular vectors are extracted using singular value decomposition.

[0034] Select the singular vectors corresponding to the first P singular values ​​to construct a semantic factor matrix, calculate and extract the weight of each element in the semantic factor matrix, and construct a coefficient matrix based on the weights, where P is an integer greater than 1.

[0035] In another feasible implementation, the step of establishing a cross-modal semantic association model based on the semantic factor matrix and coefficient matrix using the association learning algorithm, and outputting aligned cross-modal semantic vectors, includes:

[0036] The semantic factor matrix and the coefficient matrix are concatenated to construct a cross-modal semantic association model;

[0037] Feature vectors are extracted by using graph neural networks to propagate information across modalities, and the extracted feature vectors are weighted and aggregated to obtain aligned cross-modal semantic vectors.

[0038] In another feasible implementation, the step of concatenating the semantic factor matrix and the coefficient matrix to construct a cross-modal semantic association model includes:

[0039] Construct a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix to nodes;

[0040] The semantic factor matrix and the coefficient matrix are residually joined based on graph convolution operations, and the results obtained by the join are normalized using LayerNorm to form a cross-modal semantic association model.

[0041] Secondly, embodiments of this application provide a cross-modal semantic alignment system, including:

[0042] The multimodal data acquisition module is used to extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features represented in a unified format.

[0043] The projection mapping module is used to perform semantic mapping on the text features, the image features and the sound features using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space.

[0044] The matrix decomposition module is used to perform factorization on the projected representation using a matrix decomposition algorithm to obtain the semantic factor matrix and its coefficient matrix.

[0045] The federated learning module is used to establish a cross-modal semantic association model based on the semantic factor matrix and coefficient matrix using the association learning algorithm, and output the aligned cross-modal semantic vector.

[0046] The alignment module is used to perform fusion reasoning on the cross-modal semantic vectors using a large model and AI framework structure to generate the final cross-modal semantic alignment result.

[0047] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the aforementioned cross-modal semantic alignment method.

[0048] The embodiments of this application bring the following beneficial effects:

[0049] The aforementioned cross-modal semantic alignment method, system, and storage medium extract text, image, and sound features from multimodal carbon-related data and perform standardized preprocessing. These features are then mapped to a low-dimensional common semantic space using a projection algorithm. Semantic factor matrices and coefficient matrices are obtained through matrix factorization. A cross-modal semantic association model is established based on association learning, and finally, a large model is combined for fusion inference to generate alignment results. This application introduces matrix factorization technology to remove the influence of modality-specific noise on core semantics, maintaining semantic consistency even under weakly labeled data conditions. It also introduces a federated learning algorithm to construct the model, enhancing the ability to capture long-distance semantic dependencies and significantly improving alignment accuracy in complex scenarios. Furthermore, it utilizes AI and a large model to provide a clear inference path, demonstrating stronger cross-modal inference capabilities in open-domain question-answering scenarios. This effectively solves the semantic bias problem caused by inconsistent cross-modal feature distributions.

[0050] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0051] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0052] To more clearly illustrate the specific embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 A flowchart of an embodiment of the cross-modal semantic alignment method provided in this application;

[0054] Figure 2 A flowchart illustrating another embodiment of the cross-modal semantic alignment method provided in this application;

[0055] Figure 3 A schematic diagram of a cross-modal semantic alignment system provided in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0059] In related technologies, multimodal carbon-related data processing techniques have evolved from single-modal analysis to cross-modal correlation. Early studies mainly used feature concatenation to integrate information from different modalities, but failed to address the semantic gap problem. Subsequent methods attempted to construct a unified embedding space; however, under weak supervision, it is difficult to maintain semantic consistency between modalities. For example, in video content analysis scenarios, the temporal misalignment between speech descriptions and on-screen actions leads to semantic matching deviations, and traditional projection methods cannot effectively eliminate cross-modal feature distribution differences.

[0060] To address the aforementioned issues, the inventors identified three key shortcomings: the heterogeneity of multimodal carbon features makes alignment difficult; shallow mappings cannot capture deep semantic relationships; and static models struggle to adapt to dynamic data distributions. Analysis revealed that different modalities share common factors at the latent semantic level that can be aligned. Decoupling semantic elements in a low-dimensional space and establishing dynamic relationships can improve cross-modal alignment accuracy. Based on this, a phased processing framework is proposed: first, a unified feature representation is achieved through nonlinear mapping; then, semantic factors are decomposed to construct a dynamic relationship model; and finally, the reasoning capabilities of a large model are utilized to achieve end-to-end alignment.

[0061] For ease of understanding, the specific process of the embodiments of this application is described below. The embodiments of this application are applied to system devices such as computers and servers. Please refer to [link / reference]. Figure 1 This application provides an embodiment of a cross-modal semantic alignment method. This embodiment mainly achieves unified alignment of cross-modal semantic information by combining projection algorithms, matrix factorization algorithms, and association learning algorithms. Specifically, it includes the following steps:

[0062] S110. Extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features represented in a unified format.

[0063] In this step, the system device first receives multimodal input data collected from the user or external device, which includes at least natural language text, images (such as photos and icons) and sound signals (such as speech and ambient sound).

[0064] Text features are segmented and part-of-speech tagging is performed by the natural language processing module, and context-sensitive word embeddings are generated using pre-trained large language models (such as BERT and GPT). Image features are extracted using convolutional neural networks (CNNs). Audio features are extracted using short-time Fourier transform to obtain Mel-frequency cepstral coefficients (MFCCs) or high-dimensional semantic features are extracted using pre-trained audio models. All modal features undergo uniform dimensionality normalization, including zero-mean unit variance standardization and modulus normalization, to ensure that each modality feature is expressed on a uniform scale, providing a foundation for subsequent fusion.

[0065] It should be noted that standardization preprocessing refers to the process of converting features of different modalities into a uniform numerical distribution. Specifically, it can be achieved by combining batch normalization with feature scaling to eliminate the dimensional differences of features from different modalities.

[0066] Projection algorithms are methods that map high-dimensional features to a low-dimensional space. Specifically, they can be implemented using linear transformations combined with nonlinear activation functions to capture potential semantic associations across modalities.

[0067] Matrix factorization algorithms are techniques for decomposing complex matrices into low-rank matrix products. Specifically, they can be implemented using singular value decomposition combined with regularization constraints to extract semantic factors shared across modalities.

[0068] Association learning algorithms refer to methods for establishing semantic associations between modalities. Specifically, they can be implemented using graph neural networks combined with attention mechanisms to dynamically model the interaction relationships between semantic factors.

[0069] Specifically, this step extracts deep features of each modality through pre-trained models, such as using BERT to extract text context features, ResNet to extract image spatial features, and Mel spectrum analysis to extract sound time-frequency features.

[0070] S120. Using a projection algorithm, semantic mapping is performed on text features, image features, and sound features to obtain a projected representation in a low-dimensional common semantic space.

[0071] In this embodiment, each modality feature first undergoes a linear transformation module to project it onto a specified dimensional space, and then is input into a ReLU nonlinear activation function module to enhance the model's expressive power. Next, a batch normalization module is used to scale and translate the activated vectors to accelerate convergence and stabilize training. Finally, a multi-head self-attention mechanism is used to capture the global semantic dependencies between different modalities, outputting a semantically aligned low-dimensional common representation vector, i.e., the projected representation.

[0072] S130. Use matrix decomposition algorithm to factorize the projected representation to obtain the semantic factor matrix and its coefficient matrix.

[0073] In this step, the projected representations from the previous step are arranged into a semantic matrix in temporal or spatial order, and an L2 regularization term is added to constrain the weights. Then, singular value decomposition (SVD) is used to decompose the semantic matrix, yielding a set of singular values ​​and their corresponding singular vectors. The first P principal components are selected to construct a semantic factor matrix, and a coefficient matrix is ​​generated based on the projection of the original matrix into this subspace. This factor matrix represents the structure of the shared semantic space, and the coefficient matrix describes the contribution of the original features to this space.

[0074] S140. Based on the association learning algorithm, establish a cross-modal semantic association model based on the semantic factor matrix and coefficient matrix, and output the aligned cross-modal semantic vector;

[0075] Specifically, the semantic factor matrix and coefficient matrix are concatenated and used as input to construct a graph neural network model. Nodes are constructed based on semantic units and their combination relationships. During graph convolution propagation, each node can fuse semantic representations from different modalities, and cross-modal semantic relationships are extracted through multi-layered stacked GNN operations. Finally, a multi-head attention mechanism is used to weight the semantic importance of nodes, outputting an aligned cross-modal semantic vector.

[0076] S150. Utilize large models and AI frameworks to perform fusion reasoning on cross-modal semantic vectors, generating the final cross-modal semantic alignment result.

[0077] Specifically, the system device loads a multimodal fusion model on the deployed AI platform using an integrated inference engine (such as TensorRT or ONNXRuntime), and inputs the aligned semantic vector output in step S40 into the downstream classifier or semantic matching module to obtain the alignment result output for the corresponding task, such as text-image matching, speech-image retrieval, etc.

[0078] In this embodiment, the method first extracts deep features from each modality using a pre-trained model. For example, BERT is used to extract text context features, ResNet to extract image spatial features, and Mel-spectrum analysis to extract sound time-frequency features. Then, a bilinear projection method is used to map the heterogeneous features to a unified dimension, introducing a nonlinear activation function to enhance the model's expressive power. The projected features are then subjected to matrix decomposition to obtain a factor matrix representing the core semantics and a coefficient matrix reflecting instance characteristics. The product of these two matrices reconstructs the original projected features. Based on the decomposition results, a graph structure model is constructed, where nodes represent semantic factors, and edge weights are determined by the coefficient matrix. The node states are iteratively updated using a graph propagation algorithm. Finally, the optimized semantic vectors are input into a multimodal large model, leveraging its powerful reasoning capabilities to comprehensively determine semantic alignment relationships.

[0079] Compared with existing technologies, the solution provided in this application mainly introduces matrix factorization and graph neural network techniques. Traditional projection methods directly map features to a unified space, which easily leads to the loss of fine-grained semantic information. In contrast, this application retains core semantic elements by decomposing semantic factors. Existing graph models typically construct nodes based on original features, while this application utilizes the decomposed semantic factors to construct a more compact graph structure, significantly reducing computational complexity. Existing large-scale model applications often use feature concatenation as input. This application provides structured semantic representation, enabling large-scale models to perform cross-modal reasoning more effectively.

[0080] Through the above technical solutions, this application effectively solves the semantic bias problem caused by inconsistent cross-modal feature distributions, enabling precise alignment of speech descriptions and on-screen actions in video semantic understanding tasks. Matrix factorization technology removes the influence of modality-specific noise on core semantics, maintaining semantic consistency even under weakly labeled data conditions. The combination of graph neural networks and attention mechanisms enhances the model's ability to capture long-distance semantic dependencies, significantly improving alignment accuracy when handling complex scenes. Structured semantic representation provides a clear reasoning path for large models, demonstrating stronger cross-modal reasoning capabilities in open-domain question-answering scenarios.

[0081] like Figure 2 The illustration shows another implementation of the cross-modal semantic alignment method provided in this application, which specifically includes the following steps:

[0082] S210. Text features, image features, and sound features are extracted from the input multimodal carbon-related data through a feature extractor. The text corresponding to the text features is segmented and part-of-speech tagged. Context-sensitive word embedding features are extracted through a pre-trained large model.

[0083] Specifically, by acquiring data samples from different modalities, the original features of each data sample are extracted using a modality-specific encoder. The original features are then normalized using a feature regularization method. Finally, the normalized original features are subjected to preliminary semantic mapping using shared semantic annotations to obtain the features of each modality, namely, extracting text features, image features, and sound features.

[0084] Understandably, the first step is to extract text, image, and sound features from the input multimodal carbon-related data using a feature extractor; then, the text input is segmented and part-of-speech tagging is performed, and language models such as BERT are used to extract context-related word embedding features.

[0085] The image feature refers to the fixed-dimensional multi-channel image feature map extracted using a ResNet network. The audio feature is extracted using an audio convolutional network or a Wav2Vec model to obtain time-series audio semantic features.

[0086] Word segmentation refers to the process of dividing continuous text into independent semantic units. Specifically, it can be achieved using rule-based or statistical model-based word segmentation tools to eliminate redundant information in the text and extract basic semantic units.

[0087] Part-of-speech tagging refers to labeling the grammatical category of each word after word segmentation. It can be implemented using Hidden Markov Models or Conditional Random Fields algorithms to help identify the functional role of words in context.

[0088] Pre-trained large models refer to language models pre-trained with massive amounts of data, such as those using BERT or GPT structures, used to extract word vector representations that include contextual associations.

[0089] In practical applications, regularization processing is performed on the input text data, including unifying the encoding format, removing stop words, and filtering symbols; bidirectional LSTM or Transformer structures are used for part-of-speech tagging to improve the accuracy of grammatical structure analysis; large-scale pre-trained models such as BERT or ERNIE are used to extract contextualized embeddings; and each word embedding and its corresponding part-of-speech tag are combined as features for subsequent processing.

[0090] S220. After aligning the word embedding features with the image features, the image features are mapped to the word embedding features using bilinear interpolation to obtain the mapping result.

[0091] Understandably, the image features and word embedding features are mapped one-to-one according to spatial coordinates and semantic tags; based on the mapping relationship, the coordinates of four adjacent pixels are identified in the image region of the image features; the horizontal and vertical distance weights between the coordinates of the four pixels and the target word embedding position in the image features are calculated; the feature values ​​of the four pixels are weighted proportionally by the horizontal and vertical distance weights; the weighted results are accumulated to form a single-point image feature after bilinear interpolation; the single-point image feature is normalized by L2 norm to obtain the mapping result.

[0092] It should be noted that the one-to-one correspondence between spatial coordinates and semantic tags refers to establishing a mapping relationship between the spatial distribution of image features and the position of semantic tags in the text. Specifically, this can be achieved by using a coordinate encoder to convert the image grid coordinates into vectors with the same dimension as the word embedding, thus ensuring the spatial alignment between local regions of the image and semantic units of the text. The bilinear interpolation method is an interpolation algorithm that uses weighted feature fusion based on the coordinate distance weights of adjacent pixels. Specifically, it can be achieved by calculating the horizontal and vertical scaling factors using linear interpolation formulas, and then performing a bilinear weighted summation on four adjacent points, thereby achieving cross-modal feature mapping while preserving the spatial continuity of the image. L2 norm normalization refers to standardizing the feature vector by unit length. Specifically, it can be achieved by using the square root of the sum of the squares of each element of the vector as the denominator for scaling operations, thereby eliminating differences in numerical scale between different modal features.

[0093] Specifically, in the alignment process between image features and word embedding features, the image grid coordinates are first converted into vectors matching the word embedding dimensions using a coordinate encoder, establishing a spatial-semantic correspondence. After the target word embedding location is determined, the four nearest pixels in the image features are located based on its coordinates, and the distance weights between this location and the four pixels in the horizontal and vertical directions are calculated respectively. For example, when the horizontal distance from the target location to the top-left pixel is 0.3 and the vertical distance is 0.4, the weight coefficient of this pixel is (1-0.3)*(1-0.4)=0.42. The feature values ​​of the four pixels are weighted and summed according to their respective weights to form the interpolated image features matching the target word embedding location. Finally, L2 normalization is used to constrain the vector length of the interpolation result to a unit range, ensuring the numerical stability of subsequent cross-modal fusion.

[0094] When implementing alignment and embedding, a Cross-Modal Attention mechanism can be used to calculate the alignment relevance score between word embeddings and image regions; the image region with the highest relevance can be selected as the corresponding region; region average pooling (ROI Pooling) can be performed on the features of the image region to obtain low-resolution image features; and bilinear interpolation can be used to map the image feature space to the word embedding space dimension.

[0095] S230. Perform batch normalization on the mapping results to obtain the normalized results;

[0096] It should be noted that the normalization mentioned refers to balancing the norms of each modality feature; the batch normalization refers to standardizing the feature data by mean and variance, specifically by using the moving average method to calculate statistics, which is used to eliminate the dimensional differences between different modal features. An encoder refers to a neural network structure with feature transformation capabilities, such as using a Transformer encoding layer, to convert heterogeneous features into vector representations of a uniform dimension.

[0097] Specifically, the mapping results are normalized by channel-wise mean and standard deviation to maintain consistent mean across modalities; BatchNorm or LayerNorm is used for normalization during the training phase; the normalization coefficients for each modality are automatically adjusted based on modal differences to reduce the impact of modal imbalance; and the normalized feature output serves as the feature baseline before multimodal fusion.

[0098] S240. Input the normalization result into the encoder to generate multimodal carbon features in a unified format.

[0099] Specifically, the three types of modal inputs are uniformly input into the Transformer or Multi-modal Encoder network; positional encoding is used to enhance the structural representation between sequences; intermodal contextual interaction is achieved through a multi-layer attention fusion structure; and a unified format of multimodal carbon feature representation is output for subsequent projection.

[0100] In this embodiment, after obtaining the multimodal carbon features, the method further includes: adding Gaussian white noise that conforms to a standard normal distribution to the multimodal carbon features; setting the noise perturbation amplitude as a dynamic learning parameter; improving the semantic invariance and robustness of the features by comparing the learning training process; and using the enhanced features to simulate semantic stability testing under weak perturbation.

[0101] Specifically, text data is segmented and labeled with parts of speech to form a structured sequence. A pre-trained large model generates context-sensitive word embedding vectors. Image features are extracted using a convolutional neural network and spatially aligned with the word embedding features to locate the regions corresponding to semantic tags in the image feature map. When mapping image features to the word embedding space using bilinear interpolation, the coordinates of the four adjacent pixels of the target location are first determined, and the horizontal and vertical distance weights between these pixels and the target point are calculated. The feature values ​​of the four points are then weighted proportionally and summed to obtain the interpolation result. Normalization is performed by calculating the mean and standard deviation of the feature matrix, scaling and shifting each feature dimension to make the distribution range of different modal features more consistent. The encoder receives the normalized multimodal carbon features and generates a vector representation with unified dimensions through multi-layer nonlinear transformations.

[0102] S250. Using a projection algorithm, semantic mapping is performed on text features, image features, and sound features to obtain a projected representation in a low-dimensional common semantic space.

[0103] In this step, an initial projection vector is generated by performing linear transformations on the text features, image features, and sound features; a ReLU vector is obtained by performing nonlinear mapping on the initial projection vector using the ReLU activation function; the ReLU vector is scaled and translated using batch normalization to obtain the projection vector; and the global dependencies in the projection vector are extracted using a self-attention mechanism to obtain the projection representation in the low-dimensional common semantic space.

[0104] Linear transformation refers to mapping high-dimensional features to a low-dimensional space through matrix multiplication, which can be implemented using fully connected layers. Its function is to eliminate the dimensionality differences between features from different modalities. The ReLU activation function performs a non-linear truncation operation on the input values, which can be implemented using piecewise functions. Its function is to enhance the non-linear expressive power of the model. Batch normalization refers to standardizing the feature distribution, which can be implemented using mean and variance scaling. Its function is to alleviate the distribution shift problem of features from different modalities. The self-attention mechanism calculates the dependencies between features through dynamic weights, which can be implemented using a multi-head attention module. Its function is to capture the global semantic associations between cross-modal features.

[0105] Specifically, after linear transformation, the text, image, and audio features are fed into a ReLU activation function layer to eliminate negative values ​​and retain valid features. Subsequently, a batch normalization layer independently standardizes each dimension of the ReLU vector, ensuring its mean is zero and its variance is one, thus balancing the numerical ranges of features from different modalities. The normalized projection vectors are then fed into a self-attention module, which dynamically aggregates global semantic information by calculating similarity weights between the query vector, key vector, and value vector, ultimately forming a consistent projection representation in a low-dimensional common semantic space.

[0106] In another embodiment, a self-attention mechanism is used to extract global dependencies from the projection vector to obtain a projection representation in a low-dimensional common semantic space. This includes: extracting the query vector, key vector, and value vector from the projection vector; calculating the attention weights of the query vector, key vector, and value vector by scaling the dot product; performing softmax normalization on the attention weights; and multiplying and accumulating the normalized weights with the query vector, key vector, and value vector to obtain the projection representation in the low-dimensional common semantic space.

[0107] Understandably, a query vector is a vector used to represent the current feature location that needs to be focused on. Specifically, it can be implemented by mapping the projection vector to the query vector through a linear transformation. Its function is to determine the relevance weights between different feature locations.

[0108] A key vector is a vector used to match a query vector. Specifically, it can be achieved by mapping a projection vector to a key vector through a linear transformation. Its function is to generate a comparable benchmark value for each feature position.

[0109] A value vector is a vector carrying specific feature information. This can be achieved by mapping a projection vector to a value vector through a linear transformation. Its purpose is to provide aggregateable semantic content for the attention mechanism. A scaled dot product is calculated by dividing the dot product of the query vector and the key vector by the square root of the vector's dimension. This can be achieved through numerical computation and serves to prevent the dot product from becoming too large, which could lead to gradient vanishing.

[0110] Softmax normalization refers to converting attention weights into a probability distribution, which can be achieved through exponential operations and normalization calculations. Its purpose is to ensure that the sum of attention weights is 1 and non-negative.

[0111] Specifically, in the low-dimensional common semantic space, the projection vector undergoes a linear transformation to generate three sets of vectors: query, key, and value. Initial attention weights are obtained by calculating the dot product of the query and key vectors and scaling them. After softmax normalization, the weight matrix and value vectors are weighted and summed to form a projection representation that reflects global dependencies. This process effectively captures long-distance semantic dependencies in cross-modal data by dynamically adjusting the association strength between different feature locations, thus solving the problem of semantic information loss caused by local feature aggregation in traditional methods.

[0112] S260. The projection representation is factorized using a matrix factorization algorithm to obtain the semantic factor matrix and its coefficient matrix.

[0113] Understandably, the projection vectors in the projection representation are arranged into a semantic matrix according to time order or spatial position; an L2 regularization term is added to the semantic matrix, and singular values ​​and singular vectors are extracted using the singular value decomposition method; the singular vectors corresponding to the first P singular values ​​are selected to construct a semantic factor matrix, and the weights of each element in the extracted semantic factor matrix are calculated, and a coefficient matrix is ​​constructed based on the weights, where P is an integer greater than 1.

[0114] The semantic matrix refers to a two-dimensional matrix formed by arranging projection vectors at different time points or spatial locations according to rules. Specifically, it can be implemented by time series stacking or spatial grid mapping, and is used to uniformly represent the distribution structure of multimodal carbon features.

[0115] Specifically, let the projection be represented by matrix Z∈R n×d , where n represents the number of samples or the number of time / space nodes, and d represents the projection dimension.

[0116] Consider Z as containing n projection vectors z1, z2, ..., z n Arranged in chronological or spatial order to form a semantic matrix M∈R n×d .

[0117] To improve robustness, L2 regularization terms are introduced to reduce MM:

[0118] M′=M+λI,

[0119] Where I is the identity matrix, and λ∈R + This is the hyperparameter for regularization intensity.

[0120] The L2 regularization term refers to the constraint term that adds the sum of squares of matrix elements to the objective function. Specifically, the regularization strength can be controlled by the weight decay coefficient to prevent overfitting during matrix decomposition.

[0121] Singular value decomposition refers to a mathematical method that decomposes a matrix into a product of three matrices. Specifically, truncated singular value decomposition can be used to retain the main components and extract latent semantic factors from the semantic matrix.

[0122] The first P singular values ​​refer to the P largest singular values ​​retained after being sorted in descending order. The specific P value can be determined based on the cumulative contribution rate of the feature values ​​or a preset threshold, and is used to construct a low-dimensional semantic factor matrix to capture core semantic information.

[0123] Specifically, singular value decomposition (SVD) is performed on the regularized semantic matrix M′:

[0124] M′=UΣV T ,

[0125] in:

[0126] U∈R n×n U is a left singular vector matrix;

[0127] Σ∈R n×d It is a diagonal singular value matrix;

[0128] V∈R d×d It is a right singular vector matrix.

[0129] Keep the first P largest singular values:

[0130] Σ P =diag(σ1,...,σ P );

[0131] The corresponding left singular vectors form the matrix U. P ∈R n×P ;

[0132] The matrix V is formed by the corresponding right singular vectors. P ∈R d×P，

[0133] For constructing the semantic factor matrix:

[0134] F=V P ∈R d×P ,

[0135] This represents the linear interpretation direction of each projective feature across P factors.

[0136] For calculating the weight coefficient matrix:

[0137] C=MF∈R n×P ,

[0138] That is, the projection of the original semantic matrix onto the semantic factor matrix reflects the semantic contribution of each sample to each factor.

[0139] Normalization: To improve the stability of subsequent graph learning, column normalization is performed on FF and CC respectively:

[0140]

[0141] Final output: Normalized semantic factor matrix With coefficient matrix It can be input into a graph neural network for association learning.

[0142] Specifically, the projection vectors are first arranged into a semantic matrix according to temporal order or spatial location; for example, the projection vectors of video frames are arranged into row vectors by timestamps. Then, an L2 regularization term is added during the decomposition process to constrain the magnitude of the matrix elements; for example, a Frobenius norm penalty term is added to the objective function. The semantic matrix is ​​then decomposed using singular value decomposition (SVD) to obtain a diagonal matrix containing singular values ​​and left and right singular vector matrices. The left singular vectors corresponding to the first P singular values ​​are selected to construct a semantic factor matrix; for example, when P is 100, the first 100 principal components are retained. Finally, the weight coefficients of each semantic factor in the original matrix are calculated using the least squares method to form a coefficient matrix; for example, gradient descent is used to iteratively optimize the weight parameters.

[0143] S270. Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and coefficient matrix, and the aligned cross-modal semantic vector is output.

[0144] In this step, the semantic factor matrix and the coefficient matrix are concatenated to construct a cross-modal semantic association model; feature vectors are extracted by using a graph neural network to propagate information across modalities, and the extracted feature vectors are weighted and aggregated to obtain aligned cross-modal semantic vectors.

[0145] Understandably, the semantic factor matrix refers to the set of basic features reflecting cross-modal semantic commonalities obtained through matrix factorization. Specifically, it can be implemented by extracting the top P principal components using singular value decomposition, used to characterize the potential semantic associations between different modalities. The coefficient matrix refers to the set of parameters reflecting the weight distribution of each modality's features in the semantic factor matrix. Specifically, it can be implemented by calculating the weight coefficients using linear regression or least squares methods, used to quantify the contribution of different modalities to the common semantic space. Concatenation refers to the operation of connecting the semantic factor matrix and the coefficient matrix along a specific dimension. Specifically, it can be implemented using fully connected layers or tensor concatenation operations, used to construct the basic data structure for cross-modal interaction. Graph neural networks refer to deep learning models that transmit information based on graph structures. Specifically, they can be implemented using graph convolutional networks or graph attention networks, used to capture the nonlinear relationships between cross-modal nodes. Cross-modal information propagation refers to the process of transmitting feature vectors between nodes of different modalities. Specifically, it can be implemented by defining cross-modal edge weights or attention mechanisms, used to enhance the semantic associations between modalities. Weighted aggregation refers to the operation of proportionally fusing multimodal carbon feature vectors. Specifically, it can be achieved by adaptive weight allocation or attention-weighted summation, which is used to eliminate redundant information between modes and retain key features.

[0146] Specifically, the semantic factor matrix and coefficient matrix are horizontally concatenated through a fully connected layer to form a joint feature matrix. The rows of this matrix represent the basic features corresponding to the semantic factors, and the columns represent the weight distribution of each modality. In the cross-modal semantic association model, the rows of the semantic factor matrix are mapped to semantic nodes in the graph structure, and the columns of the coefficient matrix are mapped to modal nodes. The graph neural network performs cross-modal propagation operations within the graph structure, extracting features from the edges between semantic nodes and modal nodes through graph convolutional layers, while using residual connections to avoid the gradient vanishing problem. After multiple propagation layers, the feature vectors of each node are weighted using an attention mechanism, and finally aggregated according to the weight ratio to generate a semantic vector with cross-modal consistency.

[0147] Furthermore, the step of concatenating the semantic factor matrix and the coefficient matrix to construct a cross-modal semantic association model includes: constructing a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix to nodes; performing residual connections on the rows of the semantic factor matrix and the columns of the coefficient matrix based on graph convolution operations; and normalizing the connection results using LayerNorm to form a cross-modal semantic association model.

[0148] In this context, node representation refers to mapping the rows and columns of a matrix to independent entities in a graph structure. Specifically, this can be achieved by using the row and column vectors from matrix decomposition as initial node features, and then converting the row and column indices into vector form through an embedding layer. Graph convolution operations refer to passing node features based on the adjacency matrix, which can be implemented using a multi-layer graph attention network to enhance feature representation by aggregating information from neighboring nodes. Residual connections stack the features before and after convolution, which can be implemented using a skip connection structure to alleviate the gradient vanishing problem in deep network training. LayerNorm normalization standardizes the features, which can be achieved by calculating the mean and variance along the feature dimension, stabilizing the training process and accelerating convergence.

[0149] Specifically, the row vectors of the semantic factor matrix represent the latent features of different semantic components, while the column vectors of the coefficient matrix reflect the contribution weights of each modality to the semantic components. After mapping the row and column vectors to graph nodes, bidirectional connections are constructed to form a heterogeneous information network. The node states are iteratively updated through graph convolutional layers, fusing implicit associations between rows and columns during information propagation. Residual connections preserve the original matrix factorization results, preventing higher-order features from overwriting basic semantic information. Normalization operations constrain the distribution range of node features, ensuring the stability of the cross-modal association model.

[0150] S280. Utilize large models and AI frameworks to perform fusion reasoning on cross-modal semantic vectors to generate the final cross-modal semantic alignment result.

[0151] In this embodiment, when extracting vectors using federated learning, the semantic factor matrix and coefficient matrix can be concatenated and fused; the fused matrix can be input into the association learning network; cross-modal information can be propagated in the association learning network using a graph neural network to generate association feature vectors; distance metric learning can be performed on the association feature vectors to output aligned cross-modal semantic vectors.

[0152] Specifically, when generating the associated feature vector, the fused matrix is ​​weighted and aggregated based on attention map convolution; the weighted aggregation result is normalized; the normalized result is input into the multi-head attention module; and the final associated feature vector from the multi-head attention module is output.

[0153] In another embodiment, after alignment is completed, the method further includes:

[0154] Online model fine-tuning: The alignment results are input into the online fine-tuning module, and the parameters of the large model are fine-tuned based on the incremental learning strategy;

[0155] Model evaluation: The accuracy and recall of the fine-tuned large model are evaluated on a specific cross-modal task;

[0156] Feedback optimization: Input the evaluation results into the feedback optimization module to adjust the hyperparameters of the association learning network;

[0157] Model Deployment: Deploy the optimized model to the inference service layer of the AI ​​framework structure.

[0158] In another embodiment, after alignment is completed, it also includes...

[0159] Asynchronous batch processing: Store the intermediate results of steps S110 to S150 executed multiple times to the feature cache module;

[0160] Cache read: Read the required intermediate results from the feature cache module in real time to accelerate projection mapping;

[0161] Incremental update: The projected representation in the feature cache module is periodically updated;

[0162] Historical backtracking: Generate logs for the cache update process for backtracking and auditing.

[0163] In another embodiment, after step S110, the method further includes:

[0164] Security encryption: Homomorphic encryption is performed on the unified representation of the preprocessed multimodal carbon-related data;

[0165] Privacy protection: The process of projection mapping and matrix factorization is kept encrypted;

[0166] Secure decryption: Perform secure decryption on the alignment results after association learning is complete;

[0167] Access control verification: Perform access control verification on the decrypted alignment result.

[0168] In another embodiment, prior to step S150, the method further includes:

[0169] Feature enhancement: Semantic enhancement is performed on the aligned cross-modal semantic vectors based on generative adversarial networks;

[0170] Discriminative training: The enhanced vectors are input into the discriminative network and subjected to adversarial training against the real vectors;

[0171] Fusion Adjustment: Adjust the fusion parameters of the large model based on the discriminative training results;

[0172] Output the enhanced alignment result: Use the adjusted vector as the input for step S150.

[0173] In summary, this application enables deep semantic alignment of multimodal carbon-related data. In scenarios such as video content understanding and intelligent customer service dialogue, it can accurately capture the correspondence between text descriptions and visual images, and simultaneously analyze the emotional tendencies of voice information. The system reduces computational complexity through modular design, improving processing efficiency while maintaining semantic consistency, making it particularly suitable for applications requiring real-time processing of multimodal carbon-related data.

[0174] For the corresponding method embodiments described above, see [link to relevant documentation]. Figure 3 The diagram shows a cross-modal semantic alignment system, which includes: a multimodal carbon-related data acquisition module 310, a projection mapping module 320, a matrix solving module 330, a federated learning module 340, and an alignment module 350.

[0175] The multimodal carbon-related data acquisition module 310 is used to extract text features, image features and sound features from the input multimodal carbon-related data and perform standardized preprocessing to obtain multimodal carbon features represented in a unified format;

[0176] The projection mapping module 320 is used to perform semantic mapping on text features, image features and sound features using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space;

[0177] Matrix solution module 330 is used to perform factorization on the projected representation using a matrix factorization algorithm to obtain the semantic factor matrix and its coefficient matrix;

[0178] The federated learning module 340 is used to build a cross-modal semantic association model based on the association learning algorithm and output aligned cross-modal semantic vectors;

[0179] Alignment module 350 is used to perform fusion inference on cross-modal semantic vectors using large models and AI frameworks to generate the final cross-modal semantic alignment result.

[0180] Among them, the multimodal carbon-related data acquisition module 310 refers to a device that separates text, image and sound features from multimodal carbon-related data through a feature extractor and performs standardization processing. Specifically, it can use the BERT model to extract text word embeddings, ResNet to extract image features, and Mel spectrograms to extract sound features, and work with a batch normalization layer to achieve feature format unification.

[0181] The projection mapping module 320 refers to the algorithm component that maps high-dimensional features to a low-dimensional common space. Specifically, it can use a linear transformation layer in conjunction with the ReLU activation function and a self-attention mechanism to achieve cross-modal feature dimension compression.

[0182] The matrix solution module 330 refers to the computational unit that extracts semantic factors through matrix decomposition. Specifically, the singular value decomposition algorithm combined with L2 regularization constraints can be used to achieve stable extraction of semantic factors.

[0183] Federated learning module 340 refers to a distributed training framework for building cross-modal association models. Specifically, it can use graph neural networks combined with residual connections and layer normalization operations to achieve cross-modal feature interaction.

[0184] Alignment module 350 refers to the decision unit for fusion reasoning, which can be implemented by using the Transformer architecture in conjunction with a multi-head attention mechanism to achieve dynamic fusion of semantic vectors.

[0185] Specifically, the multimodal carbon-related data acquisition module 310 first extracts the original features of text, image, and sound using a pre-trained model. Bilinear interpolation is then used to achieve spatial alignment between image and text features, followed by batch normalization to eliminate intermodal dimensionality differences. The projection mapping module 320 performs linear dimensionality reduction on the normalized features, using a self-attention mechanism to capture cross-modal global dependencies and generate a compact representation in a low-dimensional semantic space. The matrix solution module 330 arranges the projected vectors into a semantic matrix according to the time series, extracts principal components through singular value decomposition to construct a semantic factor matrix, and calculates the weight coefficients of each factor. The federated learning module 340 concatenates the semantic factors and coefficient matrix into a graph-structured data, iteratively updates the node representations through graph convolutional layers, and finally outputs cross-modal aligned semantic vectors. The alignment module 350 receives the semantic vectors and uses a Transformer-based fusion network for multimodal inference to generate the final semantic alignment result.

[0186] In this embodiment, the multimodal carbon-related data acquisition module 310 is specifically used for:

[0187] Text features, image features, and sound features are extracted from the input multimodal carbon-related data using a feature extractor. The text features are then segmented and part-of-speech tagged. Context-sensitive word embedding features are extracted using a pre-trained large model.

[0188] After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain the mapping result.

[0189] Batch normalization is performed on the mapping result to obtain a normalized result, wherein the normalization is a norm that balances the features of each modality;

[0190] The normalization result is input into the encoder to generate multimodal carbon features in a unified format.

[0191] In this embodiment, the multimodal carbon-related data acquisition module 310 is specifically used for:

[0192] The image features and word embedding features are mapped one-to-one according to spatial coordinates and semantic tags;

[0193] Based on the correspondence, the coordinates of four adjacent pixels are identified in the image region of the image feature;

[0194] Calculate the horizontal and vertical distance weights between the coordinates of the four pixels and the target word embedding position in the image features;

[0195] The feature values ​​of the four pixels are weighted proportionally by the horizontal and vertical distance weights.

[0196] The weighted results are summed to form the single-point image features after bilinear interpolation;

[0197] The single-point image features are normalized using the L2 norm to obtain the mapping result.

[0198] In practical applications, the batch normalization layer of the multimodal carbon-related data acquisition module 310 can be configured to dynamically adjust the L2 norm threshold of each modality feature, for example, limiting the image feature norm to the 0-1 range and the text feature norm to the 1-2 range. The number of self-attention heads in the projection mapping module 320 can be set to 8, with each head having a 64-dimensional dimension. The singular value truncation parameter P of the matrix solution module 330 can be dynamically calculated based on the dimension of the input data, for example, set to 20% of the total dimension. The graph convolutional layer of the federated learning module can adopt a gated recurrent unit structure, and the number of iterations can be set to 3.

[0199] In this embodiment, the projection mapping module 320 is specifically used for:

[0200] A linear transformation is performed on the text features, the image features, and the sound features to generate an initial projection vector;

[0201] The initial projection vector is nonlinearly mapped using the ReLU activation function to obtain the ReLU vector;

[0202] By using batch normalization, the ReLU vector is scaled and translated to obtain the projection vector;

[0203] The global dependencies in the projection vector are extracted using a self-attention mechanism to obtain the projection representation in the low-dimensional common semantic space.

[0204] In this embodiment, the projection mapping module 320 is specifically used for:

[0205] Extract the query vector, key vector, and value vector from the projection vector;

[0206] The attention weights of the query vector, the key vector, and the value vector are calculated by scaling the dot product.

[0207] The attention weights are subjected to softmax normalization, and the normalized weights are multiplied and accumulated with the query vector, the key vector and the value vector to obtain the projection representation in the low-dimensional common semantic space.

[0208] In this embodiment, the matrix solving module 330 is specifically used for:

[0209] Arrange the projection vectors in the projection representation into a semantic matrix according to time order or spatial position;

[0210] An L2 regularization term is added to the semantic matrix, and singular values ​​and singular vectors are extracted using singular value decomposition.

[0211] Select the singular vectors corresponding to the first P singular values ​​to construct a semantic factor matrix, calculate and extract the weight of each element in the semantic factor matrix, and construct a coefficient matrix based on the weights, where P is an integer greater than 1.

[0212] In this embodiment, the federated learning module 340 is specifically used for:

[0213] The semantic factor matrix and the coefficient matrix are concatenated to construct a cross-modal semantic association model;

[0214] Feature vectors are extracted by using graph neural networks to propagate information across modalities, and the extracted feature vectors are weighted and aggregated to obtain aligned cross-modal semantic vectors.

[0215] In this embodiment, the federated learning module 340 is specifically used for:

[0216] Construct a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix to nodes;

[0217] The semantic factor matrix and the coefficient matrix are residually joined based on graph convolution operations, and the results obtained by the join are normalized using LayerNorm to form a cross-modal semantic association model.

[0218] In summary, this system achieves multimodal carbon feature space alignment during the data acquisition stage through multi-module collaborative processing, introduces an attention mechanism to capture cross-modal correlations during the projection stage, improves the stability of semantic factors through regularization constraints during the matrix factorization stage, and constructs a graph structure to achieve cross-modal information propagation during the federated learning stage, ultimately forming an end-to-end semantic alignment framework.

[0219] By employing a phased, progressive feature processing approach, the system effectively addresses the semantic gap between modalities and achieves deep semantic alignment of multimodal carbon-related data. In scenarios such as video content understanding and intelligent customer service dialogue, it accurately captures the correspondence between text descriptions and visual images, simultaneously analyzing the emotional tone of voice information. The system's modular design reduces computational complexity, improving processing efficiency while maintaining semantic consistency, making it particularly suitable for applications requiring real-time processing of multimodal carbon-related data.

[0220] This application further proposes a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, they cause the processor to implement a cross-modal semantic alignment method. This method includes extracting text, image, and sound features from multimodal carbon-related data and performing standardized preprocessing; generating a projected representation of a low-dimensional common semantic space through a projection algorithm; decomposing the data into a semantic factor matrix and a coefficient matrix using a matrix factorization algorithm; establishing a cross-modal semantic association model based on an association learning algorithm and outputting aligned semantic vectors; and finally generating semantic alignment results through fusion inference using a large model and an AI framework structure.

[0221] Computer-readable storage media refers to physical carriers capable of persistently storing data and supporting processor access. These can be implemented using solid-state drives (SSDs), flash memory chips, or optical discs. Their function is to provide a stable and reliable instruction storage environment for algorithm execution. Computer-executable instructions are sets of code written in programming languages ​​that can be compiled and executed. These can be implemented using languages ​​such as Python, C++, or Java. Their function is to translate cross-modal semantic alignment methods into operational flows that can be recognized by hardware. A processor is an integrated circuit that performs arithmetic and logical operations. It can be implemented using multi-core CPUs, GPUs, or TPU architectures. Its function is to accelerate feature extraction, matrix factorization, and model training processes through parallel computing.

[0222] Specifically, when the computer-executable instructions in the storage medium are loaded into the processor, the processor sequentially performs multimodal carbon feature extraction, semantic space projection, matrix factorization, and model building operations according to a preset procedure. For example, in the feature extraction stage, the processor calls a pre-trained large model to perform word segmentation and part-of-speech tagging on the text, while using a convolutional neural network to extract image features and Fourier transform to extract sound spectral features. In the projection stage, the processor performs linear transformations and nonlinear activations on the multimodal carbon features, combined with a self-attention mechanism to capture global semantic dependencies. In the matrix factorization stage, the processor uses a singular value decomposition algorithm to extract the semantic factor matrix from the projected representation and optimizes the stability of the decomposition results through regularization constraints. Finally, the processor inputs the semantic factor matrix and coefficient matrix into a graph neural network, uses a cross-modal information propagation mechanism to generate aligned semantic vectors, and performs multimodal inference fusion through a large model.

[0223] Through the above technical solution, this application solves the problem that traditional storage media cannot support end-to-end cross-modal semantic alignment, and realizes fully automated processing of multimodal carbon features from extraction, mapping to fusion. This solution significantly improves the computational efficiency of cross-modal semantic alignment by optimizing the collaborative mechanism between the instruction set and the processor, while utilizing matrix factorization and graph neural networks to ensure the accuracy of semantic associations. It is particularly suitable for multimodal carbon-related data processing needs in weakly supervised scenarios.

[0224] This embodiment also provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor. The processor executes the machine-executable instructions to implement the aforementioned cross-modal semantic alignment method. This electronic device can be a server or a terminal device.

[0225] See Figure 4 As shown, the electronic device includes a processor 400 and a memory 401. The memory 401 stores machine-executable instructions that can be executed by the processor 400, which executes the machine-executable instructions to implement the aforementioned cross-modal semantic alignment method.

[0226] Furthermore, Figure 4 The electronic device shown also includes a bus 402 and a communication interface 403. The processor 400, the communication interface 403 and the memory 401 are connected via the bus 402.

[0227] The memory 401 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 402 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0228] The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 400 or by instructions in software form. The processor 400 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 401. The processor 400 reads the information in memory 401 and, in conjunction with its hardware, completes the steps of the cross-modal semantic alignment method described in the aforementioned embodiment.

[0229] This embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the above-described cross-modal semantic alignment method.

[0230] The cross-modal semantic alignment method, apparatus, electronic device, and storage medium computer program product provided in the embodiments of this application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0231] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0232] Furthermore, in the description of the embodiments of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0233] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0234] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0235] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A cross-modal semantic alignment method, characterized in that, The method includes: Text features, image features, and sound features are extracted from the input multimodal carbon-related data, and standardized preprocessing is performed to obtain multimodal carbon features represented in a unified format. The text features, image features, and sound features are semantically mapped using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space. The projection representation is factored using a matrix factorization algorithm to obtain a semantic factor matrix and its coefficient matrix. Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and coefficient matrix, and the aligned cross-modal semantic vector is output. The cross-modal semantic vectors are fused and inferred using large models and AI frameworks to generate the final cross-modal semantic alignment result. The step of factoring the projected representation using a matrix factorization algorithm to obtain a semantic factor matrix and its coefficient matrix includes: Arrange the projection vectors in the projection representation into a semantic matrix according to time order or spatial position; An L2 regularization term is added to the semantic matrix, and singular values ​​and singular vectors are extracted using singular value decomposition. Select the singular vectors corresponding to the first P singular values ​​to construct a semantic factor matrix, calculate and extract the weight of each element in the semantic factor matrix, and construct a coefficient matrix based on the weights, where P is an integer greater than 1.

2. The cross-modal semantic alignment method according to claim 1, characterized in that, The process involves extracting text features, image features, and sound features from the input multimodal carbon-related data, and performing standardized preprocessing to obtain multimodal carbon features represented in a unified format, including: Text features, image features, and sound features are extracted from the input multimodal carbon-related data using a feature extractor. The text features are then segmented and part-of-speech tagged. Context-sensitive word embedding features are extracted using a pre-trained large model. After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain the mapping result. Batch normalization is performed on the mapping result to obtain a normalized result, wherein the normalization is a norm that balances the features of each modality; The normalization result is input into the encoder to generate multimodal carbon features in a unified format.

3. The cross-modal semantic alignment method according to claim 2, characterized in that, After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain the mapping result, including: The image features and word embedding features are mapped one-to-one according to spatial coordinates and semantic tags; Based on the correspondence, the coordinates of four adjacent pixels are identified in the image region of the image feature; Calculate the horizontal and vertical distance weights between the coordinates of the four pixels and the target word embedding position in the image features; The feature values ​​of the four pixels are weighted proportionally by the horizontal and vertical distance weights. The weighted results are summed to form the single-point image features after bilinear interpolation; The single-point image features are normalized using the L2 norm to obtain the mapping result.

4. The cross-modal semantic alignment method according to claim 1, characterized in that, The step of using a projection algorithm to perform semantic mapping on the text features, image features, and sound features to obtain a projected representation in a low-dimensional common semantic space includes: A linear transformation is performed on the text features, the image features, and the sound features to generate an initial projection vector; The initial projection vector is nonlinearly mapped using the ReLU activation function to obtain the ReLU vector; By using batch normalization, the ReLU vector is scaled and translated to obtain the projection vector; The global dependencies in the projection vector are extracted using a self-attention mechanism to obtain the projection representation in the low-dimensional common semantic space.

5. The cross-modal semantic alignment method according to claim 4, characterized in that, The step of extracting global dependencies from the projection vector using a self-attention mechanism to obtain a projection representation in a low-dimensional common semantic space includes: Extract the query vector, key vector, and value vector from the projection vector; The attention weights of the query vector, the key vector, and the value vector are calculated by scaling the dot product. The attention weights are subjected to softmax normalization, and the normalized weights are multiplied and accumulated with the query vector, the key vector and the value vector to obtain the projection representation in the low-dimensional common semantic space.

6. The cross-modal semantic alignment method according to claim 1, characterized in that, The association-based learning algorithm establishes a cross-modal semantic association model based on the semantic factor matrix and coefficient matrix, and outputs aligned cross-modal semantic vectors, including: The semantic factor matrix and the coefficient matrix are concatenated to construct a cross-modal semantic association model; Feature vectors are extracted by using graph neural networks to propagate information across modalities, and the extracted feature vectors are weighted and aggregated to obtain aligned cross-modal semantic vectors.

7. The method according to claim 6, characterized in that, The step of concatenating the semantic factor matrix and the coefficient matrix to construct a cross-modal semantic association model includes: Construct a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix to nodes; The semantic factor matrix and the coefficient matrix are regressively joined based on graph convolution operations, and the results are normalized using LayerNorm to form a cross-modal semantic association model.

8. A cross-modal semantic alignment system, characterized in that, The system includes: The multimodal data acquisition module is used to extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features represented in a unified format. The projection mapping module is used to perform semantic mapping on the text features, the image features and the sound features using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space. The matrix decomposition module is used to perform factorization on the projected representation using a matrix decomposition algorithm to obtain the semantic factor matrix and its coefficient matrix. The federated learning module is used to establish a cross-modal semantic association model based on the semantic factor matrix and coefficient matrix using the association learning algorithm, and output the aligned cross-modal semantic vector. The alignment module is used to perform fusion reasoning on the cross-modal semantic vectors using a large model and AI framework structure to generate the final cross-modal semantic alignment result; The step of factoring the projected representation using a matrix factorization algorithm to obtain a semantic factor matrix and its coefficient matrix includes: Arrange the projection vectors in the projection representation into a semantic matrix according to time order or spatial position; An L2 regularization term is added to the semantic matrix, and singular values ​​and singular vectors are extracted using singular value decomposition. Select the singular vectors corresponding to the first P singular values ​​to construct a semantic factor matrix, calculate and extract the weight of each element in the semantic factor matrix, and construct a coefficient matrix based on the weights, where P is an integer greater than 1.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the cross-modal semantic alignment method according to any one of claims 1-7.

Images