Cross-modal semantic alignment method and system and storage medium

Abstract

The invention provides a cross-modal semantic alignment method and system and a storage medium. According to the method, multi-modal carbon features are extracted and subjected to standardized preprocessing, the features are mapped to a low-dimensional public semantic space through a projection algorithm, cross-modal semantic association is constructed in combination with matrix decomposition and an association learning model, and finally an alignment result is generated through large model fusion reasoning. The problem that a traditional method is insufficient in precision and poor in semantic coherence in cross-modal semantic alignment is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a cross-modal semantic alignment method, system, and storage medium. Background Art

[0002] With the rapid growth of multimodal carbon-related data, establishing semantically consistent representations across modalities has become a core challenge for intelligent systems. Traditional methods often rely on a unified embedding space, but this suffers from insufficient precision in maintaining semantic consistency, particularly in weakly supervised or unsupervised scenarios. Therefore, a cross-modal semantic alignment method that integrates multiple techniques to improve semantic alignment accuracy is urgently needed. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a cross-modal semantic alignment method, system and storage medium, which can achieve high-precision semantic alignment of different modal data by constructing a cross-modal feature space, performing projection transformation, performing matrix decomposition, and introducing an association learning mechanism, thereby improving the performance of the multimodal understanding system.

[0004] In a first aspect, an embodiment of the present application provides a cross-modal semantic alignment method, comprising: Extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardization preprocessing to obtain multimodal carbon features represented in a unified format; Performing semantic mapping on the text features, the image features, and the sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space; Factoring the projected representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix; Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and the coefficient matrix, and an aligned cross-modal semantic vector is output; The cross-modal semantic vectors are fused and inferred using a large model and AI framework structure to generate the final cross-modal semantic alignment result.

[0005] In another feasible embodiment, the extracting of text features, image features, and sound features from the input multimodal carbon-related data and performing standardization preprocessing to obtain multimodal carbon features represented in a unified format includes: The feature extractor extracts text features, image features, and sound features from the input multimodal carbon-related data, performs word segmentation and part-of-speech tagging on the text corresponding to the text features, and extracts context-sensitive word embedding features through a pre-trained large model; After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain a mapping result; Performing batch normalization on the mapping results to obtain normalized results, wherein the normalization is to balance the norms of the features of each modality; The normalized results are input into an encoder to generate multimodal carbon features represented in a unified format.

[0006] In another feasible implementation, after aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain a mapping result, including: One-to-one correspondence between the image features and the word embedding features according to spatial coordinates and semantic labels; According to the corresponding relationship, coordinates of four adjacent pixels are identified in the image area of the image feature; Calculate the horizontal and vertical distance weights between the coordinates of the four pixel points and the target word embedding position in the image feature; Weighting the characteristic values of the four pixel points proportionally according to the horizontal and vertical distance weights; The weighted results are accumulated to form the single-point image features after bilinear interpolation; Perform L2 norm normalization on the single-point image feature to obtain a mapping result.

[0007] In another feasible implementation, the semantic mapping of the text features, the image features, and the sound features using a projection algorithm to obtain a projected representation in a low-dimensional common semantic space includes: Performing linear transformation on the text feature, the image feature, and the sound feature to generate an initial projection vector; Performing nonlinear mapping on the initial projection vector through a ReLU activation function to obtain a ReLU vector; Using batch normalization, the ReLU vector is scaled and translated to obtain a projection vector; A self-attention mechanism is used to extract the global dependency in the projection vector to obtain a projection representation in a low-dimensional common semantic space.

[0008] In another feasible implementation, the extracting global dependencies in the projection vector using a self-attention mechanism to obtain a projection representation in a low-dimensional common semantic space includes: extracting a query vector, a key vector, and a value vector of the projected vector; Calculating attention weights of the query vector, the key vector, and the value vector by scaling the dot product; Softmax normalization is performed on the attention weights, and based on the normalized weights, the query vector, the key vector, and the value vector are multiplied and accumulated to obtain a projection representation in a low-dimensional common semantic space.

[0009] In another feasible implementation, factoring the projected representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix includes: Arranging the projection vectors in the projection representation into a semantic matrix according to time sequence or spatial position; Adding an L2 regularization term to the semantic matrix and extracting singular values and singular vectors therein using a singular value decomposition method; The singular vectors corresponding to the first P singular values are selected to construct a semantic factor matrix, and the weight of each element in the semantic factor matrix is calculated and extracted, and a coefficient matrix is constructed based on the weight, where P is an integer greater than 1.

[0010] In another feasible implementation, the association learning algorithm is based on establishing a cross-modal semantic association model based on the semantic factor matrix and the coefficient matrix, and outputting an aligned cross-modal semantic vector, including: splicing the semantic factor matrix and the coefficient matrix, and constructing a cross-modal semantic association model; The feature vectors are extracted by cross-modal information propagation through graph neural networks, and the extracted feature vectors are weighted and aggregated to obtain the aligned cross-modal semantic vectors.

[0011] In another feasible implementation, the concatenating the semantic factor matrix and the coefficient matrix and constructing a cross-modal semantic association model includes: Constructing a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix into nodes; Based on the graph convolution operation, the rows of the semantic factor matrix and the columns of the coefficient matrix are residually connected, and the connected results are normalized using LayerNorm to form a cross-modal semantic association model.

[0012] In a second aspect, an embodiment of the present application provides a cross-modal semantic alignment system, including: The multimodal data acquisition module is used to extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features expressed in a unified format; A projection mapping module, configured to perform semantic mapping on the text features, the image features, and the sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space; A matrix solving module, configured to factorize the projection representation using a matrix decomposition algorithm to obtain a semantic factor matrix and a coefficient matrix thereof; a federated learning module for establishing a cross-modal semantic association model based on the semantic factor matrix and the coefficient matrix based on an association learning algorithm, and outputting an aligned cross-modal semantic vector; The alignment module is used to use a large model and an AI framework structure to perform fusion reasoning on the cross-modal semantic vectors to generate a final cross-modal semantic alignment result.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned cross-modal semantic alignment method.

[0014] The embodiments of the present application bring the following beneficial effects: The above-mentioned cross-modal semantic alignment method, system and storage medium extract text, image and sound features from multimodal carbon-related data and perform standardized preprocessing, map them to a low-dimensional public semantic space using a projection algorithm, obtain semantic factor matrix and coefficient matrix through matrix decomposition, establish a cross-modal semantic association model based on association learning, and finally generate alignment results by combining large-scale model fusion reasoning. This application introduces matrix decomposition technology to remove the influence of modality-specific noise on core semantics, so that semantic consistency can be maintained under weakly labeled data conditions; introduces a federated learning algorithm to build a model, enhances the ability to capture long-distance semantic dependencies, and significantly improves alignment accuracy when dealing with complex scenarios; cites AI and large models to provide a clear reasoning path, showing stronger cross-modal reasoning capabilities in open-domain question-answering scenarios; thereby effectively solving the semantic bias problem caused by inconsistent cross-modal feature distribution.

[0015] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description or be understood by practicing the present application. The objectives and other advantages of the present application are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of an embodiment of the cross-modal semantic alignment method provided in an embodiment of the present application; Figure 2 A flowchart of another embodiment of the cross-modal semantic alignment method provided in an embodiment of the present application; Figure 3 A schematic diagram of a cross-modal semantic alignment system provided in an embodiment of the present application; Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0020] The terms "first," "second," "third," "fourth," and so forth (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0021] In related technologies, multimodal carbon-related data processing has evolved from unimodal analysis to cross-modal correlation. Early research primarily used feature concatenation to integrate information from different modalities, but this failed to address the semantic gap. Subsequent approaches have attempted to construct a unified embedding space, but maintaining semantic consistency across modalities under weak supervision has proven difficult. For example, in video content analysis, the temporal misalignment between spoken descriptions and on-screen actions leads to semantic mismatches, and traditional projection methods are unable to effectively eliminate differences in cross-modal feature distribution. To address the above issues, the inventors discovered the following three key flaws: the heterogeneity of multimodal carbon features makes alignment difficult, shallow mapping cannot capture deep semantic associations, and static models are difficult to adapt to dynamic data distributions. Through analysis, it was found that different modalities have common factors that can be aligned at the potential semantic level. If the semantic elements can be decoupled in a low-dimensional space and dynamic associations can be established, the accuracy of cross-modal alignment can be improved. Based on this, a phased processing framework is proposed. First, the feature representation is unified through nonlinear mapping, then the semantic factors are decomposed to construct a dynamic association model, and finally the reasoning capability of the large model is used to achieve end-to-end alignment.

[0022] For ease of understanding, the specific process of the embodiment of the present application is described below. The embodiment of the present application is applied to system devices such as computers and servers. Please refer to Figure 1 In one embodiment of the cross-modal semantic alignment method of the present application, the embodiment mainly realizes the unified alignment of cross-modal semantic information by combining the projection algorithm, the matrix decomposition algorithm and the association learning algorithm, and specifically includes the following steps: S110, extracting text features, image features, and sound features from the input multimodal carbon-related data, and performing standardization preprocessing to obtain multimodal carbon features represented in a unified format; In this step, the system device first receives multimodal input data collected from the user or external device, and the data includes at least natural language text, images (such as photos, icons) and sound signals (such as voice, ambient sound).

[0023] Text features are segmented and tagged using the natural language processing module, and context-sensitive word embedding representations are generated using pre-trained language models (such as BERT and GPT). Image features are extracted using convolutional neural networks (CNNs) to extract feature maps. Sound features are extracted using short-time Fourier transforms to extract Mel-frequency cepstral coefficients (MFCCs), or high-dimensional semantic features are extracted using pre-trained audio models. All modal features undergo dimensional normalization, including zero-mean unit variance standardization and modulus length normalization, ensuring that each modal feature is expressed on a unified scale, providing a foundation for subsequent fusion.

[0024] It should be noted that standardization preprocessing refers to the process of converting the features of different modes into a unified numerical distribution. Specifically, it can be achieved by combining batch normalization with feature scaling to eliminate the dimensional differences of the features of each mode.

[0025] The projection algorithm refers to a method of mapping high-dimensional features to low-dimensional space. Specifically, it can be implemented by using linear transformation combined with nonlinear activation function to capture potential semantic associations across modalities.

[0026] Matrix decomposition algorithm refers to the technology of decomposing complex matrices into low-rank matrix products. Specifically, it can be implemented using singular value decomposition combined with regularization constraints to extract semantic factors shared across modalities.

[0027] The association learning algorithm refers to a method of establishing semantic associations between modalities. Specifically, it can be implemented using a graph neural network combined with an attention mechanism to dynamically model the interaction between semantic factors.

[0028] Specifically, this step extracts deep features of each modality through pre-trained models, such as using BERT to extract text context features, ResNet to extract image spatial features, and Mel spectrum analysis to extract sound time-frequency features.

[0029] S120, performing semantic mapping on text features, image features, and sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space; In this example, each modal feature first passes through a linear transformation module and is projected into a specified dimensional space. It is then input into a ReLU nonlinear activation function module to enhance the model's expressiveness. A batch normalization module then scales and translates the activated vectors to accelerate convergence and stabilize training. Finally, a multi-head self-attention mechanism captures the global semantic dependencies between modalities, outputting a semantically aligned low-dimensional common representation vector, known as the projected representation.

[0030] S130, factoring the projected representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix; In this step, the projected representations from the previous step are arranged in temporal or spatial order into a semantic matrix, and an L2 regularization term is added to constrain the weights. Next, the semantic matrix is decomposed using singular value decomposition (SVD), yielding a set of singular values and their corresponding singular vectors. The first P principal components are selected to construct a semantic factor matrix, and a coefficient matrix is generated based on the projection of the original matrix onto this subspace. This factor matrix represents the structure of the shared semantic space, while the coefficient matrix describes the contribution of the original features to this space.

[0031] S140: Based on an association learning algorithm, establish a cross-modal semantic association model based on the semantic factor matrix and the coefficient matrix, and output an aligned cross-modal semantic vector; Specifically, the semantic factor matrix and coefficient matrix are concatenated and used as input to construct a graph neural network model. Nodes are structured based on semantic units and their combined relationships. During graph convolutional propagation, each node fuses semantic representations from different modalities, extracting cross-modal semantic relationships through multi-layered GNN operations. Finally, a multi-head attention mechanism is used to weight the importance of node semantics, outputting an aligned cross-modal semantic vector.

[0032] S150: Use the large model and AI framework structure to perform fusion reasoning on cross-modal semantic vectors to generate the final cross-modal semantic alignment result.

[0033] Specifically, the system device uses an integrated inference engine (such as TensorRT or ONNXRuntime) on the deployed AI platform to load the multimodal fusion model, and inputs the aligned semantic vector output in step S40 into the downstream classifier or semantic matching module to obtain the alignment result output of the corresponding task, such as text-image matching, speech-image retrieval, etc.

[0034] In this embodiment, the method first extracts the deep features of each modality through a pre-trained model, such as using BERT to extract text context features, ResNet to extract image spatial features, and Mel spectrum analysis to extract sound time-frequency features. Then, a bilinear projection method is used to map the heterogeneous features to a unified dimension, and a nonlinear activation function is introduced in this process to enhance the model's expressiveness. The projected features are subjected to matrix decomposition to obtain a factor matrix representing the core semantics and a coefficient matrix reflecting the instance characteristics. The product of these two matrices can reconstruct the original projection features. A graph structure model is constructed based on the decomposition results, in which nodes represent semantic factors, edge weights are determined by the coefficient matrix, and the node status is iteratively updated through a graph propagation algorithm. Finally, the optimized semantic vector is input into the multimodal large model, and its powerful reasoning ability is used to comprehensively judge the semantic alignment relationship.

[0035] Compared with the existing technology, the solution provided by this application mainly introduces matrix decomposition and graph neural network technology. The traditional projection method directly maps features to a unified space, which is prone to losing fine-grained semantic information. However, this application retains core semantic elements by decomposing semantic factors. Existing graph models usually construct nodes based on original features, while this application uses decomposed semantic factors to construct a more compact graph structure, significantly reducing computational complexity. Existing large model applications mostly use feature splicing input. This application provides structured semantic representation, enabling large models to perform cross-modal reasoning more effectively.

[0036] Through the above technical solutions, this application effectively solves the problem of semantic deviation caused by inconsistent cross-modal feature distribution, and can achieve accurate alignment of voice description and screen action in video semantic understanding tasks. Matrix decomposition technology removes the influence of modality-specific noise on core semantics, so that semantic consistency can be maintained under weakly labeled data conditions. The combination of graph neural networks and attention mechanisms enhances the model's ability to capture long-distance semantic dependencies and significantly improves alignment accuracy when dealing with complex scenarios. Structured semantic representation provides a clear reasoning path for large models, demonstrating stronger cross-modal reasoning capabilities in open-domain question-answering scenarios.

[0037] like Figure 2 As shown, another embodiment of the cross-modal semantic alignment method provided by this application specifically includes the following steps: S210, extracting text features, image features, and sound features from the input multimodal carbon-related data through a feature extractor, performing word segmentation and part-of-speech tagging on the text corresponding to the text features, and extracting context-sensitive word embedding features through a pre-trained large model; Specifically, by obtaining data samples of different modalities, the original features of each data sample are extracted using a modality-specific encoder, the original features are normalized based on a feature regularization method, and the normalized original features are preliminarily semantically mapped using shared semantic annotations to obtain the features of each modality, that is, extracting text features, image features, and sound features.

[0038] It can be understood that first, the feature extractor is used to extract text, image and sound features from the input multimodal carbon-related data respectively; then the text input is segmented and tagged with parts of speech, and language models such as BERT are used to extract context-related word embedding features.

[0039] The image features are obtained by extracting a fixed-dimensional multi-channel image feature map through a ResNet network. The sound features are obtained by extracting a time-series sound semantic feature through an audio convolutional network or a Wav2Vec model.

[0040] Word segmentation refers to the process of dividing continuous text into independent semantic units. It can be implemented using word segmentation tools based on rules or statistical models to eliminate redundant information in the text and extract basic semantic units.

[0041] Part-of-speech tagging refers to labeling the grammatical category of each word after word segmentation. It can be implemented using a hidden Markov model or a conditional random field algorithm to assist in identifying the functional role of a word in the context.

[0042] A pre-trained large model refers to a language model that is pre-trained using massive amounts of data, such as a BERT or GPT structure, which is used to extract word vector representations that contain contextual associations.

[0043] In practical applications, regularization processing is performed on the input text data, including unifying the encoding format, removing stop words, and symbol filtering; bidirectional LSTM or Transformer structures are used for part-of-speech tagging to improve the accuracy of grammatical structure analysis; large-scale pre-trained models such as BERT or ERNIE are used to extract contextualized word embeddings; and each word embedding and its corresponding part-of-speech tag are used as a combined feature for subsequent processing.

[0044] S220, after aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain a mapping result; It can be understood that the image features and the word embedding features are matched one-to-one with the spatial coordinates and semantic tags; according to the correspondence, the coordinates of four adjacent pixels are identified in the image area of the image feature; the horizontal and vertical distance weights between the coordinates of the four pixel points and the target word embedding position in the image feature are calculated; the feature values of the four pixel points are proportionally weighted with the horizontal and vertical distance weights; the weighted results are accumulated to form a single-point image feature after bilinear interpolation; and the single-point image feature is normalized by the L2 norm to obtain a mapping result.

[0045] It should be noted that the one-to-one correspondence between spatial coordinates and semantic tags refers to the establishment of an associated mapping relationship between the spatial distribution of image features and the tag position of text semantics. Specifically, this can be achieved by using a coordinate encoder to convert the image grid coordinates into a vector with the same dimension as the word embedding, thereby ensuring that the local area of the image and the text semantic unit are aligned in spatial distribution. Among them, the bilinear interpolation method refers to an interpolation algorithm that performs feature weighted fusion based on the coordinate distance weights of adjacent pixel points. Specifically, it can be achieved by using a linear interpolation formula to calculate the proportional coefficients in the horizontal and vertical directions respectively, and then performing a bilinear weighted summation on the four adjacent points, thereby achieving cross-modal feature mapping while retaining the continuity of the image space. Among them, L2 norm normalization refers to the unit length standardization of the feature vector. Specifically, it can be achieved by using the square root of the sum of the squares of each element of the vector as the denominator for scaling operation, thereby eliminating the differences in the numerical scales of the features of different modalities.

[0046] Specifically, in the process of aligning image features with word embedding features, the image grid coordinates are first converted into vectors that match the word embedding dimensions through a coordinate encoder to establish a correspondence between space and semantics. After the target word embedding position is determined, the four nearest pixels in the image feature are located according to its coordinates, and the distance weights between this position and the four pixels in the horizontal and vertical directions are calculated respectively. For example, when the horizontal distance from the target position to the upper left pixel is 0.3 and the vertical distance is 0.4, the weight coefficient of the pixel is (1-0.3)*(1-0.4)=0.42. After the feature values of the four pixels are weighted and summed according to their respective weights, an interpolated image feature that matches the target word embedding position is formed. Finally, the vector length of the interpolation result is constrained to the unit range through L2 normalization to ensure the numerical stability of subsequent cross-modal fusion.

[0047] When implementing alignment and embedding, the Cross-Modal Attention mechanism can be used to calculate the alignment correlation score between the word embedding and the image region; the image region with the highest correlation is selected as the corresponding region; regional average pooling (ROI Pooling) is performed on the features of the image region to obtain low-resolution image features; and bilinear interpolation is used to map the image feature space to the word embedding space dimension.

[0048] S230, performing batch normalization on the mapping results to obtain a normalized result; It should be noted that normalization balances the norms of the features across modalities; batch normalization standardizes the mean and variance of feature data. Specifically, a moving average method can be used to calculate statistics to eliminate dimensional differences between features across modalities. An encoder is a neural network structure capable of feature conversion, such as a Transformer encoding layer, that converts heterogeneous features into a vector representation of uniform dimensionality.

[0049] Specifically, the mapping results are normalized by channel-by-channel mean and standard deviation to keep the mean consistent between modalities; BatchNorm or LayerNorm is used for normalization during the training phase; the normalization coefficient of each modality is automatically adjusted according to the modal difference to reduce the impact of modal imbalance; the normalized feature output is used as the feature baseline before multimodal fusion.

[0050] S240, inputting the normalized result into an encoder to generate multimodal carbon features represented in a unified format; Specifically, the three types of modal inputs are uniformly input into the Transformer or Multi-modal Encoder network; position encoding is used to enhance the structural representation between sequences; contextual interaction between modalities is achieved through a multi-layer attention fusion structure; and a unified format of multimodal carbon feature representation is output for subsequent projection.

[0051] In this embodiment, after obtaining the multimodal carbon features, it also includes: adding Gaussian white noise that conforms to the standard normal distribution to the multimodal carbon features; setting the noise perturbation amplitude as a dynamic learning parameter; improving the semantic invariance and robustness of the features through a comparative learning training process; the enhanced features are used to simulate semantic stability tests under weak perturbations.

[0052] Specifically, text data is segmented and POS tagged to form a structured sequence, and a pre-trained large model is used to generate a word embedding vector with contextual association. After the image features are extracted through a convolutional neural network, the spatial coordinates are aligned with the word embedding features, and the area corresponding to the semantic marker is located in the image feature map. When the bilinear interpolation method is used to map the image features to the word embedding space, the coordinates of the four adjacent pixels of the target position are first determined, and the horizontal and vertical distance weights from the target point are calculated. The feature values of the four points are proportionally weighted and then accumulated to obtain the interpolation result. The normalization process calculates the mean and standard deviation of the feature matrix, scales and translates each feature dimension, and makes the distribution range of different modal features tend to be consistent. The encoder receives the normalized multimodal carbon features and generates a vector representation with unified dimensions through multiple layers of nonlinear transformations.

[0053] S250, using a projection algorithm to perform semantic mapping on text features, image features, and sound features to obtain a projection representation in a low-dimensional common semantic space; In this step, an initial projection vector is generated by performing linear transformation on the text features, the image features, and the sound features; a ReLU vector is obtained by performing nonlinear mapping on the initial projection vector using a ReLU activation function; the ReLU vector is scaled and translated using batch normalization to obtain a projection vector; and a self-attention mechanism is used to extract global dependencies in the projection vector to obtain a projection representation in a low-dimensional common semantic space.

[0054] Among them, linear transformation refers to mapping high-dimensional features to low-dimensional space through matrix multiplication. Specifically, it can be implemented using a fully connected layer. Its function is to eliminate the dimensional differences between features of different modalities. The ReLU activation function refers to the nonlinear truncation of input values. Specifically, it can be implemented using a piecewise function. Its function is to enhance the nonlinear expression capability of the model. Batch normalization refers to the standardization of feature distribution. Specifically, it can be implemented by scaling the mean and variance. Its function is to alleviate the distribution shift problem of features of different modalities. The self-attention mechanism refers to the calculation of the dependency between features through dynamic weights. Specifically, it can be implemented using a multi-head attention module. Its function is to capture the global semantic association of cross-modal features.

[0055] Specifically, after linear transformation of text, image, and sound features, the initial projection vector is input into the ReLU activation function layer, which eliminates negative values and retains valid features. Subsequently, the batch normalization layer independently normalizes each dimension of the ReLU vector to zero mean and one variance, thereby balancing the numerical ranges of features from different modalities. The normalized projection vector is then input into the self-attention module, which dynamically aggregates global semantic information by calculating the similarity weights between the query vector, key vector, and value vector, ultimately forming a consistent projection representation in a low-dimensional common semantic space.

[0056] In another embodiment, a self-attention mechanism is used to extract global dependencies in the projection vector to obtain a projection representation in a low-dimensional public semantic space, including: extracting the query vector, key vector and value vector of the projection vector; calculating the attention weights of the query vector, key vector and value vector by scaling the dot product; performing softmax normalization on the attention weights, and multiplying and accumulating the normalized weights with the query vector, key vector and value vector to obtain a projection representation in the low-dimensional public semantic space.

[0057] It is understandable that the query vector refers to a vector used to represent the feature position that currently needs to be paid attention to. Specifically, it can be achieved by mapping the projection vector to the query vector through a linear transformation. Its function is to determine the correlation weight between different feature positions.

[0058] The key vector refers to the vector used to match the query vector. Specifically, this can be achieved by mapping the projection vector to the key vector through a linear transformation. Its role is to generate a comparable reference value for each feature position.

[0059] A value vector is a vector that carries specific feature information. This can be achieved by mapping the projection vector to a value vector through a linear transformation. Its purpose is to provide aggregatable semantic content for the attention mechanism. A scaled dot product is the process of dividing the dot product of the query vector and the key vector by the square root of the vector's dimension. This can be achieved through numerical calculations and prevents excessive dot product expansion, which can lead to vanishing gradients.

[0060] Softmax normalization refers to converting attention weights into probability distributions, which can be achieved through exponential operations and normalization calculations. Its role is to ensure that the sum of attention weights is 1 and non-negative.

[0061] Specifically, in a low-dimensional common semantic space, the projected vector undergoes a linear transformation to generate three vectors: query, key, and value. Initial attention weights are obtained by calculating the dot product of the query and key vectors and scaling them. After softmax normalization, the weight matrix is weighted and summed with the value vector to form a projected representation that reflects global dependencies. This process effectively captures long-range semantic dependencies in cross-modal data by dynamically adjusting the association strength between different feature positions, thereby addressing the semantic information loss caused by local feature aggregation in traditional methods.

[0062] S260, factoring the projected representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix; It can be understood that the projection vectors in the projection representation are arranged into a semantic matrix in time sequence or spatial position; an L2 regularization term is added to the semantic matrix, and the singular values and singular vectors therein are extracted using the singular value decomposition method; the singular vectors corresponding to the first P singular values are selected to construct a semantic factor matrix, and the weights of each element in the semantic factor matrix are calculated and extracted, and a coefficient matrix is constructed based on the weights, where P is an integer greater than 1.

[0063] The semantic matrix refers to a two-dimensional matrix formed by arranging the projection vectors of different time points or spatial positions in a regular manner. It can be implemented by time series stacking or spatial grid mapping, and is used to uniformly represent the distribution structure of multimodal carbon characteristics.

[0064] Specifically, let the projection be represented by the matrix Z∈R n×d , where n represents the number of samples or the number of time / space nodes, and d represents the projection dimension.

[0065] Treat Z as consisting of n projection vectors z1,z2,...,z n , arranged in time sequence or space to form a semantic matrix M∈R n×d .

[0066] To improve robustness, the L2 regularization term is introduced to reduce MM: M′=M+λI, Where I is the identity matrix, λ∈R + is the regularization strength hyperparameter.

[0067] The L2 regularization term refers to adding a constraint term on the sum of squares of matrix elements in the objective function. Specifically, the weight decay coefficient can be used to control the regularization strength to prevent overfitting problems during matrix decomposition.

[0068] The singular value decomposition refers to a mathematical method of decomposing a matrix into the product of three matrices. Specifically, truncated singular value decomposition can be used to retain the main components to extract the latent semantic factors in the semantic matrix.

[0069] The first P singular values refer to the P largest singular values retained after sorting in descending order. The P value can be determined based on the cumulative contribution rate of the eigenvalues or a preset threshold, and is used to construct a low-dimensional semantic factor matrix to capture core semantic information.

[0070] Specifically, the regularized semantic matrix M′ is subjected to singular value decomposition (SVD): M′=UΣV T , in: U∈R n×n U is the left singular vector matrix; Σ∈R n×d is a diagonal singular value matrix; V∈R d×d is the right singular vector matrix.

[0071] Keep the first P largest singular values: Σ P =diag(σ1,...,σ P ); The corresponding left singular vectors form the matrix U P ∈R n×P ; The corresponding right singular vectors form the matrix V P ∈R d×P， For constructing the semantic factor matrix: F=V P ∈R d×P , Represents the linear explanation direction of each dimension of projection feature on P factors.

[0072] For calculating the weight coefficient matrix: C = MF ∈ R n×P , That is, the projection of the original semantic matrix on the semantic factor matrix reflects the semantic contribution of each sample to each factor.

[0073] Normalization: To improve the stability of subsequent graph learning, column normalization is performed on FF and CC respectively:

[0074] Final output: normalized semantic factor matrix With the coefficient matrix Can be input into graph neural networks for association learning.

[0075] Specifically, the projection vectors are first arranged into a semantic matrix in time sequence or spatial position, for example, the projection vectors of the video frames are arranged as row vectors according to timestamps. Subsequently, an L2 regularization term is added in the decomposition process to constrain the amplitude of the matrix elements, for example, a Frobenius norm penalty term is added to the objective function. The semantic matrix is decomposed by singular value decomposition to obtain a diagonal matrix containing singular values and a left and right singular vector matrix. The left singular vectors corresponding to the first P singular values are selected to construct a semantic factor matrix, for example, when P is 100, the first 100 principal components are retained. Finally, the weight coefficients of each semantic factor in the original matrix are calculated by the least squares method to form a coefficient matrix, for example, the weight parameters are iteratively optimized using the gradient descent method.

[0076] S270: Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and the coefficient matrix, and an aligned cross-modal semantic vector is output; In this step, the semantic factor matrix and the coefficient matrix are spliced together to construct a cross-modal semantic association model; feature vectors are extracted by using a graph neural network to propagate information across modalities, and the extracted feature vectors are weighted and aggregated to obtain aligned cross-modal semantic vectors.

[0077] It is understood that the semantic factor matrix refers to the set of basic features obtained through matrix decomposition that reflect cross-modal semantic commonalities. Specifically, this can be achieved by extracting the first P principal components using the singular value decomposition method. It is used to characterize the potential semantic associations between different modalities. The coefficient matrix refers to the set of parameters that reflects the weight distribution of each modal feature in the semantic factor matrix. Specifically, this can be achieved by calculating the weight coefficients through linear regression or least squares method. It is used to quantify the contribution of different modalities to the common semantic space. Splicing refers to the operation of connecting the semantic factor matrix and the coefficient matrix along a specific dimension. Specifically, this can be achieved using a fully connected layer or tensor splicing operation. It is used to build the basic data structure for cross-modal interaction. Graph neural networks are deep learning models that transfer information based on graph structures. Specifically, they can be implemented using graph convolutional networks or graph attention networks. They are used to capture nonlinear relationships between cross-modal nodes. Cross-modal information propagation refers to the process of transferring feature vectors between nodes of different modalities. Specifically, this can be achieved by defining cross-modal edge weights or an attention mechanism. It is used to enhance the semantic associations between modalities. Weighted aggregation refers to the operation of proportional fusion of multimodal carbon feature vectors, which can be implemented by adaptive weight allocation or attention weighted summation to eliminate redundant information between modalities and retain key features.

[0078] Specifically, the semantic factor matrix and the coefficient matrix are horizontally spliced through a fully connected layer to form a joint feature matrix. The rows of this matrix represent the basic features corresponding to the semantic factors, and the columns represent the weight distribution of each modality. In the cross-modal semantic association model, the rows of the semantic factor matrix are mapped to semantic nodes in the graph structure, and the columns of the coefficient matrix are mapped to modal nodes. The graph neural network performs cross-modal propagation operations in the graph structure, extracts features from the edges between semantic nodes and modal nodes through the graph convolution layer, and uses residual connections to avoid the gradient vanishing problem. After multiple layers of propagation, the feature vectors of each node are calculated with weight coefficients through the attention mechanism, and are finally aggregated according to the weight ratio to generate a semantic vector with cross-modal consistency.

[0079] Furthermore, the semantic factor matrix and the coefficient matrix are spliced together, and a cross-modal semantic association model is constructed, including: constructing a node representation, mapping the rows of the semantic factor matrix and the columns of the coefficient matrix into nodes; performing residual connections on the rows of the semantic factor matrix and the columns of the coefficient matrix based on a graph convolution operation, and using LayerNorm to normalize the connection results to form a cross-modal semantic association model.

[0080] Among them, node representation refers to mapping the rows and columns of the matrix into independent entities in the graph structure. Specifically, the row vectors and column vectors after matrix decomposition can be used as the initial features of the node, and the row and column indices can be converted into vector form through the embedding layer. The graph convolution operation refers to transferring node features based on the adjacency matrix. Specifically, it can be implemented by a multi-layer graph attention network, which enhances the feature expression ability by aggregating information from adjacent nodes. Residual connection refers to superimposing the features before and after convolution. Specifically, it can be implemented by a jump connection structure, which is used to alleviate the gradient vanishing problem in deep network training. LayerNorm normalization refers to standardizing the features. Specifically, it can be implemented by calculating the mean and variance along the feature dimension, which is used to stabilize the training process and accelerate convergence.

[0081] Specifically, the row vectors of the semantic factor matrix represent the latent features of different semantic components, while the column vectors of the coefficient matrix reflect the weight of each modality's contribution to the semantic component. After mapping the row and column vectors to graph nodes, bidirectional connections are constructed to form a heterogeneous information network. Graph convolutional layers iteratively update node states, integrating implicit connections between rows and columns during information propagation. Residual connections preserve the original matrix decomposition results, preventing high-order features from overwriting underlying semantic information. Normalization constrains the distribution range of node features, ensuring the stability of the cross-modal association model.

[0082] S280 uses a large model and AI framework structure to perform fusion reasoning on cross-modal semantic vectors to generate the final cross-modal semantic alignment results.

[0083] In this embodiment, when using federated learning to extract vectors, the semantic factor matrix and the coefficient matrix can be spliced and fused; the fused matrix is input into the association learning network; the graph neural network is used to propagate cross-modal information in the association learning network to generate an association feature vector; distance metric learning is performed on the association feature vector to output the aligned cross-modal semantic vector.

[0084] Among them, when generating the associated feature vector, the fused matrix is weighted aggregated based on the attention graph convolution; the weighted aggregation result is normalized; the normalized result is input into the multi-head attention module; and the final associated feature vector from the multi-head attention module is output.

[0085] In another embodiment, after completing the alignment, the method further includes: Model online fine-tuning: The alignment results are input into the online fine-tuning module to fine-tune the large model parameters based on the incremental learning strategy; Model evaluation: Evaluate the accuracy and recall of the fine-tuned large model on specific cross-modal tasks; Feedback optimization: inputting the evaluation results into the feedback optimization module to adjust the hyperparameters of the association learning network; Model deployment: Deploy the optimized model to the inference service layer of the AI framework structure.

[0086] In another embodiment, after the alignment is completed, the method further includes: Asynchronous batch processing: storing the intermediate results of steps S110 to S150 executed multiple times into the feature cache module; Cache reading: reading required intermediate results from the feature cache module in real time to accelerate projection mapping; Incremental update: regularly performing incremental updates on the projection representation in the feature cache module; Historical backtracking: Generate logs for the cache update process for backtracking and auditing.

[0087] In another embodiment, after step S110, the method further includes: Secure encryption: performing homomorphic encryption on the unified representation of the preprocessed multimodal carbon-related data; Privacy protection: the encryption state is maintained during the projection mapping and matrix factorization process; Secure decryption: After the association learning is completed, the alignment results are securely decrypted; Permission verification: performing access permission verification on the decrypted alignment result.

[0088] In another embodiment, before step S150, the method further includes: Feature enhancement: semantically enhancing the aligned cross-modal semantic vector based on a generative adversarial network; Discriminant training: The enhanced vector is input into the discriminant network and subjected to adversarial training with the true vector; Fusion adjustment: adjusting the large model fusion parameters according to the discriminant training results; Outputting the enhanced alignment result: taking the adjusted vector as the input of step S150.

[0089] In summary, this application enables deep semantic alignment of multimodal carbon-related data. In scenarios such as video content understanding and intelligent customer service conversations, it can accurately capture the correspondence between text descriptions and visual images, and simultaneously analyze the emotional tendencies of voice information. The system's modular design reduces computational complexity, improving processing efficiency while maintaining semantic consistency. It is particularly suitable for applications requiring real-time processing of multimodal carbon-related data.

[0090] Corresponding to the above method embodiment, see Figure 3 A schematic diagram of a cross-modal semantic alignment system is shown, which includes: a multimodal carbon-related data acquisition module 310, a projection mapping module 320, a matrix solution module 330, a federated learning module 340 and an alignment module 350.

[0091] The multimodal carbon-related data acquisition module 310 is used to extract text features, image features, and sound features from the input multimodal carbon-related data and perform standardization preprocessing to obtain multimodal carbon features represented in a unified format; The projection mapping module 320 is used to perform semantic mapping on text features, image features and sound features using a projection algorithm to obtain a projection representation in a low-dimensional public semantic space; The matrix solving module 330 is used to factorize the projection representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix; The federated learning module 340 is used to establish a cross-modal semantic association model based on an association learning algorithm and output an aligned cross-modal semantic vector; The alignment module 350 is used to use the large model and AI framework structure to perform fusion reasoning on the cross-modal semantic vectors to generate the final cross-modal semantic alignment result.

[0092] Among them, the multimodal carbon-related data acquisition module 310 refers to a device that separates text, image and sound features from multimodal carbon-related data through a feature extractor and performs standardization processing. Specifically, the BERT model can be used to extract text word embedding, ResNet can be used to extract image features, and Mel spectrum can be used to extract sound features, and the batch normalization layer can be used to achieve feature format unification.

[0093] The projection mapping module 320 refers to an algorithm component that maps high-dimensional features to a low-dimensional common space. Specifically, a linear transformation layer can be used in conjunction with a ReLU activation function and a self-attention mechanism to achieve cross-modal feature dimension compression.

[0094] The matrix solution module 330 refers to a calculation unit that extracts semantic factors through matrix decomposition. Specifically, a singular value decomposition algorithm can be used in conjunction with an L2 regularization constraint to achieve stable extraction of semantic factors.

[0095] The federated learning module 340 refers to a distributed training framework for constructing a cross-modal association model. Specifically, a graph neural network combined with residual connections and layer normalization operations can be used to achieve cross-modal feature interaction.

[0096] The alignment module 350 refers to the decision-making unit of fusion reasoning, which can specifically adopt the Transformer architecture in conjunction with the multi-head attention mechanism to realize the dynamic fusion of semantic vectors.

[0097] Specifically, the multimodal carbon-related data acquisition module 310 first extracts raw features from text, images, and audio using a pre-trained model. Bilinear interpolation is then used to spatially align image and text features. Batch normalization is then used to eliminate dimensional differences between modalities. The projection mapping module 320 performs linear dimensionality reduction on the normalized features, using a self-attention mechanism to capture global cross-modal dependencies and generate a compact representation in a low-dimensional semantic space. The matrix solution module 330 arranges the projected vectors into a semantic matrix in time series, extracts principal components through singular value decomposition, constructs a semantic factor matrix, and calculates the weight coefficients for each factor. The federated learning module 340 concatenates the semantic factors and coefficient matrix into graph-structured data. Graph convolutional layers are then used to iteratively update node representations, ultimately outputting a cross-modally aligned semantic vector. After receiving the semantic vector, the alignment module 350 uses a Transformer-based fusion network for multimodal reasoning to generate the final semantic alignment result.

[0098] In this embodiment, the multimodal carbon-related data acquisition module 310 is specifically used to: The feature extractor extracts text features, image features, and sound features from the input multimodal carbon-related data, performs word segmentation and part-of-speech tagging on the text corresponding to the text features, and extracts context-sensitive word embedding features through a pre-trained large model; After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain a mapping result; Performing batch normalization on the mapping results to obtain normalized results, wherein the normalization is to balance the norms of the features of each modality; The normalized results are input into an encoder to generate multimodal carbon features represented in a unified format.

[0099] In this embodiment, the multimodal carbon-related data acquisition module 310 is specifically used to: One-to-one correspondence between the image features and the word embedding features according to spatial coordinates and semantic labels; According to the corresponding relationship, coordinates of four adjacent pixels are identified in the image area of the image feature; Calculate the horizontal and vertical distance weights between the coordinates of the four pixel points and the target word embedding position in the image feature; Weighting the characteristic values of the four pixel points proportionally according to the horizontal and vertical distance weights; The weighted results are accumulated to form the single-point image features after bilinear interpolation; Perform L2 norm normalization on the single-point image feature to obtain a mapping result.

[0100] In practical applications, the batch normalization layer of the multimodal carbon-related data acquisition module 310 can be configured to dynamically adjust the L2 norm threshold of each modal feature, for example, limiting the norm of image features to the range of 0-1 and the norm of text features to the range of 1-2. The number of self-attention heads in the projection mapping module 320 can be set to 8, each with a dimension of 64. The singular value truncation parameter P in the matrix solution module 330 can be dynamically calculated based on the input data dimension, for example, set to 20% of the total dimension. The graph convolution layer of the federated learning module can adopt a gated recurrent unit structure, and the number of iterations can be set to 3.

[0101] In this embodiment, the projection mapping module 320 is specifically used to: Performing linear transformation on the text feature, the image feature, and the sound feature to generate an initial projection vector; Performing nonlinear mapping on the initial projection vector through a ReLU activation function to obtain a ReLU vector; Using batch normalization, the ReLU vector is scaled and translated to obtain a projection vector; A self-attention mechanism is used to extract the global dependency in the projection vector to obtain a projection representation in a low-dimensional common semantic space.

[0102] In this embodiment, the projection mapping module 320 is specifically used to: extracting a query vector, a key vector, and a value vector of the projected vector; Calculating attention weights of the query vector, the key vector, and the value vector by scaling the dot product; Softmax normalization is performed on the attention weights, and based on the normalized weights, the query vector, the key vector, and the value vector are multiplied and accumulated to obtain a projection representation in a low-dimensional common semantic space.

[0103] In this embodiment, the matrix solving module 330 is specifically used for: Arranging the projection vectors in the projection representation into a semantic matrix according to time sequence or spatial position; Adding an L2 regularization term to the semantic matrix and extracting singular values and singular vectors therein using a singular value decomposition method; The singular vectors corresponding to the first P singular values are selected to construct a semantic factor matrix, and the weight of each element in the semantic factor matrix is calculated and extracted, and a coefficient matrix is constructed based on the weight, where P is an integer greater than 1.

[0104] In this embodiment, the federated learning module 340 is specifically used to: splicing the semantic factor matrix and the coefficient matrix, and constructing a cross-modal semantic association model; The feature vectors are extracted by cross-modal information propagation through graph neural networks, and the extracted feature vectors are weighted and aggregated to obtain the aligned cross-modal semantic vectors.

[0105] In this embodiment, the federated learning module 340 is specifically used to: Constructing a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix into nodes; Based on the graph convolution operation, the rows of the semantic factor matrix and the columns of the coefficient matrix are residually connected, and the connected results are normalized using LayerNorm to form a cross-modal semantic association model.

[0106] In summary, this system realizes multimodal carbon feature space alignment in the data acquisition stage through multi-module collaborative processing, introduces the attention mechanism to capture cross-modal associations in the projection stage, improves the stability of semantic factors through regularization constraints in the matrix decomposition stage, and constructs a graph structure in the federated learning stage to realize cross-modal information propagation, ultimately forming an end-to-end semantic alignment framework.

[0107] Through phased and progressive feature processing, the system effectively addresses the semantic gap between modalities and achieves deep semantic alignment of multimodal carbon-related data. This allows for accurate correlation between textual descriptions and visual images, and simultaneous analysis of the emotional undertones of spoken information in scenarios such as video content understanding and intelligent customer service conversations. The system's modular design reduces computational complexity, improving processing efficiency while maintaining semantic consistency. This makes it particularly suitable for applications requiring real-time processing of multimodal carbon-related data.

[0108] The present application further proposes a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement a cross-modal semantic alignment method, which includes extracting text, image and sound features from multimodal carbon-related data and performing standardized preprocessing, generating a projection representation of a low-dimensional public semantic space through a projection algorithm, decomposing it into a semantic factor matrix and a coefficient matrix using a matrix decomposition algorithm, establishing a cross-modal semantic association model based on an association learning algorithm and outputting the aligned semantic vector, and finally generating a semantic alignment result through fusion reasoning of a large model and an AI framework structure.

[0109] Among them, computer-readable storage media refers to a physical carrier that can persist data and support processor reading. Specifically, it can be implemented in solid-state drives, flash memory chips or optical disks. Its role is to provide a stable and reliable instruction storage environment for algorithm execution. Computer executable instructions refer to a set of codes written in a programming language and can be compiled and executed. Specifically, it can be implemented in Python, C++ or Java. Its role is to convert the cross-modal semantic alignment method into an operation process that can be recognized by hardware. A processor refers to an integrated circuit that performs arithmetic and logical operations. Specifically, it can be implemented in a multi-core CPU, GPU or TPU architecture. Its role is to accelerate feature extraction, matrix decomposition and model training processes through parallel computing.

[0110] Specifically, when the computer executable instructions in the storage medium are loaded into the processor, the processor performs multimodal carbon feature extraction, semantic space projection, matrix decomposition and model building operations in sequence according to preset steps. For example, in the feature extraction stage, the processor calls the pre-trained large model to perform word segmentation and part-of-speech tagging on the text, while using convolutional neural networks to extract image features and Fourier transform to extract sound spectrum features. In the projection stage, the processor performs linear transformation and nonlinear activation on the multimodal carbon features, and combines the self-attention mechanism to capture global semantic dependencies. In the matrix decomposition stage, the processor uses the singular value decomposition algorithm to extract the semantic factor matrix from the projection representation, and optimizes the stability of the decomposition result through regularization constraints. Finally, the processor inputs the semantic factor matrix and the coefficient matrix into the graph neural network, uses the cross-modal information propagation mechanism to generate aligned semantic vectors, and performs multimodal reasoning fusion through the large model.

[0111] Through the above technical solution, this application solves the problem that traditional storage media cannot support end-to-end cross-modal semantic alignment, and realizes the full-process automation of multimodal carbon feature extraction, mapping, and fusion. This solution significantly improves the computational efficiency of cross-modal semantic alignment by optimizing the collaborative mechanism of instruction sets and processors. At the same time, it uses matrix decomposition and graph neural networks to ensure the accuracy of semantic associations. It is particularly suitable for multimodal carbon-related data processing needs in weakly supervised scenarios.

[0112] This embodiment further provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-mentioned cross-modal semantic alignment method. The electronic device can be a server or a terminal device.

[0113] See also Figure 4 As shown, the electronic device includes a processor 400 and a memory 401 , wherein the memory 401 stores machine executable instructions that can be executed by the processor 400 , and the processor 400 executes the machine executable instructions to implement the above-mentioned cross-modal semantic alignment method.

[0114] Furthermore, Figure 4 The electronic device shown further includes a bus 402 and a communication interface 403 , and the processor 400 , the communication interface 403 and the memory 401 are connected via the bus 402 .

[0115] The memory 401 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is achieved through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 402 may be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0116] The processor 400 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 400 or by instructions in the form of software. The above-mentioned processor 400 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory 401. Processor 400 reads information from memory 401 and, in conjunction with its hardware, completes the steps of the cross-modal semantic alignment method of the aforementioned embodiment.

[0117] This embodiment also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned cross-modal semantic alignment method.

[0118] The computer program product of the cross-modal semantic alignment method, device, electronic device and storage medium provided in the embodiments of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In addition, in the description of the embodiments of this application, unless otherwise specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0121] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0122] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of this application and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0123] Finally, it should be noted that the above embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technical field can still modify the technical solutions described in the above embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A cross-modal semantic alignment method, characterized in that: The method comprises: Extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features represented in a unified format; Performing semantic mapping on the text features, the image features, and the sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space; Factoring the projected representation using a matrix decomposition algorithm to obtain a semantic factor matrix and its coefficient matrix; Based on the association learning algorithm, a cross-modal semantic association model is established based on the semantic factor matrix and the coefficient matrix, and an aligned cross-modal semantic vector is output; The cross-modal semantic vectors are fused and inferred using a large model and AI framework structure to generate the final cross-modal semantic alignment result.

2. The cross-modal semantic alignment method according to claim 1, characterized in that: The method extracts text features, image features, and sound features from the input multimodal carbon-related data and performs standardization preprocessing to obtain multimodal carbon features represented in a unified format, including: The feature extractor extracts text features, image features, and sound features from the input multimodal carbon-related data, performs word segmentation and part-of-speech tagging on the text corresponding to the text features, and extracts context-sensitive word embedding features through a pre-trained large model; After aligning the word embedding features with the image features, a bilinear interpolation method is used to map the image features to the word embedding features to obtain a mapping result; Performing batch normalization on the mapping results to obtain normalized results, wherein the normalization is to balance the norms of the features of each modality; The normalized results are input into an encoder to generate multimodal carbon features represented in a unified format.

3. The cross-modal semantic alignment method according to claim 2, characterized in that: After aligning the word embedding features with the image features, the image features are mapped to the word embedding features using a bilinear interpolation method to obtain a mapping result, including: One-to-one correspondence between the image features and the word embedding features according to spatial coordinates and semantic labels; According to the corresponding relationship, coordinates of four adjacent pixel points are identified in the image area of the image feature; Calculate the horizontal and vertical distance weights between the coordinates of the four pixel points and the target word embedding position in the image feature; Weighting the characteristic values of the four pixel points proportionally with the horizontal and vertical distance weights; The weighted results are accumulated to form the single-point image features after bilinear interpolation; Perform L2 norm normalization on the single-point image feature to obtain a mapping result.

4. The cross-modal semantic alignment method according to claim 1, characterized in that: The method of performing semantic mapping on the text features, the image features, and the sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space includes: Performing linear transformation on the text feature, the image feature, and the sound feature to generate an initial projection vector; Performing nonlinear mapping on the initial projection vector through a ReLU activation function to obtain a ReLU vector; Using batch normalization, the ReLU vector is scaled and translated to obtain a projection vector; A self-attention mechanism is used to extract the global dependency in the projection vector to obtain a projection representation in a low-dimensional common semantic space.

5. The cross-modal semantic alignment method according to claim 4, characterized in that: The method of extracting the global dependency in the projection vector by using the self-attention mechanism to obtain the projection representation in the low-dimensional common semantic space includes: extracting a query vector, a key vector, and a value vector of the projected vector; Calculating attention weights of the query vector, the key vector, and the value vector by scaling the dot product; Softmax normalization is performed on the attention weights, and based on the normalized weights, the query vector, the key vector, and the value vector are multiplied and accumulated to obtain a projection representation in a low-dimensional common semantic space.

6. The cross-modal semantic alignment method according to claim 1, characterized in that: The projected representation is factored using a matrix decomposition algorithm to obtain a semantic factor matrix and a coefficient matrix thereof, including: Arranging the projection vectors in the projection representation into a semantic matrix according to time sequence or spatial position; Adding an L2 regularization term to the semantic matrix and extracting singular values and singular vectors therein using a singular value decomposition method; The singular vectors corresponding to the first P singular values are selected to construct a semantic factor matrix, and the weight of each element in the semantic factor matrix is calculated and extracted, and a coefficient matrix is constructed based on the weight, where P is an integer greater than 1.

7. The cross-modal semantic alignment method according to claim 1, characterized in that: The association learning algorithm is based on the semantic factor matrix and the coefficient matrix to establish a cross-modal semantic association model, and output an aligned cross-modal semantic vector, including: splicing the semantic factor matrix and the coefficient matrix, and constructing a cross-modal semantic association model; The feature vectors are extracted by cross-modal information propagation through graph neural networks, and the extracted feature vectors are weighted and aggregated to obtain the aligned cross-modal semantic vectors.

8. The method according to claim 7, characterized in that: The step of concatenating the semantic factor matrix and the coefficient matrix and constructing a cross-modal semantic association model includes: Constructing a node representation by mapping the rows of the semantic factor matrix and the columns of the coefficient matrix into nodes; Based on the graph convolution operation, the rows of the semantic factor matrix and the columns of the coefficient matrix are residually connected, and the connected results are normalized using LayerNorm to form a cross-modal semantic association model.

9. A cross-modal semantic alignment system, characterized by: The system comprises: The multimodal data acquisition module is used to extract text features, image features, and sound features from the input multimodal carbon-related data, and perform standardized preprocessing to obtain multimodal carbon features expressed in a unified format; A projection mapping module, configured to perform semantic mapping on the text features, the image features, and the sound features using a projection algorithm to obtain a projection representation in a low-dimensional common semantic space; A matrix solving module, configured to factorize the projection representation using a matrix decomposition algorithm to obtain a semantic factor matrix and a coefficient matrix thereof; a federated learning module for establishing a cross-modal semantic association model based on the semantic factor matrix and the coefficient matrix based on an association learning algorithm, and outputting an aligned cross-modal semantic vector; The alignment module is used to use a large model and an AI framework structure to perform fusion reasoning on the cross-modal semantic vectors to generate a final cross-modal semantic alignment result.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the cross-modal semantic alignment method according to any one of claims 1 to 8.

Images