Multi-modal AI data fusion processing method and device, equipment and medium

Abstract

The invention relates to a multi-modal AI data fusion processing method, device and equipment and a medium, and the method comprises the steps: firstly extracting visual, auditory and text modal features through a pre-training encoder, executing dimension alignment, and generating a standard data feature set with unified dimensions; a cross-modal semantic graph is constructed based on a cosine similarity algorithm, and the problem of semantic mismatch of heterogeneous data is solved; residual enhancement is carried out on the map nodes, and noise interference is eliminated; fusing the optimized features and the semantic topology in combination with a graph convolutional network to generate aggregation graph representation; the fusion features are mapped to a low-dimensional semantic space through a variational auto-encoder, and cross-modal correlation essence is captured; the key dimension contribution degree is quantified, a visual report is generated, and semantic association rules among modals are disclosed, so that the dimension isomerism limitation of a traditional fusion technology is broken through, quantifiable cross-modal semantic mapping is established, the whole process traceability from feature fusion to decision interpretation is realized, and the method is suitable for popularization and application. And the multi-modal decision black box problem in the fields of medical diagnosis, automatic driving and the like is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence data processing technology, and specifically to a multimodal AI data fusion processing method, device, equipment and medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, an increasing number of application scenarios require the simultaneous processing of data from multiple modalities, including visual, auditory, and textual data. Multimodal AI data fusion processing aims to transform diverse data, such as images, sounds, and natural language, into a unified semantic representation, thereby supporting more comprehensive and in-depth intelligent analysis and decision-making. This approach has shown great potential in areas such as intelligent assistants, autonomous driving systems, and medical diagnostic platforms. It can fuse the spatial features of images, the temporal characteristics of sounds, and the logical structure of language into a unified semantic representation, enabling machines to understand complex environments in a multidimensional way, similar to humans.

[0003] However, existing technologies face three fundamental challenges in the process of multimodal fusion. First, the dimensional heterogeneity between modalities makes the feature space difficult to be compatible - high-dimensional pixel matrices, temporal acoustic features and discrete text symbols have essential differences in the data structure level. Traditional feature splicing or weighted fusion methods will destroy modal specificity. Secondly, the semantic mismatch problem seriously restricts the fusion effect. The feature space of different modalities describing the same object lacks continuity. The existing similarity calculation method cannot establish quantifiable cross-modal semantic associations. The most critical pain point is the lack of explainability of the decision-making process. Users cannot trace the contribution of each modal data to the final decision, resulting in a "trust crisis" in high-risk scenarios such as medical diagnosis and autonomous driving. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a multimodal AI data fusion processing method, device, equipment and medium that can solve the problems of dimensional heterogeneity, semantic mismatch and decision black box, and realize explainable cross-modal fusion.

[0005] The purpose of the present invention is achieved by the following scheme:

[0006] In a first aspect, the present invention provides a multimodal AI data fusion processing method, comprising the following steps:

[0007] S1: Perform feature extraction and dimension alignment on the input raw visual data, raw auditory data, and raw text data. Pre-trained encoders are used to extract visual features, auditory time series vectors, and text semantic vectors. These features are then fused using spatial pooling to compress the unified dimensional features and generate a standard data feature set with unified dimensionality.

[0008] S2: Process the standard data feature set based on the cosine similarity algorithm, analyze the correlation strength between feature vectors, and perform dynamic threshold screening on the calculation results to obtain a cross-modal semantic map;

[0009] S3: Perform multi-head attention weight allocation on the cross-modal semantic graph, calculate the correlation between nodes and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference;

[0010] S4: Graph convolution is performed on the optimized feature set and the cross-modal semantic graph. Node information is propagated based on the adjacency matrix and the associated edge weights are threshold pruned to obtain an aggregated graph representation of the fused cross-modal features.

[0011] S5: Perform latent space mapping on the aggregate graph representation, learn semantic distribution features through variational autoencoders, and perform convergence verification on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal associations;

[0012] S6: Quantify the contribution of the low-dimensional semantic space, identify the key dimensions with the highest decision weight, and perform dimensionality reduction visualization on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output the semantic association rules and key dimension contribution distribution between cross-modal data.

[0013] In a second aspect, the present invention provides a multimodal AI data fusion processing device, which is configured with the following modules:

[0014] The feature extraction preprocessing module is used to perform feature extraction and dimension alignment on the input raw visual data, raw auditory data, and raw text data. It extracts visual features, auditory time series vectors, and text semantic vectors through a pre-trained encoder, and fuses spatial pooling compression to unify the dimensional features to generate a standard data feature set with unified dimensionality.

[0015] The association graph construction module is used to process the standard data feature set based on the cosine similarity algorithm, analyze the correlation strength between feature vectors, and perform dynamic threshold screening on the calculation results to obtain a cross-modal semantic graph;

[0016] The feature optimization and enhancement module is used to assign multi-head attention weights to the cross-modal semantic graph, calculate the correlation between nodes, and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference;

[0017] The graph convolution fusion module is used to perform graph convolution fusion processing on the optimized feature set and the cross-modal semantic graph. It propagates node information based on the adjacency matrix and performs threshold pruning on the associated edge weights to obtain an aggregated graph representation of the fused cross-modal features.

[0018] The latent space mapping module is used to perform latent space mapping on the aggregate graph representation. It learns semantic distribution features through a variational autoencoder and performs convergence verification on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal associations.

[0019] The semantic association analysis module is used to quantify the contribution of low-dimensional semantic space, identify the key dimensions with the highest decision weight, and perform dimensionality reduction visualization on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output the semantic association rules and key dimension contribution distribution between cross-modal data.

[0020] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned multimodal AI data fusion processing methods when executing the computer program.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned multimodal AI data fusion processing methods.

[0022] To sum up, the multimodal AI data fusion processing method provided in this application can effectively solve the problems of multimodal data dimensional heterogeneity and noise interference in the AI ​​data fusion processing process through systematic technical collaboration. Specifically, the system unifies the feature representations of vision, hearing, and text through pre-trained encoders and spatial pooling compression technology, eliminates the structural differences between the original data sources, and realizes adaptive dimensional alignment of multimodal data; uses cosine similarity calculation and dynamic threshold screening to construct a semantic graph with precise mapping, achieves the quantitative generation of cross-modal semantic associations, and effectively captures the inter-translatable relationships between different modalities; through multi-head attention weight allocation combined with residual enhancement mechanism, the representation purity of key features is significantly improved, thereby realizing adaptive suppression of feature noise; graph convolution and edge weight pruning based on the adjacency matrix ensure the stable association of multi-source data in the topological space and establish a robust cross-modal topological fusion architecture; uses variational autoencoders to learn potential distribution features and verify them through reconstruction errors, revealing the essential laws of cross-modal associations and realizing interpretable mapping of low-dimensional semantic space; outputs semantic association analysis reports through contribution quantification and dimensionality reduction projection, generates visual decision-making basis for key dimensions, and provides accurate cross-modal association rule identification and dimensional contribution evaluation for high-reliability scenarios such as autonomous driving and medical diagnosis.

[0023] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a multimodal AI data fusion processing method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a process for generating a cross-modal semantic graph describing multimodal semantic associations provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a process for generating a semantic association analysis report according to an embodiment of the present application;

[0027] Figure 4 A schematic structural diagram of a multimodal AI data fusion processing device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0028] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0030] In one embodiment, Figure 1 As shown, a multimodal AI data fusion processing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0031] S1: Perform feature extraction and dimension alignment on the input raw visual data, raw auditory data, and raw text data. Use the pre-trained encoder to extract visual features, auditory time series vectors, and text semantic vectors, and fuse them with spatial pooling compression to generate a standard data feature set with a unified dimension.

[0032] Specifically, the system receives raw visual data, raw auditory data, and raw text data from different data streams. For visual data, the system's pre-trained encoder, based on a deep convolutional neural network, analyzes the image's pixel matrix layer by layer, extracting basic features such as edges and color transitions at the bottom layer, local shape contours at the middle layer, and finally semantic concept features at the higher layer. This process constructs a high-dimensional visual feature tensor, whose dimensions may include information such as the height, width, and number of channels of feature maps at different levels. For auditory data, the system uses a pre-trained encoder based on time series processing, such as a Long Short-Term Memory Network (LSTM) or a Gated Recurrent Unit (GRU) structure, to perform sequence modeling on the sound signal. This decomposes the continuous sound wave signal into a series of acoustic feature vectors corresponding to time steps, capturing temporal variations and generating auditory time series vectors that incorporate temporal dependencies. The dimensions cover both the length of the time series and the feature dimensions of each time step. For text data input, the system uses a pre-trained language model, starting with word embeddings and combining context for semantic encoding. The output is a text semantic vector, whose dimensions reflect lexical distribution characteristics and semantic combination characteristics. Due to the natural differences in the feature dimensions of data of different modalities, the system performs spatial pooling operations to compress and align the dimensions. Spatial pooling moves on the spatial dimension of the visual feature tensor according to the set pooling window size and step size, aggregates local area features through maximum pooling or average pooling functions, reduces spatial resolution, and retains key feature information. For auditory time series vectors, the system adopts a time series pooling strategy to aggregate features in segments according to the time dimension and compress the length of the time series. The text semantic vector performs appropriate feature compression according to the semantic level and maps it to a unified dimensional space through linear transformation or nonlinear mapping. After the above processing, the system finally generates a standard data feature set of unified dimensions, preparing for subsequent fusion processing.

[0033] S2: Process the standard data feature set based on the cosine similarity algorithm, analyze the correlation strength between feature vectors, and perform dynamic threshold screening on the calculation results to obtain a cross-modal semantic map.

[0034] Specifically, cosine similarity is a commonly used metric used to measure the degree of directional similarity between two vectors. Its value ranges from -1 to 1, with values ​​closer to 1 indicating greater directional similarity between the vectors. The system calculates the cosine similarity between eigenvectors of different modalities to analyze the strength of their association.

[0035] Specifically, for each pair of feature vectors in the feature set, the system calculates their dot product, then calculates the modulus of each vector separately. Finally, the dot product is divided by the product of the moduli to obtain the cosine similarity value. After obtaining the cosine similarity value between the feature vectors, the system applies a dynamic threshold screening process. The dynamic threshold setting is not fixed but adaptively adjusted based on the distribution characteristics of the current dataset and the specific requirements of the fusion task. The system determines a reasonable screening threshold by analyzing the distribution statistics of the similarity values, such as the mean and standard deviation. Feature vector pairs corresponding to similarity values ​​below this threshold are considered to have a weak correlation and may negatively impact the subsequent fusion results, so they are filtered out by the system. The feature vector pairs that remain have a higher correlation strength and can better reflect the semantic relationship between data from different modalities. Through this process, the system constructs a cross-modal semantic graph, where nodes represent feature vectors and edges represent the correlation strength between feature vectors, namely the cosine similarity value.

[0036] S3: Perform multi-head attention weight allocation on the cross-modal semantic graph, calculate the correlation between nodes and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference.

[0037] Specifically, the system introduces a multi-head attention mechanism to assign weights to nodes in the graph. The multi-head attention mechanism is implemented within the system through parallel operations of multiple independent attention heads. Each attention head performs a linear transformation on the node features and calculates the attention weights between pairs of nodes. These weights are obtained through the scaled dot product attention mechanism, that is, the dot product operation is performed on the transformed features and then scaled. Finally, the weights are normalized between 0 and 1 through the softmax function to represent the strength of the association between nodes. The system captures the potential complex associations between nodes from multiple perspectives. Different attention heads focus on different subspaces of the feature space, which helps to mine more comprehensive semantic information. After obtaining the multi-head attention weights, the system focuses on high-weight nodes. This is because high-weight nodes often carry more important semantic information, while low-weight nodes may contain more interference information.

[0038] After obtaining the correlation between nodes, the system performs residual enhancement on high-weight nodes. Specifically, for the feature vectors corresponding to high-weight nodes, the system adds their original feature vectors to the feature vectors updated by the attention mechanism to obtain enhanced feature vectors. This residual connection method can effectively alleviate the gradient vanishing problem in deep networks and ensure that the features of high-weight nodes can be fully retained and utilized in subsequent processing. Through this optimization process, the system obtains an optimized feature set that has been enhanced with residuals. While retaining the original feature information, this feature set further highlights the feature expressiveness of key nodes, enhances the robustness and semantic expression capabilities of the features, and provides a higher-quality data foundation for subsequent fusion processing.

[0039] S4: Perform graph convolution fusion processing on the optimized feature set and the cross-modal semantic graph, propagate node information based on the adjacency matrix and perform threshold pruning on the associated edge weights to obtain an aggregated graph representation of the fused cross-modal features.

[0040] Specifically, the system combines an optimized feature set with a cross-modal semantic graph, using a graph convolutional network (GCN) to perform feature fusion and information propagation. The GCN performs convolution operations on graph-structured data within the system. Its core lies in the adjacency matrix, which stores the connection relationships between nodes and the weights of associated edges. The system propagates node information across the graph based on the weight information stored in the adjacency matrix. Specifically, for each node, the system performs a weighted aggregation of its features with those of adjacent nodes, with the weights determined by the corresponding edge weights in the adjacency matrix. This process enables the mutual transfer and fusion of features between nodes, so that each node's features not only contain its own information but also incorporate the semantic features of surrounding nodes, resulting in an updated feature representation. For example, around a visual-textual association node, through information propagation, the visual node can integrate the semantic information of the text node, and the text node can also absorb the spatial feature information of the visual node, achieving deep interaction of cross-modal features.

[0041] During the information dissemination process, in order to avoid overfitting and highlight key semantic associations, the system performs threshold pruning on the weights of associated edges. The system pre-sets a weight threshold, which is determined based on the semantic association strength requirements and data distribution characteristics in the actual application scenario. Associations corresponding to edge weights below the threshold are considered to be weaker connections that may contribute less to semantic understanding, and the system removes them. After retaining high-weight edges, the system constructs a more compact and efficient feature fusion architecture, so that the fused features can more accurately reflect the comprehensive semantic features and inherent associations of multimodal data. Ultimately, the system obtains an aggregate graph representation of the fused cross-modal features. The aggregate graph representation highlights key semantic associations while maintaining semantic integrity, providing strong support for subsequent deeper semantic analysis and decision-making.

[0042] S5: Perform latent space mapping on the aggregate graph representation, learn semantic distribution features through variational autoencoders, and perform convergence verification on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal associations.

[0043] Specifically, the system uses the aggregate graph representation as input data, initiates the latent space mapping and semantic distribution learning process, and uses a generative model called a variational autoencoder (VAE) for processing. The variational autoencoder consists of two parts within the system: an encoder and a decoder. The encoder is responsible for mapping the input aggregate graph representation to the latent space. It transforms the data features through a multi-layer neural network, gradually extracts the semantic features of the data, and outputs the mean vector and variance vector in the latent space. These two vectors define a multivariate Gaussian distribution, thereby modeling the latent distribution of the data. The decoder samples from the latent space and restores the sampled latent feature vectors to the original data features. Its network structure echoes that of the encoder. Through gradual upsampling and feature reconstruction operations, it generates reconstructed data with a similar structure to the input data.

[0044] During training, the system rigorously monitors and verifies the convergence of the encoding-decoding reconstruction error. Reconstruction error typically uses metrics such as mean squared error or cross-entropy loss to measure the difference between the input data and the reconstructed data. The system calculates the loss gradient via a backpropagation algorithm and uses an optimization algorithm to update the network parameters of the encoder and decoder. As training progresses, the system continuously verifies whether the reconstruction error meets the preset convergence criteria, namely, whether the error stabilizes within a certain range and no longer decreases significantly. When the convergence criteria are met, the system concludes that the variational autoencoder has successfully learned the semantic distribution characteristics of the data and can accurately restore the original data features from the latent space while capturing the inherent semantic distribution patterns of the data. Ultimately, the system obtains a low-dimensional semantic space that reflects the essence of cross-modal associations, achieving efficient semantic abstraction and representation of complex multimodal data, and providing concise and expressive feature representations for further semantic analysis and decision-making.

[0045] S6: Quantify the contribution of the low-dimensional semantic space, identify the key dimensions with the highest decision weight, and perform dimensionality reduction visualization on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output the semantic association rules and key dimension contribution distribution between cross-modal data.

[0046] Specifically, the system performs feature analysis on the low-dimensional semantic space and uses feature importance evaluation algorithms, such as gradient-based feature importance analysis or permutation-based importance evaluation methods, to identify the key dimensions with the highest weight in the decision-making process. In gradient-based feature importance analysis, the system calculates the gradient of each feature dimension with respect to the model output. The magnitude of the gradient reflects the sensitivity of the feature to the output. Feature dimensions with larger absolute values ​​of the gradient are considered to be key dimensions and play a core role in decision-making. In permutation-based importance evaluation methods, the system observes changes in model performance by randomly permuting the value of a feature dimension. If the performance drops significantly, it indicates that the feature dimension has made an important contribution to the decision. Through these methods, the system accurately locates the feature components that play a key role in semantic association and the final decision.

[0047] At the same time, to present complex semantic associations more intuitively, the system can use dimensionality reduction visualization techniques, such as t-SNE or PCA, to map high-dimensional clustering relationships into two-dimensional or three-dimensional space for visualization. The t-SNE algorithm calculates the conditional probability distribution between high-dimensional data within the system and constructs a similar probability distribution in low-dimensional space. It uses gradient descent to minimize the KL divergence between the two distributions, thereby achieving nonlinear dimensionality reduction of the data. This method can better preserve the local structural characteristics of the data, allowing feature points with similar semantics to cluster together in low-dimensional space, forming an intuitive clustering effect. The PCA method calculates the covariance matrix of the data, solves its eigenvalues ​​and eigenvectors, selects the eigenvectors corresponding to the largest eigenvalues ​​as the projection direction, and projects the high-dimensional data into low-dimensional space, highlighting the main change trends and characteristic distribution of the data.

[0048] After feature analysis and dimensionality reduction visualization, the system generates a semantic association analysis report. This report details the semantic association rules between cross-modal data, including how different modal data relate to and influence each other at the semantic level, as well as the distribution of key dimension contributions—that is, the weight and specific contribution of each key feature dimension in the semantic association establishment and decision-making process. Presented in the form of intuitive charts, data statistics, and detailed text descriptions, the semantic association analysis report provides a strong basis for users to understand the multimodal data fusion decision-making process, helping them make accurate and reliable intelligent decisions based on multimodal data in practical applications, and improving the transparency and trustworthiness of the system.

[0049] To sum up, the multimodal AI data fusion processing method provided in this application can effectively solve the problems of multimodal data dimensional heterogeneity and noise interference in the AI ​​data fusion processing process through systematic technical collaboration. Specifically, the system unifies the feature representations of vision, hearing, and text through pre-trained encoders and spatial pooling compression technology, eliminates the structural differences between the original data sources, and realizes adaptive dimensional alignment of multimodal data; uses cosine similarity calculation and dynamic threshold screening to construct a semantic graph with precise mapping, achieves the quantitative generation of cross-modal semantic associations, and effectively captures the inter-translatable relationships between different modalities; through multi-head attention weight allocation combined with residual enhancement mechanism, the representation purity of key features is significantly improved, thereby realizing adaptive suppression of feature noise; graph convolution and edge weight pruning based on the adjacency matrix ensure the stable association of multi-source data in the topological space and establish a robust cross-modal topological fusion architecture; uses variational autoencoders to learn potential distribution features and verify them through reconstruction errors, revealing the essential laws of cross-modal associations and realizing interpretable mapping of low-dimensional semantic space; outputs semantic association analysis reports through contribution quantification and dimensionality reduction projection, generates visual decision-making basis for key dimensions, and provides accurate cross-modal association rule identification and dimensional contribution evaluation for high-reliability scenarios such as autonomous driving and medical diagnosis.

[0050] In one embodiment, S1 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0051] S11: Perform spatial feature extraction on the input raw visual data, call the Vision Transformer model to hierarchically analyze the local information of the image, and generate a visual feature matrix containing spatial structural features.

[0052] Specifically, the system calls the Vision Transformer (ViT) model, which can effectively parse local information in images in a hierarchical manner. Specifically, the ViT model divides the input raw visual data into multiple fixed-size image blocks, and then converts these image blocks into a one-dimensional vector sequence. Through multi-layer Transformer encoders, the system is able to capture the global spatial relationship and local feature details between image blocks. Each layer of Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network, thereby gradually extracting high-level spatial structural features of the image. After a series of transformations and calculations, the system finally generates a visual feature matrix containing rich spatial structural features. This matrix not only retains the local details of the image, but also reflects the overall spatial layout information of the image, laying the foundation for subsequent multimodal fusion processing.

[0053] S12: Based on the bidirectional long short-term memory network, the input raw auditory data is temporally encoded to extract the temporal evolution law of acoustic features and generate a fixed-dimensional auditory time series vector.

[0054] Specifically, the system performs temporal encoding based on a bidirectional long short-term memory (Bi-LSTM) network. The Bi-LSTM consists of two LSTM modules, forward and backward. The forward module processes acoustic features sequentially according to the chronological order of the raw auditory input data, capturing temporal evolution from the past to the present. The backward module processes the features in reverse chronological order, capturing temporal contextual information from the future to the present. The system converts the raw auditory data into an acoustic feature sequence and then inputs it into the Bi-LSTM. At each time step, the forward and backward modules output hidden state vectors, which are concatenated to form a feature representation that contains the complete temporal context. After processing all time steps, the system performs further aggregation operations on the output feature sequence, such as averaging or maximum pooling, to ultimately generate a fixed-dimensional auditory time series vector. This vector not only preserves the temporal evolution of the acoustic features but also incorporates past and future temporal contextual associations, providing a compact and expressive feature representation of the auditory data for subsequent fusion processing.

[0055] S13: Perform semantic aggregation processing on the input raw text data, integrate the contextual association characteristics of language elements, and generate text semantic vectors that represent logical relationships.

[0056] Specifically, the system preprocesses the input raw text data, including operations such as word segmentation and stop word removal, and converts the text into a vocabulary sequence. Then, the pre-trained word embedding model is used to map the vocabulary to a high-dimensional semantic space to generate a vocabulary embedding vector sequence. Next, the system uses context-aware neural network structures, such as bidirectional gated recurrent units (Bi-GRU) or Transformer encoders, to process the vocabulary embedding vector sequence. These network structures can capture the front-end dependencies and contextual information between words, thereby achieving semantic aggregation. Finally, the system generates a text semantic vector that represents the logical relationship of the text through global pooling operations or taking feature vectors at specific positions in the sequence. This vector not only reflects the core semantic information of the text content, but also reflects the logical association and contextual dependency between words, laying a text semantic foundation for subsequent multimodal fusion processing.

[0057] S14: Perform adaptive spatial pooling on the visual feature matrix, combine the auditory time series vector and text semantic vector for multimodal feature fusion, and generate a standard data feature set of unified dimension.

[0058] Specifically, the system applies adaptive spatial pooling to the visual feature matrix. Adaptive spatial pooling dynamically adjusts the pooling window size and stride based on the visual feature matrix's content to retain the most important spatial feature information while aligning its dimensionality with the auditory temporal vectors and textual semantic vectors. The system defines a target dimension and uses pooling operations to gradually compress the spatial dimensionality of the visual feature matrix to this target dimension, for example, by employing max pooling or average pooling strategies to aggregate feature values ​​in local regions. During the pooling process, the system continuously evaluates the pooling effect to ensure that key spatial features are preserved. Once the dimensionality of the visual feature matrix is ​​aligned with that of the auditory temporal vectors and textual semantic vectors, the system fuses the feature data from these three modalities. This fusion operation can be achieved through feature concatenation, element-wise weighted summation, or a learned fusion network, generating a standardized data feature set with unified dimensionality. This feature set integrates the spatial structural information of vision, the temporal evolution of auditory perception, and the logical semantic content of text, forming a compact representation that comprehensively characterizes the features of multimodal data, providing a unified data foundation for subsequent intelligent analysis and decision-making tasks.

[0059] In one embodiment, Figure 2 As shown, S2 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0060] S21: Perform cross-modal similarity calculation on the standard data feature set, quantify the directional consistency of visual-auditory and visual-text feature vectors, and generate a similarity matrix. The similarity matrix is ​​used to represent the projection matching degree of different modal features in the semantic space.

[0061] Specifically, the system uses the cosine similarity algorithm to measure the directional similarity of different modal feature vectors in the high-dimensional semantic space by calculating the ratio of the dot product and the module length product between them. For the combination of visual feature vectors and auditory feature vectors, the system calculates them one by one to generate a series of similarity values, which reflect the degree of projection matching of visual and auditory features in the semantic space. Similarly, the system also calculates the similarity of visual feature vectors and text feature vectors to comprehensively evaluate the strength of semantic association between them. After comprehensive calculation, the system generates a complete similarity matrix, the rows and columns of which correspond to different feature vectors, and the elements in the matrix are the corresponding similarity values. This similarity matrix intuitively represents the projection matching degree of different modal features in the semantic space. It not only reflects the pairwise association between visual, auditory and text features, but also provides a quantitative basis for subsequent cross-modal semantic association analysis.

[0062] S22: Perform dynamic threshold screening on the similarity matrix, establish inter-modal connections for feature vector combinations with similarity values ​​higher than a preset semantic association threshold, and generate weighted semantic association edges that reflect cross-modal semantic mapping relationships.

[0063] Specifically, the system dynamically determines a preset semantic association threshold based on the statistical characteristics of the current dataset and the required semantic association strength. This semantic association threshold is adjusted in real time according to the data distribution characteristics and the needs of the application scenario to ensure that the selected feature vector combinations have practical semantic association significance. For each similarity value in the similarity matrix, the system compares it with the semantic association threshold. When the similarity value exceeds the preset semantic association threshold, the corresponding feature vector combination is determined to have significant association at the semantic level and is worthy of further establishing a connection relationship between modalities. Furthermore, the system creates weighted semantic association edges for these highly similar feature vector combinations. The weights of these semantic association edges are directly assigned by the corresponding similarity values. The larger the weight, the closer the semantic association between modalities. In this way, the system selects those with strong semantic associations from numerous feature vector combinations, building a connecting bridge for constructing a cross-modal semantic graph and ensuring that only feature combinations with true semantic associations proceed to the next step of processing.

[0064] S23: Graph construction is performed on the standard data feature set based on weighted semantic association edges, with the feature vectors of the standard data feature set as nodes and the weighted semantic association edges as the connection topology, to generate a cross-modal semantic graph that describes multimodal semantic associations.

[0065] Specifically, the system treats each feature vector in the standard data feature set as a node in the graph. Each node carries the feature information of the corresponding modality, whether visual, auditory, or textual, and is presented as a node in the graph. Weighted semantically associated edges serve as the connection topology, determining the connection method and closeness between nodes. Based on the definition of weighted semantically associated edges, the system connects nodes with strong semantic associations to construct a directed weighted graph structure.

[0066] In this cross-modal semantic graph, nodes represent feature vectors of different modalities, edges indicate semantic associations between modalities, and edge weights quantify the strength of these semantic associations. This graph comprehensively and intuitively describes the complex semantic associations between multimodal data, providing a structured semantic association model for subsequent intelligent analysis and decision-making tasks, helping to deeply explore the inherent connections and potential value of multimodal data. In this way, the system integrates previously dispersed multimodal features into a unified graph structure representation, enabling subsequent analysis and processing within a unified framework and improving the ability to comprehensively understand multimodal data.

[0067] The multi-stage collaborative processing of a multimodal AI data fusion processing method provided above can effectively solve the problem of semantic association ambiguity of multimodal data described in the background technology. Through the calculation of visual-auditory and visual-text dual-channel similarity, the projection matching degree of different modal features in the semantic space is scientifically characterized, which can achieve accurate quantification of cross-modal feature associations; based on the preset semantic association threshold, high-confidence modal connections are automatically established to eliminate low-correlation noise interference, which can achieve dynamic screening of semantic associations; with feature vectors as nodes and weighted semantic association edges as connection topology, a computable multimodal relationship graph is formed to construct a topological semantic mapping network. This technology chain breaks through the semantic understanding deviation of traditional methods in cross-modal data integration, and provides structured semantic mapping support for scenarios such as medical image-pathology report association analysis and autonomous driving multi-sensor data fusion, significantly improving the parsing accuracy and decision reliability of cross-modal semantic associations in complex environments.

[0068] In one embodiment, S3 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0069] S31: Perform multi-head attention weight allocation processing on the cross-modal semantic graph, calculate the node association through the query-key vector dot product, and generate a weight coefficient matrix.

[0070] Specifically, the system pre-configures multiple independent attention heads, each of which has independent query and key vector transformation matrices. For each node in the graph, the system first uses these transformation matrices to convert its feature vector into a query vector and a key vector. By calculating the dot product of the query vector and the key vector, the system obtains an attention score matrix, each element of which represents the strength of the association between the corresponding node pairs. To ensure numerical stability, the system scales these scores by dividing them by the square root of the query and key vector dimensions. Subsequently, the system applies a softmax function to normalize the scores so that the sum of the attention weights of each node to all its neighboring nodes is 1, thereby generating a weight coefficient matrix. The elements in each weight coefficient matrix represent the degree of association between the corresponding node pairs. The higher the weight value, the closer the association between the two nodes at the semantic level. This process allows the system to capture the complex relationships between nodes from multiple different perspectives, providing a detailed association measure for subsequent feature enhancement.

[0071] S32: Perform residual enhancement processing on the weight coefficient matrix, activate the residual connection channel when the node weight in the weight coefficient matrix is ​​higher than the preset dynamic enhancement threshold, and generate an enhanced feature set that eliminates noise interference.

[0072] Specifically, a preset dynamic enhancement threshold is set within the system, which is adjusted in real time according to the feature distribution characteristics of the current data set to adapt to different semantic association strengths. When the node weight in the weight coefficient matrix is ​​higher than the dynamic enhancement threshold, the system activates the residual connection channel. The function of the residual connection channel is to add the original feature vector and the feature vector after multi-head attention weighting element by element to generate an enhanced feature set. This residual enhancement mechanism can not only retain the important information in the original features, but also amplify the influence of key features by fusing the output of multi-head attention, while suppressing the interference of noise features. In this way, the system can effectively eliminate the impact of noise on feature expression, ensure that the enhanced feature set is more focused on semantically related features, and provide a more reliable data basis for subsequent analysis and processing.

[0073] S33: Perform layer normalization on the enhanced feature set to eliminate the feature distribution offset of cross-modal data and generate an optimized feature set with stable distribution.

[0074] Specifically, the system performs layer normalization on the enhanced feature set to eliminate the feature distribution offset across modal data and ensure the stability and consistency of the features. The layer normalization operation is performed independently for each feature dimension within the system, and the mean and standard deviation of each feature dimension are calculated. For each eigenvector in the enhanced feature set, the system calculates the mean and standard deviation of each dimension, and then uses these statistics to normalize the feature vector, that is, subtract the mean from each eigenvalue and divide it by the standard deviation. After this step, the system adjusts the distribution of each feature dimension to a state with zero mean and unit variance, thereby eliminating the offset in the feature distribution of different modal data.

[0075] Layer normalization not only stabilizes the model training process but also improves feature comparability and consistency, enabling features from different modalities to be compared and integrated on a unified scale. Ultimately, the system generates a stable and optimized feature set, providing a more reliable and consistent feature representation for subsequent intelligent analysis and decision-making tasks, and improving the accuracy and efficiency of the system's comprehensive understanding of multimodal data.

[0076] In one embodiment, S4 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0077] S41: Perform adjacency matrix extraction on the cross-modal semantic graph, analyze the structural associations between nodes through topological connection relationships, and generate an adjacency matrix representing the topological connections.

[0078] Specifically, the system traverses all nodes in the graph, identifies the connection relationship between the nodes, and records whether there is an edge between each pair of nodes and the corresponding weight value. For each node, the system checks its association with all other nodes, and stores the connection relationship between the nodes in the form of a matrix based on the connection topology of the graph. The rows and columns of the adjacency matrix correspond to the nodes in the graph, and the elements in the matrix indicate whether there is a connection between the corresponding nodes and the weight of the connection. The weight value reflects the strength of the semantic association between the nodes. A high weight value indicates a strong association, and a low weight value indicates a weak association. In this way, the system converts the complex graph structure into a structured adjacency matrix, providing a basic data structure for subsequent graph processing operations.

[0079] S42: performing edge weight pruning on the adjacency matrix, interrupting inefficient paths corresponding to associated edges whose edge weight values ​​are lower than a connection threshold, and generating an optimized adjacency matrix.

[0080] Specifically, the system performs edge weight pruning on the adjacency matrix to optimize the graph structure and reduce the impact of noise. For each edge weight value in the adjacency matrix, the system compares it with the connection threshold, which is dynamically adjusted according to the feature distribution and semantic association strength of the current data set. If the edge weight value is lower than the connection threshold, the system determines that the corresponding associated edge may be an inefficient path, which contributes little to the semantic association, or may even be noise. Based on this, the system interrupts the paths corresponding to these low-weight edges, that is, removes these edges from the adjacency matrix, thereby generating an optimized adjacency matrix. The optimized adjacency matrix only retains strongly associated edges with edge weight values ​​higher than the connection threshold, which helps to focus on the connections that are truly semantically meaningful in the knowledge graph, improves the efficiency and accuracy of subsequent processing, and reduces the interference of noise on the results.

[0081] S43: Perform graph convolution on the optimized feature set and the optimized adjacency matrix, and generate an aggregated graph representation that integrates cross-modal features by weighting the neighboring node features and updating the representation vector of the central node.

[0082] Specifically, during graph convolution, the system uses an optimized adjacency matrix as a guide, leveraging inter-node connectivity and weight information to propagate and update features. For each central node, the system first collects the feature vectors of all its neighboring nodes and weights them according to the weights in the optimized adjacency matrix. The weights determine the contribution of each neighboring node to the central node's feature update, with features from higher-weighted neighbors having a greater impact on the central node. The system aggregates the weighted neighboring features and combines them with the original features of the central node, updating the central node's representation vector through a nonlinear transformation. This process is implemented internally via a multi-layer graph convolutional network, with each layer performing feature propagation and updates, gradually fusing feature information from different modalities. Ultimately, the system generates an aggregated graph representation that fuses cross-modal features. This representation not only preserves the semantic information of the original features but also integrates inter-node correlation information through graph convolution operations, forming a comprehensive graph representation that provides a richer and more accurate data foundation for subsequent intelligent analysis and decision-making tasks.

[0083] In one embodiment, S5 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0084] S51: Perform variational encoding on the aggregate graph representation, learn Gaussian distribution parameters through the encoder network, and generate a low-dimensional feature vector.

[0085] Specifically, the system uses the encoder network in the variational autoencoder (VAE) to extract features and compress the dimensions of the aggregate graph representation. The encoder network is composed of multiple layers of neural networks, which gradually maps the high-dimensional aggregate graph representation into a low-dimensional latent space. In this process, the encoder network not only learns the main features of the data, but also pays special attention to the statistical characteristics of the data, by outputting two vectors to represent the parameters of the Gaussian distribution: the mean vector and the variance vector. These two vectors define a multivariate Gaussian distribution that is used to capture the potential feature distribution of the data. The system generates a low-dimensional feature vector from this Gaussian distribution through sampling operations. This vector retains the key semantic information of the aggregate graph representation, while introducing a certain degree of randomness, so that the generated feature vector has better generalization ability.

[0086] S52: Perform reconstruction error verification processing on the low-dimensional feature vector, reconstruct the original data distribution represented by the aggregate graph through the decoder and calculate the difference value between the reconstruction result and the original aggregate graph representation to generate a reconstruction error scalar.

[0087] Specifically, the system uses the decoder network in the variational autoencoder to decode the low-dimensional feature vectors and reconstruct the original data distribution represented by the aggregate graph. The decoder network is also a multi-layer neural network structure, and its function is to map the low-dimensional feature vectors back to the original data space to generate a reconstruction result. In order to evaluate the difference between the reconstruction result and the original aggregate graph representation, the system calculates the reconstruction error. The reconstruction error is usually measured using a loss function such as mean squared error (MSE) or binary cross entropy. The specific choice depends on the nature of the data and the task requirements. The system calculates a scalar reconstruction error by comparing the difference between the reconstruction result and the original data in each feature dimension. This error value reflects the ability of the low-dimensional feature vector to reconstruct the original data and the learning effect of the variational autoencoder network.

[0088] S53: Perform model solidification processing on the reconstruction error scalar, lock the variational autoencoder network parameters whose error value is lower than the preset convergence threshold, and generate a low-dimensional semantic space that reflects the essence of cross-modal association.

[0089] Specifically, the system solidifies the reconstruction error scalar. When the reconstruction error scalar remains below a convergence threshold, the system concludes that the variational autoencoder has successfully learned the semantic distribution characteristics of the data and that the model parameters have stabilized. Note that this convergence threshold is predetermined based on the characteristics of the training data and the requirements of the application scenario. At this point, the system locks the current variational autoencoder network parameters, halting further updates. The system uses these solidified parameters to generate a low-dimensional semantic space that reflects the nature of cross-modal associations. This space not only preserves the key semantic information of the original data but also, leveraging the generative capabilities of the variational autoencoder, provides a representation of the data's underlying distribution. This low-dimensional semantic space provides efficient and expressive feature representations for further intelligent analysis, pattern recognition, and data generation tasks, helping to improve the system's overall performance in multimodal data processing. By solidifying the model parameters, the system ensures fast and stable data processing and analysis in real-world applications, providing reliable semantic representation support.

[0090] In one embodiment, Figure 3 As shown, S6 of a multimodal AI data fusion processing method provided by the present invention specifically includes the following steps:

[0091] S61: Based on the contribution quantification of the low-dimensional semantic space, the output layer weights are calculated through the gradient back propagation algorithm to generate the contribution coefficients of the visual modality dimension, auditory modality dimension and text modality dimension.

[0092] Specifically, the system uses the gradient back propagation algorithm to calculate the contribution of each neuron to the final output layer by layer, starting from the output layer. During the calculation process, the system uses the chain rule to gradually backpropagate the error gradient of the output layer to each hidden layer, so as to accurately calculate the weight contribution of each feature dimension to the final output. Specifically, the system performs gradient calculations for the visual modality dimension, the auditory modality dimension, and the text modality dimension respectively, and generates corresponding contribution coefficients. These contribution coefficients reflect the importance of each modality dimension in semantic association analysis. Dimensions with higher values ​​mean that they have a more significant impact on the final output result. The system fully considers the structural characteristics of the low-dimensional semantic space during the calculation process to ensure that the contribution coefficient can accurately reflect the actual contribution of each modality dimension, providing a reliable quantitative basis for the subsequent identification of key dimensions.

[0093] S62: Perform key dimension marking processing on the contribution coefficients of the visual modality dimension, the auditory modality dimension, and the textual modality dimension, mark the semantic dimensions whose contribution coefficient values ​​are higher than the preset significance threshold as semantic key dimensions, and generate a key dimension set.

[0094] Specifically, the system scans the contribution coefficients of all modal dimensions. When the contribution coefficient of a certain dimension exceeds the preset significance threshold, the system marks it as a semantic key dimension, where the preset significance threshold is predetermined based on the specific needs of the application scenario and the statistical characteristics of the data set. These marked key dimensions represent features that play a decisive role in semantic understanding during the multimodal data fusion process. The system aggregates these key dimensions to form a key dimension set. This process helps the system effectively screen out feature dimensions that make important contributions to semantic associations, provides a focus for subsequent dimensionality reduction projection and semantic analysis, and ensures the efficiency and pertinence of the analysis.

[0095] S63: Perform dimensionality reduction projection processing on the key dimension set, map the associated features of the visual modality dimension, auditory modality dimension, and text modality dimension to a two-dimensional plane through the t-SNE algorithm, and generate a semantic association analysis report.

[0096] Specifically, the system performs dimensionality reduction projection on the key dimension set, using the t-SNE algorithm to map the correlation features of the visual, auditory, and textual modal dimensions onto a two-dimensional plane. The t-SNE algorithm internally computes the conditional probability distribution between high-dimensional data points to capture the local structural characteristics of the data, and then constructs a similar probability distribution in a lower-dimensional space. Leveraging this property of the t-SNE algorithm, the system maps the high-dimensional features in the key dimension set onto a two-dimensional plane, placing similar semantic features close together in two-dimensional space and dissimilar semantic features further apart. The system generates a semantic correlation analysis report in the form of an intuitive two-dimensional chart, showcasing the correlations between features from different modalities. The report also includes detailed data statistics and analysis results, helping users understand the semantic correlations within multimodal data and providing a basis for decision-making. This approach not only reveals the key features of multimodal data but also makes complex semantic relationships intuitively visible, enhancing system interpretability and user trust. During the dimensionality reduction projection process, the system precisely controls algorithm parameters to ensure the accuracy and visual quality of the projection results.

[0097] The multimodal AI data fusion processing method provided in this application can effectively solve the problem of inaccurate multimodal semantic analysis described in the background technology through a hierarchical processing mechanism. The gradient back propagation algorithm is used to accurately calculate the weight distribution of visual, auditory and textual modalities in decision-making, so as to achieve scientific quantification of cross-modal dimension contributions; based on the preset significance threshold, high contribution coefficient dimensions are automatically labeled, and low correlation feature interference is filtered out, so as to achieve adaptive recognition of semantic key dimensions; multi-dimensional correlation features are mapped to a two-dimensional visualization plane through nonlinear dimensionality reduction projection to generate an interactive cross-modal semantic association report. This method breaks through the limitations of traditional technologies in key dimension identification and association display, and provides an intuitive and explainable analysis basis for scenarios such as image-pathology report cross-validation in medical diagnosis and multi-sensor decision fusion in autonomous driving, significantly enhancing the decision transparency and deployment effectiveness of cross-modal semantic associations in high-reliability scenarios.

[0098] Preferably, if Figure 4 As shown, the present invention provides a multimodal AI data fusion processing device 700, which is configured with the following modules:

[0099] Feature extraction preprocessing module 710 is used to perform feature extraction and dimension alignment processing on the input raw visual data, raw auditory data, and raw text data. It extracts visual features, auditory time series vectors, and text semantic vectors through a pre-trained encoder, and fuses spatial pooling compression to unify the dimensional features to generate a standard data feature set with unified dimensionality.

[0100] The association graph construction module 720 is used to process the standard data feature set based on the cosine similarity algorithm, analyze the association strength between feature vectors, and perform dynamic threshold screening on the calculation results to obtain a cross-modal semantic graph;

[0101] Feature optimization and enhancement module 730 is used to perform multi-head attention weight allocation on the cross-modal semantic graph, calculate the correlation between nodes and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference;

[0102] Graph convolution fusion module 740, used to perform graph convolution fusion processing on the optimized feature set and the cross-modal semantic graph, propagate node information based on the adjacency matrix and perform threshold pruning on the associated edge weights to obtain an aggregated graph representation of the fused cross-modal features;

[0103] Latent space mapping module 750 is used to perform latent space mapping processing on the aggregate graph representation, learn semantic distribution features through variational autoencoders, and perform convergence verification processing on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal associations;

[0104] The semantic association analysis module 760 is used to quantify the contribution of the low-dimensional semantic space, identify the key dimensions with the highest decision weights, and perform dimensionality reduction visualization on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output the semantic association rules and key dimension contribution distribution between cross-modal data.

[0105] To sum up, the multimodal AI data fusion processing device provided in this application can effectively solve the problems of multimodal data dimensional heterogeneity and noise interference in the AI ​​data fusion processing process through systematic technical collaboration. Specifically, the system unifies the feature representations of vision, hearing, and text through pre-trained encoders and spatial pooling compression technology, eliminates the structural differences between the original data sources, and realizes adaptive dimensional alignment of multimodal data; uses cosine similarity calculation and dynamic threshold screening to construct a semantic graph with precise mapping, achieves the quantitative generation of cross-modal semantic associations, and effectively captures the inter-translatable relationships between different modalities; through multi-head attention weight allocation combined with residual enhancement mechanism, the representation purity of key features is significantly improved, thereby realizing adaptive suppression of feature noise; graph convolution and edge weight pruning based on the adjacency matrix ensure the stable association of multi-source data in the topological space and establish a robust cross-modal topological fusion architecture; uses variational autoencoders to learn potential distribution features and verify them through reconstruction errors, revealing the essential laws of cross-modal associations and realizing interpretable mapping of low-dimensional semantic space; outputs semantic association analysis reports through contribution quantification and dimensionality reduction projection, generates visual decision-making basis for key dimensions, and provides accurate cross-modal association rule identification and dimensional contribution evaluation for high-reliability scenarios such as autonomous driving and medical diagnosis.

[0106] Preferably, the feature extraction preprocessing module 710 provided in this application is configured with the following units:

[0107] The visual feature extraction unit is used to extract spatial features from the input raw visual data, call the Vision Transformer model to hierarchically analyze the local information of the image, and generate a visual feature matrix containing spatial structural features;

[0108] Auditory temporal coding unit: This unit performs temporal coding on the input raw auditory data based on a bidirectional long short-term memory network, extracts the temporal evolution of acoustic features, and generates a fixed-dimensional auditory temporal vector.

[0109] The text semantic aggregation unit is used to perform semantic aggregation processing on the input raw text data, integrate the contextual association characteristics of language elements, and generate text semantic vectors that represent logical relationships;

[0110] The multimodal feature fusion unit is used to perform adaptive spatial pooling processing on the visual feature matrix, combine the auditory time series vector and the text semantic vector to perform multimodal feature fusion, and generate a standard data feature set of unified dimension.

[0111] Preferably, the association graph construction module 720 provided in this application is configured with the following units:

[0112] The cross-modal similarity calculation unit is used to perform cross-modal similarity calculation on the standard data feature set, quantify the directional consistency of visual-auditory and visual-text feature vectors, and generate a similarity matrix. The similarity matrix is ​​used to represent the projection matching degree of different modal features in the semantic space;

[0113] A semantic association edge generation unit is used to perform dynamic threshold screening on the similarity matrix, establish inter-modal connections for feature vector combinations with similarity values ​​higher than a preset semantic association threshold, and generate weighted semantic association edges that reflect cross-modal semantic mapping relationships;

[0114] The cross-modal semantic graph construction unit is used to construct a graph for the standard data feature set based on weighted semantic association edges, using the feature vectors of the standard data feature set as nodes and the weighted semantic association edges as the connection topology to generate a cross-modal semantic graph that describes multimodal semantic associations.

[0115] Preferably, the feature optimization and enhancement module 730 provided in this application is configured with the following units:

[0116] The multi-head attention weight allocation unit is used to perform multi-head attention weight allocation processing on the cross-modal semantic graph, calculate the node association through the query-key vector dot product, and generate a weight coefficient matrix;

[0117] The residual enhancement processing unit is used to perform residual enhancement processing on the weight coefficient matrix. When the node weight in the weight coefficient matrix is ​​higher than the preset dynamic enhancement threshold, the residual connection channel is activated to generate an enhanced feature set that eliminates noise interference;

[0118] The layer normalization processing unit is used to perform layer normalization processing on the enhanced feature set, eliminate the feature distribution offset of cross-modal data, and generate an optimized feature set with stable distribution.

[0119] Preferably, the graph convolution fusion module 740 provided in this application is configured with the following units:

[0120] The adjacency matrix extraction unit is used to extract the adjacency matrix of the cross-modal semantic graph, analyze the structural associations between nodes through topological connection relationships, and generate an adjacency matrix representing the topological connections;

[0121] The edge weight pruning unit is used to prune the edge weights of the adjacency matrix, interrupting the inefficient paths corresponding to the associated edges whose edge weight values ​​are lower than the connection threshold, and generating an optimized adjacency matrix;

[0122] The graph convolution feature fusion unit is used to perform graph convolution processing on the optimized feature set and the optimized adjacency matrix, and generate an aggregated graph representation that fuses cross-modal features by weighting the neighboring node features and updating the representation vector of the central node.

[0123] Preferably, the latent space mapping module 750 provided in this application is configured with the following units:

[0124] A variational coding processing unit is used to perform variational coding on the aggregate graph representation, learn Gaussian distribution parameters through the encoder network, and generate a low-dimensional feature vector;

[0125] A reconstruction error verification unit is used to perform reconstruction error verification processing on the low-dimensional feature vector, reconstruct the original data distribution represented by the aggregate graph through the decoder, and calculate the difference between the reconstruction result and the original aggregate graph representation to generate a reconstruction error scalar;

[0126] The model solidification processing unit is used to perform model solidification processing on the reconstruction error scalar, lock the variational autoencoder network parameters whose error values ​​are lower than the preset convergence threshold, and generate a low-dimensional semantic space that reflects the essence of cross-modal association.

[0127] Preferably, the semantic association analysis module 760 provided in this application is configured with the following units:

[0128] The modal contribution calculation unit is used to quantify the contribution of the low-dimensional semantic space, calculate the output layer weights through the gradient back propagation algorithm, and generate the contribution coefficients of the visual modal dimension, the auditory modal dimension, and the textual modal dimension;

[0129] A key dimension labeling unit is used to perform key dimension labeling processing on the contribution coefficients of the visual modality dimension, the auditory modality dimension, and the textual modality dimension, label the semantic dimensions whose contribution coefficient values ​​are higher than the preset significance threshold as semantic key dimensions, and generate a key dimension set;

[0130] The semantic association visualization unit is used to perform dimensionality reduction projection processing on the key dimension set, map the associated features of the visual modality dimension, auditory modality dimension and text modality dimension to a two-dimensional plane through the t-SNE algorithm, and generate a semantic association analysis report.

[0131] In one embodiment, the present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned multimodal AI data fusion processing method when executing the computer program.

[0132] In one embodiment, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned multimodal AI data fusion processing method is implemented.

[0133] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0135] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal AI data fusion processing method, characterized in that: The following steps are involved: S1: Perform feature extraction and dimension alignment on the input raw visual data, raw auditory data, and raw text data. Pre-trained encoders are used to extract visual features, auditory time series vectors, and text semantic vectors. These features are then fused using spatial pooling to compress the unified dimensional features and generate a standard data feature set with unified dimensionality. S2: Processing the standard data feature set based on the cosine similarity algorithm, analyzing the correlation strength between feature vectors and performing dynamic threshold screening on the calculation results to obtain a cross-modal semantic graph; S3: Perform multi-head attention weight allocation on the cross-modal semantic graph, calculate the correlation between nodes and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference; S4: performing graph convolution fusion processing on the optimized feature set and the cross-modal semantic graph, propagating node information based on the adjacency matrix and performing threshold pruning processing on the associated edge weights to obtain an aggregated graph representation of the fused cross-modal features; S5: performing latent space mapping processing on the aggregate graph representation, learning semantic distribution features through a variational autoencoder, and performing convergence verification processing on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal association; S6: Perform contribution quantification processing on the low-dimensional semantic space, identify the key dimensions with the highest decision weights, and perform dimensionality reduction visualization processing on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output semantic association rules and key dimension contribution distribution between cross-modal data.

2. The method according to claim 1, characterized in that Said S1 comprises: S11: Perform spatial feature extraction on the input raw visual data, call the Vision Transformer model to hierarchically analyze the local information of the image, and generate a visual feature matrix containing spatial structural features; S12: Based on the bidirectional long short-term memory network, the input raw auditory data is temporally encoded to extract the temporal evolution of acoustic features and generate a fixed-dimensional auditory time series vector. S13: Perform semantic aggregation processing on the input raw text data, integrate the contextual association characteristics of language elements, and generate text semantic vectors that represent logical relationships; S14: Adaptively perform spatial pooling processing on the visual feature matrix, combine the auditory time series vector and the text semantic vector to perform multimodal feature fusion, and generate a standard data feature set of unified dimension.

3. The method according to claim 1, characterized in that The S2 includes: S21: performing cross-modal similarity calculation on the standard data feature set, quantifying the directional consistency of the visual-auditory and visual-text feature vectors, and generating a similarity matrix, wherein the similarity matrix is ​​used to represent the projection matching degree of different modal features in the semantic space; S22: performing dynamic threshold screening on the similarity matrix, establishing inter-modal connections for feature vector combinations with similarity values ​​higher than a preset semantic association threshold, and generating weighted semantic association edges reflecting cross-modal semantic mapping relationships; S23: Based on the weighted semantic association edges, the standard data feature set is subjected to graph construction processing, and the feature vectors of the standard data feature set are used as nodes and the weighted semantic association edges are used as connection topology to generate a cross-modal semantic graph describing multimodal semantic associations.

4. The method according to claim 1, wherein The S3 includes: S31: performing multi-head attention weight allocation processing on the cross-modal semantic graph, calculating node association through query-key vector dot product, and generating a weight coefficient matrix; S32: performing residual enhancement processing on the weight coefficient matrix, activating the residual connection channel when the node weight in the weight coefficient matrix is ​​higher than a preset dynamic enhancement threshold, and generating an enhanced feature set that eliminates noise interference; S33: performing layer normalization processing on the enhanced feature set to eliminate feature distribution offset of cross-modal data and generate an optimized feature set with stable distribution.

5. The method according to claim 1, characterized in that The S4 includes: S41: performing adjacency matrix extraction processing on the cross-modal semantic graph, parsing the structural associations between nodes through topological connection relationships, and generating an adjacency matrix representing the topological connections; S42: performing edge weight pruning on the adjacency matrix, interrupting inefficient paths corresponding to associated edges whose edge weight values ​​are lower than a connection threshold, and generating an optimized adjacency matrix; S43: Perform graph convolution processing on the optimized feature set and the optimized adjacency matrix, and generate an aggregated graph representation that fuses cross-modal features by weighting neighbor node features and updating the representation vector of the central node.

6. The method according to claim 1, characterized in that The S5 includes: S51: performing variational encoding processing on the aggregate graph representation, learning Gaussian distribution parameters through an encoder network, and generating a low-dimensional feature vector; S52: Performing reconstruction error verification processing on the low-dimensional feature vector, reconstructing the original data distribution represented by the aggregate graph through a decoder and calculating the difference between the reconstruction result and the original aggregate graph representation to generate a reconstruction error scalar; S53: Performing model solidification processing on the reconstruction error scalar, locking the variational autoencoder network parameters whose error values ​​are lower than a preset convergence threshold, and generating a low-dimensional semantic space that reflects the essence of cross-modal association.

7. The method according to any one of claims 1 to 6, characterized in that The S6 includes: S61: Based on the contribution quantification of the low-dimensional semantic space, the output layer weight is calculated by the gradient back propagation algorithm to generate the contribution coefficients of the visual modality dimension, the auditory modality dimension and the text modality dimension; S62: performing key dimension labeling processing on the contribution coefficients of the visual modality dimension, the auditory modality dimension, and the textual modality dimension, marking the semantic dimensions whose contribution coefficient values ​​are higher than a preset significance threshold as semantic key dimensions, and generating a key dimension set; S63: Perform dimensionality reduction projection processing on the key dimension set, map the associated features of the visual modality dimension, the auditory modality dimension, and the text modality dimension to a two-dimensional plane through the t-SNE algorithm, and generate a semantic association analysis report.

8. A multimodal AI data fusion processing device, characterized in that: The device comprises: The feature extraction preprocessing module is used to perform feature extraction and dimension alignment on the input raw visual data, raw auditory data, and raw text data. It extracts visual features, auditory time series vectors, and text semantic vectors through a pre-trained encoder, and fuses spatial pooling compression to unify the dimensional features to generate a standard data feature set with unified dimensionality. An association graph construction module is used to process the standard data feature set based on the cosine similarity algorithm, analyze the association strength between feature vectors and perform dynamic threshold screening on the calculation results to obtain a cross-modal semantic graph; A feature optimization and enhancement module is used to perform multi-head attention weight allocation on the cross-modal semantic graph, calculate the correlation between nodes, and perform residual enhancement processing on high-weight nodes to obtain an optimized feature set that eliminates noise interference; A graph convolution fusion module is used to perform graph convolution fusion processing on the optimized feature set and the cross-modal semantic graph, propagate node information based on the adjacency matrix and perform threshold pruning on the associated edge weights to obtain an aggregated graph representation of the fused cross-modal features; a latent space mapping module for performing latent space mapping processing on the aggregate graph representation, learning semantic distribution features through a variational autoencoder, and performing convergence verification processing on the encoding-decoding reconstruction error to obtain a low-dimensional semantic space that reflects the essence of cross-modal associations; The semantic association analysis module is used to quantify the contribution of the low-dimensional semantic space, identify the key dimensions with the highest decision weights, and perform dimensionality reduction visualization on the clustering relationship to obtain a semantic association analysis report. The semantic association analysis report is used to output the semantic association rules and key dimension contribution distribution between cross-modal data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images