A dynamic sparse cross-modal fusion data feature extraction method and system

Abstract

The application discloses a kind of dynamic sparse cross-modal fusion data feature extraction method and system, the method includes multi-modal feature coding and alignment, dynamic sparse cross-modal fusion, is completed by two modules of near-end operator sparse controller and Top-k sparse cross-modal attention module collaborative;After dynamic sparse fusion, the feature containing highly concentrated cross-modal information is obtained, the feature of cross-modal information is converted into compact structure, semantic rich data representation by hierarchical data encoder module;Hybrid expert architecture is used;The system includes modal feature encoder, dynamic sparse fusion layer, data encoder and task adapter.The application greatly reduces the calculation complexity of traditional cross-modal fusion;Make data have global consistency while retaining fine-grained details;Significantly reduce the computing overhead, while maintaining the richness of cross-modal semantics, applicable to image, audio and other multi-modal data feature extraction tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data processing, and particularly relates to a dynamic sparse cross-modal fusion data feature extraction method and system. BACKGROUND

[0002] With the rapid development of multi-modal large language models, cross-modal feature fusion has become a key technology to improve model performance. Traditional cross-modal fusion methods mainly rely on dense fully connected attention mechanisms or simple concatenation operations. Although these methods can capture global associations between modalities, they have high computational complexity and high resource consumption. Especially when dealing with high-dimensional data such as images and audio, the computational overhead of dense attention mechanisms grows quadratically, severely limiting the scalability of the model. In recent years, sparse attention mechanisms have been introduced to alleviate computational pressure. For example, methods based on total variation sparse attention achieve structured sparsity through proximal operators; generalized structured sparse functions are used for cross-modal metric learning. However, these methods usually use static sparse patterns, which cannot dynamically adjust the sparsity ratio according to the semantic complexity of the input data, resulting in limited performance in heterogeneous data scenarios. In addition, existing sparse cross-modal attention mechanisms often lack explicit modeling of hierarchical data representation, making it difficult to generate compact and semantically rich cross-modal features. In terms of data representation, existing research uses large language models to extract data from heterogeneous data, but does not fully consider the sparsity of cross-modal interaction; there are also methods that enhance the discoverability of data through language models, but their representation ability is limited by the redundant computation of dense fusion. In addition, although sparse interactive attention networks are optimized for multi-modal sentiment recognition, they do not address the problem of dynamic sparse routing and data encoding collaboration.

[0003] The limitations of existing technologies mainly manifest in the following three aspects: static sparsity: the sparse pattern of existing sparse attention mechanisms is fixed and cannot adapt to the dynamic needs of different input complexities; data representation deficiency: traditional methods do not combine sparse interaction with hierarchical data encoding, resulting in loss of semantic information; computational efficiency bottleneck: the high computational cost of dense cross-modal fusion limits its application in large models. Therefore, a data feature extraction method that can overcome the above limitations is needed. SUMMARY

[0004] The purpose of the present application is to provide a dynamic sparse cross-modal fusion data feature extraction method and system.

[0005] Technical solution: The dynamic sparse cross-modal fusion data feature extraction method according to the present application comprises the following steps:

[0006] Step 1: Multi-modal feature encoding and alignment, converting original multi-modal data of different sources and different structures into unified feature representations for subsequent processing;

[0007] Step 2: Dynamic sparse cross-modal fusion, which is completed by the cooperation of the proximal operator sparsity controller and the Top-k sparse cross-modal attention module;

[0008] Step 3: After dynamic sparse fusion, the features containing highly condensed cross-modal information are obtained, and the hierarchical data encoder module is used to convert the cross-modal information features into compact and semantically rich data representations;

[0009] Step 4: A hybrid expert architecture is adopted, and multiple parallel dynamic sparse fusion layers are instantiated as "experts". Each expert is designed to handle a specific combination of modal interactions. Through a sparse gating network, the system activates only the most relevant experts for computation based on the modal types of the input data.

[0010] Further, the step 1 comprises:

[0011] Step 1-1: For each input modality, a pre-trained model is used as a feature encoder for processing;

[0012] Step 1-2: After processing by the respective encoders, the original data is converted into high-dimensional feature vectors. Through a learnable linear projection layer, the feature vectors of all modalities are mapped to the same shared semantic subspace to obtain a feature matrix where m represents the modality index, n is the sequence length, d is the aligned feature dimension, R is the set of feature vectors.

[0013] Further, the input modalities of step 1-1 include text, images, and audio.

[0014] Further, the step 2 comprises:

[0015] Step 2-1: Before attention calculation, the proximal operator sparsity controller intelligently determines the "sparsity degree" of this interaction;

[0016] Step 2-2: Through dynamic sparse attention calculation, the sparsity ratio is obtained, and the Top-k sparse cross-modal attention module performs actual fusion calculation.

[0017] Further, the step 2-1 comprises:

[0018] Step 2-1-1: Calculate the global correlation between modalities to form a feature correlation matrix F where element represents the correlation between modality i and modality jthe overall correlation strength between the two, the input feature matrix of the modal i , the input feature matrix of the modal j , T is a transpose matrix;

[0019] Step 2-1-2: After vectorization, input a multi-layer perception to predict an initial sparse ratio F ; ;

[0020] Step 2-1-3: Apply a proximal gradient optimization algorithm to constraint to obtain the final dynamic sparse ratio:

[0021] ;

[0022] wherein, is the final dynamic sparse ratio, is the initial dynamic sparse ratio, represents an element-wise sign function, is a sparse constraint strength parameter.

[0023] Further, the step 2-2 comprises:

[0024] Step 2-2-1: After alignment, the feature matrix is linearly transformed to obtain standard query vectors, key vectors and value vectors;

[0025] Step 2-2-2: Calculate the original attention score between the query vector and the key vector through a learnable correlation matrix U , wherein, is a Sigmoid activation function, T is a transpose matrix; is the query vector of the modal i , is the key vector of the modal j , is the value vector of the modal j ;

[0026] Step 2-2-3: According to the final dynamic sparse ratio, select the most important values from the original attention score matrix S ij through a Top-k sparse cross-modal attention module, and generate a binary sparse mask M ij , wherein the positions with value 1 in the mask correspond to key interactions that need to be retained, and the positions with value 0 correspond to redundant interactions that are discarded;

[0027] Step 2-2-4: The standard attention calculation result is multiplied with the sparse mask M ij Element-wise multiplication is performed to obtain sparse attention weights A ij And the weighted fusion feature is calculated.

[0028] Further, the step 3 comprises:

[0029] Step 3-1: For each modality's post-fusion feature Z i , its meta information is selectively extracted through a gating mechanism, thereby filtering out noise and extracting the core data of the modality m i ；

[0030] Step 3-2: The core data of all modalities is constructed into a graph structure, where M represents each modality, and the edge weight between nodes is determined by the similarity of the data, and the information propagation and aggregation are performed using a graph neural network, so that the representation of each node is integrated into the information of other modalities, and finally a hierarchical data vector m' is generated.

[0031] The dynamic sparse cross-modal fusion data feature extraction system disclosed by the application comprises a modality feature encoder, a dynamic sparse fusion layer, a data encoder and a task adapter, and each component is connected through a standardized data interface to form an end-to-end processing pipeline.

[0032] The modality feature encoder is used to extract low-dimensional features of the original data using a pre-trained model, and outputs a feature matrix with uniform dimensions.

[0033] The dynamic sparse fusion layer comprises a proximal operator sparse controller and a Top-k sparse cross-modal attention module, and is used to output the filtered cross-modal feature representation.

[0034] The data encoder adopts a two-level processing architecture, first extracts modality-specific data through a gating mechanism, and then aggregates cross-modal data using a graph neural network to finally generate a unified semantic representation vector.

[0035] The task adapter is used to configure the output layer according to the downstream task requirements, supports flexible task expansion, and realizes parallel processing using a hybrid expert architecture.

[0036] Beneficial effects: Compared with the prior art, the present application has the following remarkable advantages: the present application greatly reduces the computational complexity of traditional cross-modal fusion; the synergistic design of the gating mechanism and the graph aggregation enables the data to retain fine-grained details while maintaining global consistency; the introduction of the proximal gradient optimization enables continuous sparse ratio adjustment within the range of 0.1-0.9, greatly shortens the response time, the hybrid expert architecture supports parallel scheduling of up to 256 modal experts, the sparse gating network stabilizes the expert activation rate between 15%-20%, and the linear speedup ratio on the multi-node cluster is greatly improved; significantly reduces the computational overhead while maintaining the richness of cross-modal semantics, and is suitable for feature extraction tasks of multi-modal data such as images and audio. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The multi-modal large language model system architecture of the present application;

[0038] Figure 2 The detailed structure diagram of the dynamic sparse fusion layer of the present application. DETAILED DESCRIPTION

[0039] The technical solutions of the present application will be further described below in conjunction with the drawings.

[0040] The dynamic sparse cross-modal fusion data feature extraction method described in the present application comprises the following steps:

[0041] Step 1: Multi-modal feature encoding and alignment, converting original multi-modal data of different sources and different structures into unified feature representations for subsequent processing;

[0042] Step 1-1: For each input modality (such as text, image, audio, etc.), a pre-trained model is used as a feature encoder for processing; BERT model is used to process text data, ResNet (Residual Network) model is used to process image data, and HuBERT (Hidden-Unit BERT) model is used to process audio data.

[0043] Step 1-2: After passing through the respective encoders, the original data is converted into high-dimensional feature vectors, and through a learnable linear projection layer, the feature vectors of all modalities are mapped into the same shared semantic subspace to obtain the feature matrix

[0044] , wherein m represents the modality index, n is the sequence length, d is the feature dimension after alignment, R is the feature vector set.

[0045] Step 2: Dynamic sparse cross-modal fusion, which is completed collaboratively by two modules: the Proximal Operator Sparse Controller (POSC) and the Top-k Sparse Cross-modal Attention (TkSCMA) module;

[0046] Step 2-1: Before performing attention calculation, the "sparseness" of this interaction is intelligently determined by the Proximal Operator Sparse Controller (POSC).

[0047] Step 2-1-1: Calculate the global correlation between modes to form a feature correlation matrix. F , of which elements Representing modes i With mode j The overall correlation strength between them Representing modes i The input feature matrix, Representing modes j The input feature matrix, T It is the transpose matrix;

[0048] Step 2-1-2: Calculate the correlation matrix F After vectorization, the data is input into a multilayer perceptron to predict an initial sparsity ratio. ;

[0049] Step 2-1-3: Apply the proximal gradient optimization algorithm to... By applying constraints, the final dynamic sparsity ratio is obtained:

[0050] ;

[0051] in, It is the final dynamic sparsity ratio. It is the initial dynamic sparsity ratio. The symbolic function representing element-wise operations. It is the sparse constraint strength parameter.

[0052] Step 2-2: After obtaining the sparsity ratio through dynamic sparse attention calculation, the actual fusion calculation is performed by the Top-k Sparse Cross-modal Attention (TkSCMA) module.

[0053] Step 2-2-1: Aligned Feature Matrix After linear transformation, standard query vector, key vector, and value vector are obtained;

[0054] Step 2-2-2: Through a learnable correlation matrix U Calculate the raw attention score between the query vector and the key vector. ,in, It is the Sigmoid activation function. Tis a transpose matrix; is a query vector of the modal i , is a key vector of the modal j , is a value vector of the modal j ;

[0055] Step 2-2-3: According to the final dynamic sparse ratio, the most important values are selected from the original attention score matrix S ij by the Top-k sparse cross-modal attention module, and a binary sparse mask M ij is generated, where the positions with value 1 in the mask correspond to key interactions that need to be retained, and the positions with value 0 correspond to redundant interactions that are discarded.

[0056] Step 2-2-4: Element-wise multiplication is performed between the standard attention calculation result and the sparse mask M ij to obtain the sparse attention weight A ij , and the weighted fusion feature is calculated.

[0057] Step 3: After dynamic sparse fusion, the feature containing highly condensed cross-modal information is obtained, and the hierarchical data encoder (HME) module is used to convert the cross-modal information feature into a compact structure and rich semantic data representation.

[0058] Step 3-1: For each modal fused feature Z i , a gating mechanism is used to selectively extract its meta information, thereby filtering out noise and extracting the core data of the modal m i ；

[0059] Step 3-2: The core data of all modalities is constructed into a graph structure, where M represents each modality, and the edge weight between nodes is determined by the similarity of the data. Graph neural networks are used for information propagation and aggregation, so that the representation of each node is integrated into the information of other modalities, and finally a hierarchical data vector m' is generated.

[0060] Step 4: To improve the scalability and efficiency of the mixed expert architecture in large-scale applications, the present invention can adopt a mixed expert architecture. Under this architecture, multiple parallel dynamic sparse fusion layers can be instantiated as "experts", each of which can be designed to handle a specific combination of modal interactions (such as a "text-image" expert). Through a sparse gating network, the system can activate only a few most relevant experts for computation based on the modal types of the input data, while the remaining experts remain dormant, thereby achieving efficient use of computing resources.

[0061] As shown in Figure 1 is a multi-modal large language model system architecture diagram. The input layer receives multi-modal data input, including original text data input interface, original image data input interface, original audio data input interface, and original video data input interface. The feature extraction module is the feature extraction process for each modality. The text feature extractor is based on BERT / GPT text feature encoding, the image feature extractor is based on ResNet image feature encoding, the audio feature extractor is based on HuBERT audio feature encoding, the video feature extractor is based on 3D-CNN video feature encoding, and the dynamic sparse fusion layer is the core innovative module of the invention. The Top-k sparse cross-modal attention (TkSCMA) module performs cross-modal sparse attention calculation, the proximal operator sparse controller (POSC) dynamically adjusts the attention sparse ratio, and the hierarchical data encoder (HME) generates hierarchical data representation. The output layer outputs the final data result, and the data output provides a unified semantic representation for downstream tasks. The dynamic sparse fusion layer replaces the traditional dense fusion mechanism, realizes structured sparse attention through the Top-k sparse cross-modal attention module, and first introduces the proximal operator for dynamic sparse control through the proximal operator sparse controller. The hierarchical data encoder (HME) innovatively adopts a two-level encoding architecture of gating mechanism and graph aggregation.

[0062] As shown in Figure 2The dynamic sparse fusion layer is shown in detail. The Top-k sparse cross-modal attention is a core attention calculation module. The query projection converts the input features into a query vector, the key projection converts the input features into a key vector, the value projection converts the input features into a value vector, the sparse mask generator generates a Top-k mask according to the proximal operator sparse controller instruction, and the attention calculation performs sparse attention weight calculation. The proximal operator sparse controller is a dynamic sparse adjustment module, including a correlation matrix calculation mode inter-feature correlation, a multi-layer perception (MLP) generating an initial sparse ratio candidate value, and a proximal optimization applying an L1 constraint to obtain a final sparse ratio. The hierarchical data encoder is a data generation module, including a gated latent projection extracting mode-specific data and a cross-modal graph aggregation integrating multi-modal data. The Top-k sparse cross-modal attention (TkSCMA) adopts a Top-k sparse mechanism guided by a learnable correlation matrix. The proximal operator sparse controller innovatively applies proximal gradient optimization to sparse ratio control. The hierarchical data encoder (HME) realizes selective data retention and cross-modal integration of graph structure through a gating mechanism.

[0063] Efficient implementation of a multi-modal retrieval system

[0064] In a cross-modal retrieval scenario, such as "text-to-image" or "image-to-video", the system needs to quickly find the most matching content to the query intent in a large amount of data.

[0065] Feature preprocessing and alignment: For text queries, use models such as BERT to extract semantic features; for content in image or video libraries, use ResNet or 3D-CNN to extract visual features. All features are aligned to a unified 512-dimensional shared semantic space through a linear projection layer.

[0066] Dynamic sparse similarity calculation: When the user inputs the query, the system does not perform dense similarity calculation on all entries in the database. Instead, it uses the Top-k sparse cross-modal attention (TkSCMA) module to perform sparse interaction between the query features and the features of other modalities. The proximal operator sparse controller (POSC) module dynamically adjusts the sparse ratio ρ according to the complexity of the query. For example, for a simple keyword query (such as "beach"), the system uses a higher sparse degree (such as ρ=0.2) to focus on the most core semantic concepts; for a complex long sentence query (such as "a girl in a red dress running at sunset"), the sparse degree is reduced (such as ρ=0.7) to capture more detailed interactions.

[0067] Fast matching based on data: The HME module generates a compact 256-dimensional data vector for each entry in the database and the query. This vector is a highly condensed and semantic representation of the original high-dimensional features. The retrieval process is transformed into a fast nearest-neighbor search in this low-dimensional data space, greatly improving the retrieval speed. What will be returned to the user is a list of retrieval results sorted according to the distance of the data vectors.

[0068] Automatic video document generation system

[0069] This embodiment aims to automatically transform unstructured video data into a structured text report or summary, i.e., a "video document".

[0070] Parallel extraction of multi-modal features: The system receives video stream input and synchronously processes at least three modalities: 1) visual features (keyframe and dynamic information extracted by a spatio-temporal convolutional network); 2) audio features (background sound, ambient sound processed by an audio encoder); 3) text features (speech transcribed into text by automatic speech recognition ASR technology).

[0071] Event-driven sparse fusion: The Top-k Sparse Cross-Modal Attention (TkSCMA) module is responsible for fusing the feature streams of the three modalities. The fusion process is event-driven, for example, the system will focus on enhancing the association between the speaker's facial features, their voice features, and the text they speak. At the same time, it will also strongly associate specific events in the picture (such as the turning of PPT in a course, the display of charts in a meeting) with the corresponding voice explanation content.

[0072] Scene-aware sparse control: The Proximal Operator Sparse Controller (POSC) module controls the sparsity of the fusion according to the dynamics of the video content. In a long single-person speech (static scene), the interaction between vision and text can be very sparse (low ρ) to save computing resources; while in a multi-person discussion or a rapidly changing scene (dynamic scene), the system will automatically reduce the sparsity (high ρ) to ensure that all key cross-modal information is captured.

[0073] Hierarchical data-driven content generation: The Hierarchical Data Encoder (HME) module is the key to achieving high-quality document generation. It encodes the fused feature stream into a hierarchical data structure with time sequence. This structure may contain a series of event triples, such as [<timestamp: 00:15:30>, <event: "chart example">, <main speaker: "Professor Li">, <core content: "showed a chart showing year-on-year growth in sales">]. Finally, a large language model takes this structured data stream as input and constraint to generate a text report that is logically clear, focused, and coherent in content.

[0074] The dynamic sparse cross-modal fusion data feature extraction system disclosed by the application is characterized in that a feature fusion process in a traditional multi-modal large language model (MMLM) is reconstructed through a structured sparse mechanism, including a modal feature encoder, a dynamic sparse fusion layer, a data encoder and a task adapter, each component is connected through a standardized data interface to form an end-to-end processing pipeline, and the calculation efficiency and semantic representation ability are improved together. In terms of technical value, the application first combines dynamic structured sparse learning with hierarchical data encoding, solving the contradiction between static sparsity and semantic loss in traditional methods. The Top-k sparse cross-modal attention (TkSCMA) module realizes accurate screening of cross-modal interaction through a learnable binary mask, the near operator sparse controller uses near gradient optimization to dynamically adjust the sparsity, and the hierarchical data encoder (HME) module builds a progressive representation pipeline from raw features to high-level semantics. This design automatically balances the calculation overhead and representation quality according to the complexity of the input data, providing a new technical path for multi-modal data processing.

[0075] The modal feature encoder is used to extract low-dimensional features of the original data by using a pre-trained model, and outputs a feature matrix with uniform dimensions.

[0076] The dynamic sparse fusion layer includes a near operator sparse controller (POSC) and a Top-k sparse cross-modal attention (TkSCMA) module, and is used to output filtered cross-modal feature representations.

[0077] The data encoder adopts a two-stage processing architecture, first extracts modal-specific data through a gating mechanism, then aggregates cross-modal data using a graph neural network, and finally generates a unified semantic representation vector.

[0078] The task adapter is used to configure the output layer according to the requirements of the downstream task, supports flexible task expansion, and realizes parallel processing using a hybrid expert architecture.

Claims

1. A method for feature extraction from dynamic sparse cross-modal fusion data, characterized in that, Includes the following steps: Step 1: Multimodal feature encoding and alignment, transforming raw multimodal data from different sources and with varying structures into a unified feature representation that can be processed later; Step 2: Dynamic sparse cross-modal fusion, which is completed collaboratively by two modules: the proximal operator sparse controller and the Top-k sparse cross-modal attention module; Step 3: After dynamic sparse fusion, features containing highly condensed cross-modal information are obtained. The hierarchical data encoder module transforms the cross-modal information features into a compact and semantically rich data representation. Step 4: Adopt a hybrid expert architecture, instantiate multiple parallel dynamic sparse fusion layers as "experts", each expert is designed to handle a specific combination of modal interactions. Through a sparse gating network, the system activates only the most relevant expert for computation based on the modality type of the input data. The proximal operator sparsity controller is a dynamic sparsity adjustment module, which includes calculating the correlation matrix for intermodal feature correlation, generating initial sparsity ratio candidate values ​​using a multilayer perceptron (MLP), and applying L1 constraints to the proximal optimization to obtain the final sparsity ratio. The Top-k sparse cross-modal attention module is the core attention calculation module. It guarantees that the Query projection converts the input features into query vectors, the Key projection converts the input features into key vectors, the Value projection converts the input features into value vectors, the sparse mask generator generates Top-k masks according to the instructions of the near-end operator sparse controller, and the attention calculation performs the sparsified attention weight calculation. The hierarchical data encoder is the data generation module, which includes gated latent projection to extract modality-specific data and cross-modal graph aggregation to integrate multimodal data. The hierarchical data encoder achieves selective data retention and cross-modal integration of graph structures through a gating mechanism.

2. The method for extracting features from dynamic sparse cross-modal fusion data according to claim 1, characterized in that, Step 1 includes: Step 1-1: For each input modality, a pre-trained model is used as a feature encoder for processing; Steps 1-2: After passing through their respective encoders, the original data is transformed into high-dimensional feature vectors. Through a learnable linear projection layer, the feature vectors of all modalities are mapped to the same shared semantic subspace, resulting in the feature matrix. ,in m Represents modal index, n For sequence length, d For the aligned feature dimensions, R It is a set of feature vectors.

3. The method for feature extraction from dynamic sparse cross-modal fusion data according to claim 2, characterized in that, The input modalities in step 1-1 include text, images, and audio.

4. The method for feature extraction of dynamic sparse cross-modal fusion data according to claim 1, characterized in that, Step 2 includes: Step 2-1: Before performing attention calculation, the "sparseness" of this interaction is intelligently determined by the proximal operator sparse controller; Step 2-2: Obtain the sparsity ratio through dynamic sparse attention calculation. Then, the actual fusion computation is performed by the Top-k sparse cross-modal attention module.

5. The dynamic sparse cross-modal fusion data feature extraction method according to claim 4, characterized in that, Step 2-1 includes: Step 2-1-1: Calculate the global correlation between modes to form a feature correlation matrix. F , of which elements Representing modes i With mode j The overall correlation strength between them Representing modes i The input feature matrix, Representing modes j The input feature matrix, T It is the transpose matrix; Step 2-1-2: Calculate the correlation matrix F After vectorization, the data is input into a multilayer perceptron to predict an initial sparsity ratio. ; Step 2-1-3: Apply the proximal gradient optimization algorithm to... By applying constraints, the final dynamic sparsity ratio is obtained: ； in, It is the final dynamic sparsity ratio. It is the initial dynamic sparsity ratio. The symbolic function representing element-wise operations. It is the sparse constraint strength parameter.

6. The dynamic sparse cross-modal fusion data feature extraction method according to claim 4, characterized in that, Step 2-2 includes: Step 2-2-1: Aligned Feature Matrix After linear transformation, standard query vector, key vector, and value vector are obtained; Step 2-2-2: Through a learnable correlation matrix U Calculate the raw attention score between the query vector and the key vector. ,in, It is the Sigmoid activation function. T It is the transpose matrix; It is modal i query vectors It is modal j The key vector, It is modal j The value vector; Step 2-2-3: Based on the final dynamic sparsity ratio, extract the values ​​from the original attention score matrix using a Top-k sparse cross-modal attention module. S ij Select the most important value from the set and generate a binary sparse mask. M ij In the mask, the positions with a value of 1 correspond to key interactions that need to be retained, and the positions with a value of 0 correspond to redundant interactions that are discarded. Step 2-2-4: Combine the standard attention calculation result with the sparse mask. M ij Element-wise multiplication yields sparsed attention weights. A ij And calculate the weighted fusion features.

7. The method for feature extraction of dynamic sparse cross-modal fusion data according to claim 1, characterized in that, Step 3 includes: Step 3-1: For each modality, the fused features Z i By selectively extracting metadata through a gating mechanism, noise can be filtered out, and the core modal data can be extracted. m i ； Step 3-2: Collect core data for all modalities Construct it into a graph structure, where M Each modality is represented by a graph neural network, and the edge weights between nodes are determined by the similarity of the data. The graph neural network is used for information propagation and aggregation, so that the representation of each node incorporates information from other modalities, and finally generates a hierarchical data vector m′.

8. A dynamic sparse cross-modal fusion data feature extraction system, characterized in that, It includes a modal feature encoder, a dynamic sparse fusion layer, a data encoder, and a task adapter. The components are connected through standardized data interfaces to form an end-to-end processing pipeline. A modal feature encoder is used to extract low-dimensional features from raw data using a pre-trained model and output a feature matrix with uniform dimensions. The dynamic sparse fusion layer includes a proximal operator sparse controller and a Top-k sparse cross-modal attention module, which outputs a filtered cross-modal feature representation. The data encoder adopts a two-level processing architecture. First, it extracts modality-specific data through a gating mechanism, and then uses a graph neural network to aggregate cross-modal data, ultimately generating a unified semantic representation vector. The task adapter is used to configure the output layer according to the requirements of downstream tasks, supports flexible task expansion, and adopts a hybrid expert architecture to achieve parallel processing. The proximal operator sparsity controller is a dynamic sparsity adjustment module, which includes calculating the correlation matrix for intermodal feature correlation, generating initial sparsity ratio candidate values ​​using a multilayer perceptron (MLP), and applying L1 constraints to the proximal optimization to obtain the final sparsity ratio. The Top-k sparse cross-modal attention module is the core attention calculation module. It guarantees that the Query projection converts the input features into query vectors, the Key projection converts the input features into key vectors, the Value projection converts the input features into value vectors, the sparse mask generator generates Top-k masks according to the instructions of the near-end operator sparse controller, and the attention calculation performs the sparsified attention weight calculation. The hierarchical data encoder is the data generation module, which includes gated latent projection to extract modality-specific data and cross-modal graph aggregation to integrate multimodal data. The hierarchical data encoder achieves selective data retention and cross-modal integration of graph structures through a gating mechanism.

Images