Dynamic sparse cross-modal fusion data feature extraction method and system

Abstract

The invention discloses a dynamic sparse cross-modal fusion data feature extraction method and system, and the method comprises the steps: multi-modal feature coding and alignment, and dynamic sparse cross-modal fusion, which are cooperatively completed by a near-end operator sparse controller and a Top-k sparse cross-modal attention module. After dynamic sparse fusion, features containing highly concentrated cross-modal information are obtained, and through a hierarchical data encoder module, the features of the cross-modal information are converted into data representation with a compact structure and rich semantics; a hybrid expert architecture is adopted; the system comprises a modal feature encoder, a dynamic sparse fusion layer, a data encoder and a task adapter. According to the invention, the calculation complexity of traditional cross-modal fusion is greatly reduced; the data has global consistency while keeping fine-grained details; and meanwhile, the richness of cross-modal semantics is kept, and the method is suitable for feature extraction tasks of multi-modal data such as images, audios and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data processing, and particularly relates to a dynamic sparse cross-modal fusion data feature extraction method and system. BACKGROUND

[0002] With the rapid development of multi-modal large language models, cross-modal feature fusion has become a key technology to improve model performance. Traditional cross-modal fusion methods mainly rely on dense fully connected attention mechanisms or simple concatenation operations. Although these methods can capture global associations between modalities, they have high computational complexity and high resource consumption. Especially when dealing with high-dimensional data such as images and audio, the computational overhead of dense attention mechanisms grows quadratically, severely limiting the scalability of the model. In recent years, sparse attention mechanisms have been introduced to alleviate computational pressure. For example, methods based on total variation sparse attention achieve structured sparsity through proximal operators; generalized structured sparse functions are used for cross-modal metric learning. However, these methods usually use static sparse patterns, which cannot dynamically adjust the sparsity ratio according to the semantic complexity of the input data, resulting in limited performance in heterogeneous data scenarios. In addition, existing sparse cross-modal attention mechanisms often lack explicit modeling of hierarchical data representation, making it difficult to generate compact and semantically rich cross-modal features. In terms of data representation, existing research uses large language models to extract data from heterogeneous data, but does not fully consider the sparsity of cross-modal interaction; there are also methods to enhance the discoverability of data through language models, but their representation ability is limited by the redundant computation of dense fusion. In addition, although sparse interactive attention networks are optimized for multi-modal sentiment recognition, they do not address the problem of dynamic sparse routing and data encoding collaboration.

[0003] The limitations of existing technologies mainly lie in the following three aspects: static sparsity: the sparse pattern of existing sparse attention mechanisms is fixed and cannot adapt to the dynamic needs of different input complexities; data representation deficiency: traditional methods do not combine sparse interaction with hierarchical data encoding, resulting in loss of semantic information; computational efficiency bottleneck: the high computational cost of dense cross-modal fusion limits its application in large models. Therefore, a data feature extraction method is needed that can overcome the above limitations. SUMMARY

[0004] The purpose of the present application is to provide a dynamic sparse cross-modal fusion data feature extraction method and system.

[0005] Technical solution: The dynamic sparse cross-modal fusion data feature extraction method according to the present application comprises the following steps: Step 1: Multi-modal feature encoding and alignment, converting original multi-modal data of different sources and different structures into unified feature representations for subsequent processing; Step 2: Dynamic sparse cross-modal fusion, which is completed by the two modules of the proximal operator sparse controller and the Top-k sparse cross-modal attention module; Step 3: After dynamic sparse fusion, the features containing highly concentrated cross-modal information are obtained, and the hierarchical data encoder module is used to convert the cross-modal information features into a compact structure and rich semantic data representation; Step 4: A hybrid expert architecture is used to instantiate multiple parallel dynamic sparse fusion layers as "experts". Each expert is designed to handle a specific combination of modal interactions. Through a sparse gating network, the system activates only the most relevant experts for calculation according to the modal type of the input data.

[0006] Further, the step 1 comprises: Step 1-1: For each input modality, a pre-trained model is used as a feature encoder for processing; Step 1-2: After processing by the respective encoder, the original data is converted into a high-dimensional feature vector. Through a learnable linear projection layer, the feature vectors of all modalities are mapped to the same shared semantic subspace to obtain a feature matrix wherein m represents the modality index, n is the sequence length, d is the aligned feature dimension, R is the feature vector set.

[0007] Further, the input modalities of step 1-1 include text, image, and audio.

[0008] Further, the step 2 comprises: Step 2-1: Before attention calculation, the proximal operator sparse controller intelligently determines the "sparsity degree" of the current interaction; Step 2-2: Through dynamic sparse attention calculation, the sparse ratio is obtained, and then the Top-k sparse cross-modal attention module performs actual fusion calculation.

[0009] Further, the step 2-1 comprises: Step 2-1-1: Calculate the global correlation between modalities to form a feature correlation matrix F wherein the element represents the overall correlation strength between modality i and modality j , the input feature matrix of modality , the input feature matrix of modality i , the transposed matrix. j T is the transposed matrix.​​ Step 2-1-2: Convert the correlation matrix F After vectorization, input a multi-layer perceptron to predict an initial sparse ratio ; Step 2-1-3: Apply the proximal gradient optimization algorithm to Constraints are applied to obtain the final dynamic sparsity ratio: ; in, is the final dynamic sparsity ratio, is the initial dynamic sparsity ratio, represents an element-wise sign function, is the sparsity constraint strength parameter.

[0010] Furthermore, the step 2-2 includes: Step 2-2-1: Aligned feature matrix After linear transformation, the standard query vector, key vector and value vector are obtained; Step 2-2-2: Through the learnable correlation matrix U , calculate the raw attention score between the query vector and the key vector ,in, is the Sigmoid activation function, T is the transposed matrix; is modal i The query vector, is modal j The key vector of is modal j A vector of values ​​of ; Step 2-2-3: According to the final dynamic sparse ratio, the original attention score matrix is ​​obtained through the Top-k sparse cross-modal attention module S ij Select the most important value and generate a binary sparse mask M ij , the positions with a value of 1 in the mask correspond to key interactions that need to be retained, and the positions with a value of 0 correspond to redundant interactions that are discarded; Step 2-2-4: Compare the standard attention calculation results with the sparse mask M ij Perform element-wise multiplication to obtain sparse attention weights A ij , and calculate the weighted fusion features.

[0011] Furthermore, the step 3 includes: Step 3-1: For each modality’s fused features Zi , selectively extracts its meta-information through a gating mechanism, thereby filtering out noise and extracting the core data of the modal m i ； Step 3-2: The core data of all modalities is constructed into a graph structure, where M represent each modality, and the edge weights between nodes are determined by the similarity of the data. The information is propagated and aggregated using a graph neural network, so that the representation of each node is integrated with the information of other modalities, and finally a hierarchical data vector m' is generated.

[0012] The dynamic sparse cross-modal fusion data feature extraction system described in the application comprises a modal feature encoder, a dynamic sparse fusion layer, a data encoder and a task adapter, and the components are connected through standardized data interfaces to form an end-to-end processing pipeline. The modal feature encoder is used to extract low-dimensional features of the original data using a pre-trained model, and outputs a feature matrix with uniform dimensions. The dynamic sparse fusion layer includes a proximal operator sparse controller and a Top-k sparse cross-modal attention module, and is used to output the filtered cross-modal feature representation. The data encoder adopts a two-level processing architecture, first extracts modality-specific data through a gating mechanism, and then aggregates cross-modal data using a graph neural network to finally generate a unified semantic representation vector. The task adapter is used to configure the output layer according to the downstream task requirements, supports flexible task expansion, and realizes parallel processing using a hybrid expert architecture.

[0013] Advantages: Compared with the prior art, the application has the following significant advantages: the application greatly reduces the computational complexity of traditional cross-modal fusion; the cooperative design of the gating mechanism and the graph aggregation enables the data to retain fine-grained details while maintaining global consistency; the introduction of proximal gradient optimization enables continuous sparse ratio adjustment within the range of 0.1-0.9, greatly shortens the response time, and the hybrid expert architecture supports parallel scheduling of up to 256 modal experts; the sparse gating network stabilizes the expert activation rate between 15%-20%, greatly improves the linear speedup ratio on multiple node clusters; significantly reduces the computational overhead while maintaining the richness of cross-modal semantics, and is suitable for feature extraction tasks of image, audio and other multi-modal data. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a multi-modal large language model system architecture of the application; Figure 2 is a detailed structure diagram of the dynamic sparse fusion layer of the application. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0016] The dynamic sparse cross-modal fusion data feature extraction method of the present invention comprises the following steps: Step 1: Multimodal feature encoding and alignment, converting raw multimodal data from different sources and structures into a unified feature representation that can be used for subsequent processing; Step 1-1: For each input modality (such as text, image, audio, etc.), use a pre-trained model as the feature encoder for processing; use the BERT model to process text data, the ResNet (Residual Network) model to process image data, and the HuBERT (Hidden-Unit BERT) model to process audio data.

[0017] Step 1-2: After passing through their respective encoders, the original data is converted into high-dimensional feature vectors. Through the learnable linear projection layer, the feature vectors of all modalities are mapped into the same shared semantic subspace to obtain the feature matrix ,in m represents the modal index, n is the sequence length, d is the feature dimension after alignment, R is a set of feature vectors.

[0018] Step 2: Dynamic sparse cross-modal fusion, which is accomplished by the collaboration of two modules: the Proximal Operator Sparse Controller (POSC) and the Top-k Sparse Cross-Modal Attention (TkSCMA) module; Step 2-1: Before performing attention calculation, the proximal operator sparse controller (POSC) intelligently determines the "sparseness" of this interaction; Step 2-1-1: Calculate the global correlation between modes to form a feature correlation matrix F , where the elements Indicates modality i With modal j The overall correlation strength between Indicates modality i The input feature matrix, Indicates modality j The input feature matrix, T is the transposed matrix; Step 2-1-2: Convert the correlation matrix F After vectorization, input a multi-layer perceptron to predict an initial sparse ratio ; Step 2-1-3: Apply the proximal gradient optimization algorithm to The constraint is performed to obtain the final dynamic sparse ratio: ; wherein, is the final dynamic sparse ratio, is the initial dynamic sparse ratio, denotes an element-wise sign function, is a sparse constraint strength parameter.

[0019] Step 2-2: After obtaining the sparse ratio through dynamic sparse attention calculation, the actual fusion calculation is performed by the Top-k Sparse Cross-Modal Attention (TkSCMA) module.

[0020] Step 2-2-1: The aligned feature matrix After linear transformation, the standard query vector, key vector and value vector are obtained; Step 2-2-2: The original attention score between the query vector and the key vector is calculated through the learnable correlation matrix U wherein, is a Sigmoid activation function, T is a transposed matrix; is the query vector of the modality i , is the key vector of the modality j , is the value vector of the modality j ; Step 2-2-3: According to the final dynamic sparse ratio, the most important values are selected from the original attention score matrix S ij by the Top-k Sparse Cross-Modal Attention module, and a binary sparse mask M ij is generated, wherein the positions with value 1 in the mask correspond to the key interactions that need to be retained, and the positions with value 0 correspond to the redundant interactions that are discarded; Step 2-2-4: The standard attention calculation result is element-wise multiplied with the sparse mask M ij to obtain the sparse attention weight A ij , and the weighted fusion feature is calculated.

[0021] Step 3: After dynamic sparse fusion, the feature containing highly condensed cross-modal information is obtained, and the hierarchical data encoder (HME) module is used to convert the cross-modal information feature into a compact structure and rich semantic data representation; Step 3-1: For the fused feature of each modality Z ​i , and the core data of the modal is selectively extracted through a gating mechanism to filter out noise and extract the core data of the modal m i ； Step 3-2: The core data of all modalities is constructed into a graph structure, where represents each modality, and the edge weight between nodes is determined by the similarity of the data. The information is propagated and aggregated using a graph neural network to make the representation of each node integrate the information of other modalities, and finally generate a hierarchical data vector m'. M

[0022] Step 4: Hybrid expert architecture is parallelized to improve the scalability and efficiency of the system in large-scale applications. The hybrid expert architecture can be used in this application. Under this architecture, multiple parallel dynamic sparse fusion layers can be instantiated as "experts", each of which can be designed to specifically handle a specific combination of modal interactions (such as a "text-image" expert). Through a sparse gating network, the system can activate only a few most relevant experts for calculation according to the modal type of the input data, and the remaining experts remain dormant, thereby achieving efficient use of computing resources.

[0023] As shown in Figure 1 is a multi-modal large language model system architecture diagram. The input layer receives multi-modal data input, including an original text data input interface, an original image data input interface, an original audio data input interface, and an original video data input interface. The feature extraction module is the feature extraction process for each modality. The text feature extractor is based on BERT / GPT text feature encoding, the image feature extractor is based on ResNet image feature encoding, the audio feature extractor is based on HuBERT audio feature encoding, the video feature extractor is based on 3D-CNN video feature encoding, and the dynamic sparse fusion layer is the core innovative module of the application. The TkSCMA module performs cross-modal sparse attention calculation, the POSC dynamically adjusts the attention sparse ratio, and the HME generates hierarchical data representation. The output layer outputs the final data result, and the data output provides a unified semantic representation for downstream tasks. The dynamic sparse fusion layer replaces the traditional dense fusion mechanism, realizes structured sparse attention through the TkSCMA module, dynamically controls sparsity through the POSC, and innovatively uses the gating mechanism and graph aggregation two-level encoding architecture in the HME.

[0024] As shown in Figure 2 ​The dynamic sparse fusion layer is shown in detail. The Top-k sparse cross-modal attention is a core attention calculation module. The query projection converts the input features into a query vector, the key projection converts the input features into a key vector, the value projection converts the input features into a value vector, the sparse mask generator generates a Top-k mask according to the proximal operator sparse controller instruction, and the attention calculation performs sparse attention weight calculation. The proximal operator sparse controller is a dynamic sparse adjustment module, including a correlation matrix calculation mode inter-feature correlation, a multi-layer perception (MLP) generating an initial sparse ratio candidate value, and a proximal optimization applying an L1 constraint to obtain a final sparse ratio. The hierarchical data encoder is a data generation module, including a gated latent projection extracting mode-specific data and a cross-modal graph aggregation integrating multi-modal data. The Top-k sparse cross-modal attention (TkSCMA) adopts a Top-k sparse mechanism guided by a learnable correlation matrix. The proximal operator sparse controller innovatively applies proximal gradient optimization to sparse ratio control. The hierarchical data encoder (HME) realizes selective data retention and cross-modal integration of graph structure through a gating mechanism.

[0025] Efficient implementation of a multi-modal retrieval system In a cross-modal retrieval scenario, such as "text-to-image" or "image-to-video", the system needs to quickly find the most matching content to the query intent in a large amount of data.

[0026] Feature preprocessing and alignment: For text queries, use models such as BERT to extract semantic features; for content in image or video libraries, use ResNet or 3D-CNN to extract visual features. All features are aligned to a unified 512-dimensional shared semantic space through a linear projection layer.

[0027] Dynamic sparse similarity calculation: When the user inputs the query, the system does not perform dense similarity calculation on all entries in the database. Instead, it uses the Top-k sparse cross-modal attention (TkSCMA) module to perform sparse interaction between the query features and the features of other modalities. The proximal operator sparse controller (POSC) module dynamically adjusts the sparse ratio ρ according to the complexity of the query. For example, for a simple keyword query (such as "beach"), the system uses a higher sparse degree (such as ρ=0.2) and focuses only on the most core semantic concepts; for a complex long sentence query (such as "a girl in a red dress running at sunset"), the sparse degree is reduced (such as ρ=0.7) to capture more detailed interactions.

[0028] Fast data-based matching: The HME module generates a compact 256-dimensional data vector for each entry in the database and the query. This vector is a highly condensed and semantic representation of the original high-dimensional features. The retrieval process is transformed into a fast nearest-neighbor search in this low-dimensional data space, greatly improving the retrieval speed. What will be returned to the user is a list of retrieval results sorted according to the data vector distance.

[0029] Automatic video document generation system This embodiment aims to automatically transform unstructured video data into a structured text report or summary, i.e., a "video document".

[0030] Parallel multi-modal feature extraction: The system receives a video stream input and synchronously processes at least three modalities: 1) visual features (extracting key frames and dynamic information through a spatio-temporal convolutional network); 2) audio features (processing background sound, ambient sound through an audio encoder); 3) text features (transcribing speech into text through automatic speech recognition, ASR technology).

[0031] Event-driven sparse fusion: The Top-k Sparse Cross-Modal Attention (TkSCMA) module is responsible for fusing the feature streams of these three modalities. The fusion process is event-driven, for example, the system will focus on enhancing the correlation between the speaker's facial features, their voice features, and the spoken text. At the same time, it will also strongly associate specific events in the picture (such as the turning of PPT in a course, the display of charts in a meeting) with the corresponding voice explanation content.

[0032] Scene-aware sparse control: The Proximal Operator Sparse Controller (POSC) module controls the sparsity of the fusion according to the dynamics of the video content. In a long single-person speech (static scene), the interaction between vision and text can be very sparse (low ρ) to save computing resources; while in a multi-person discussion or a rapidly changing scene (dynamic scene), the system will automatically reduce the sparsity (high ρ) to ensure that all key cross-modal information is captured.

[0033] Hierarchical data-driven content generation: The Hierarchical Data Encoder (HME) module is the key to achieving high-quality document generation. It encodes the fused feature stream into a hierarchical data structure with time sequence. This structure may contain a series of event triples, such as [<timestamp: 00:15:30>, <event: "chart example">, <main speaker: "Professor Li">, <core content: "showed a chart showing year-on-year growth in sales">]. Finally, a large language model takes this structured data stream as input and constraint to generate a text report that is logically clear, focused, and coherent in content.

[0034] The dynamic sparse cross-modal fusion data feature extraction system disclosed by the application is characterized in that a feature fusion process in a traditional multi-modal large language model (MMLM) is reconstructed through a structured sparse mechanism, including a modal feature encoder, a dynamic sparse fusion layer, a data encoder and a task adapter, each component is connected through a standardized data interface to form an end-to-end processing pipeline, and the calculation efficiency and semantic representation ability are improved together. In terms of technical value, the application first combines dynamic structured sparse learning with hierarchical data encoding, solving the contradiction between static sparsity and semantic loss in traditional methods. The Top-k sparse cross-modal attention (TkSCMA) module realizes accurate screening of cross-modal interaction through a learnable binary mask, the near operator sparse controller uses near gradient optimization to dynamically adjust the sparsity, and the hierarchical data encoder (HME) module builds a progressive representation pipeline from raw features to high-level semantics. This design automatically balances the calculation overhead and representation quality according to the complexity of the input data, providing a new technical path for multi-modal data processing.

[0035] The modal feature encoder is used to extract low-dimensional features of the original data by using a pre-trained model, and outputs a feature matrix with uniform dimensions; The dynamic sparse fusion layer includes a near operator sparse controller (POSC) and a Top-k sparse cross-modal attention (TkSCMA) module, and is used to output filtered cross-modal feature representations; The data encoder adopts a two-level processing architecture, first extracts modal-specific data through a gating mechanism, then aggregates cross-modal data using a graph neural network, and finally generates a unified semantic representation vector; The task adapter is used to configure the output layer according to the requirements of the downstream task, supports flexible task expansion, and realizes parallel processing using a hybrid expert architecture.

Claims

1. A dynamic sparse cross-modal fusion data feature extraction method, characterized in that: The steps include: Step 1: Multimodal feature encoding and alignment, converting raw multimodal data from different sources and structures into a unified feature representation that can be used for subsequent processing; Step 2: Dynamic sparse cross-modal fusion, which is accomplished by the collaboration of two modules: the proximal operator sparse controller and the Top-k sparse cross-modal attention module; Step 3: After dynamic sparse fusion, features containing highly concentrated cross-modal information are obtained. Through the hierarchical data encoder module, the features of cross-modal information are converted into a compact and semantically rich data representation; Step 4: Adopt a hybrid expert architecture and instantiate multiple parallel dynamic sparse fusion layers as "experts". Each expert is designed to specialize in processing a specific combination of modal interactions. Through a sparse gating network, the system activates only the most relevant experts for calculation based on the modal type of the input data.

2. The dynamic sparse cross-modal fusion data feature extraction method according to claim 1 is characterized in that: The step 1 comprises: Step 1-1: For each input modality, use the pre-trained model as the feature encoder for processing; Step 1-2: After passing through their respective encoders, the original data is converted into high-dimensional feature vectors. Through the learnable linear projection layer, the feature vectors of all modalities are mapped into the same shared semantic subspace to obtain the feature matrix ,in m represents the modal index, n is the sequence length, d is the feature dimension after alignment, R is a set of feature vectors.

3. The dynamic sparse cross-modal fusion data feature extraction method according to claim 2, characterized in that: The input modalities in step 1-1 include text, image, and audio.

4. The dynamic sparse cross-modal fusion data feature extraction method according to claim 1, characterized in that: The step 2 includes: Step 2-1: Before performing attention calculation, the proximal operator sparse controller intelligently determines the "sparseness" of this interaction; Step 2-2: Obtain the sparse ratio through dynamic sparse attention calculation Finally, the actual fusion calculation is performed by the Top-k sparse cross-modal attention module.

5. The dynamic sparse cross-modal fusion data feature extraction method according to claim 4 is characterized in that: The step 2-1 includes: Step 2-1-1: Calculate the global correlation between modes to form a feature correlation matrix F , where the elements Indicates modality i With modal j The overall correlation strength between Indicates modality i The input feature matrix, Indicates modality j The input feature matrix, T is the transposed matrix; Step 2-1-2: Convert the correlation matrix F After vectorization, input a multi-layer perceptron to predict an initial sparse ratio ; Step 2-1-3: Apply the proximal gradient optimization algorithm to Constraints are applied to obtain the final dynamic sparsity ratio: ； in, is the final dynamic sparsity ratio, is the initial dynamic sparsity ratio, represents an element-wise sign function, is the sparsity constraint strength parameter.

6. The dynamic sparse cross-modal fusion data feature extraction method according to claim 4, characterized in that: The step 2-2 includes: Step 2-2-1: Aligned feature matrix After linear transformation, the standard query vector, key vector and value vector are obtained; Step 2-2-2: Through the learnable correlation matrix U , calculate the raw attention score between the query vector and the key vector ,in, is the Sigmoid activation function, T is the transposed matrix; is modal i The query vector, is modal j The key vector of is modal j A vector of values ​​for ; Step 2-2-3: According to the final dynamic sparse ratio, the original attention score matrix is ​​obtained through the Top-k sparse cross-modal attention module S ij Select the most important value and generate a binary sparse mask M ij , the positions with a value of 1 in the mask correspond to key interactions that need to be retained, and the positions with a value of 0 correspond to redundant interactions that are discarded; Step 2-2-4: Compare the standard attention calculation results with the sparse mask M ij Perform element-wise multiplication to obtain sparse attention weights A ij , and calculate the weighted fusion features.

7. The dynamic sparse cross-modal fusion data feature extraction method according to claim 1, characterized in that: The step 3 comprises: Step 3-1: For each modality’s fused features Z i , selectively extract its meta-information through the gating mechanism, thereby filtering out noise and extracting the core data of the modality m i ； Step 3-2: Convert all modals to core data Construct a graph structure, where M Representing each modality, the edge weights between nodes are determined by the similarity of the data. Graph neural networks are used for information propagation and aggregation, so that the representation of each node is integrated with the information of other modalities, and finally a hierarchical data vector m′ is generated.

8. A dynamic sparse cross-modal fusion data feature extraction system, characterized by: It includes a modal feature encoder, a dynamic sparse fusion layer, a data encoder, and a task adapter. Each component is connected through a standardized data interface to form an end-to-end processing pipeline. Modal feature encoder, which uses a pre-trained model to extract low-dimensional features of the original data and outputs a feature matrix with unified dimensions; Dynamic sparse fusion layer, including proximal operator sparse controller and Top-k sparse cross-modal attention module, is used to output filtered cross-modal feature representation; The data encoder uses a two-stage processing architecture to first extract modality-specific data through a gating mechanism, then aggregate cross-modal data using a graph neural network, and finally generate a unified semantic representation vector; The task adapter is used to configure the output layer according to the requirements of downstream tasks, supports flexible task expansion, and adopts a hybrid expert architecture to achieve parallel processing.

Images