Hierarchical self-adaptive multi-modal fusion method and system for multi-modal financial large model

Abstract

The invention discloses a hierarchical adaptive multi-modal fusion method and system for a multi-modal financial large model, and the method comprises the steps: collecting financial text data, financial image data and time series data, and carrying out the preprocessing; performing modal feature extraction including text features, image features and time sequence features, and generating a text feature vector, an image feature vector and a time sequence feature vector; designing a full connection layer for each modal feature, then connecting a Sigmoid activation function, generating a gating weight vector, and obtaining a weighted feature vector; carrying out information interaction between modes by adopting a multi-head attention mechanism to form a feature vector after interaction enhancement; and constructing a financial domain knowledge graph, generating a domain knowledge vector, and carrying out vector splicing on the feature vector after interaction enhancement and the domain knowledge vector to generate a fusion feature vector. The method aims at improving the understanding and analysis ability of a large language model in application scenes such as financial analysis, risk assessment and market prediction.

Description

Technical Field

[0001] This invention relates to an adaptive multimodal fusion method and system, and more particularly to a hierarchical adaptive multimodal fusion method and system for large-scale multimodal financial models. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially in the field of Natural Language Processing (NLP), large language models (LLMs) such as BERT (Bidirectional Encoder Representations from Transformers) and the GPT series (Generative Pre-trained Transformers) have achieved remarkable results in text generation, semantic understanding, and question-answering systems. These models, through large-scale pre-training and deep neural network architectures, can efficiently process and generate natural language text, and are widely used in search engines, intelligent customer service, content recommendation, and many other fields. However, the data types in the financial industry are complex and diverse, encompassing multiple modalities such as text, images, and time-series data. For example, financial statements, market trend charts, and historical stock price data commonly used in financial analysis all belong to different data modalities. Single-modal data processing methods are insufficient to comprehensively capture and analyze the multi-dimensional characteristics of financial information, limiting the effectiveness of large language models in the financial field. Existing technologies have seen some research attempts to introduce multimodal data into large language models to enhance their performance in complex tasks, but these are mainly applied to general domains and lack in-depth optimization and customization for the financial sector. Furthermore, specialized NLP models for the financial domain are pre-trained specifically on financial text to improve performance in tasks such as financial text classification and sentiment analysis. These models enhance their understanding of financial terminology and context by incorporating proprietary corpora and tags specific to the financial domain. However, these models typically only process textual modalities and do not consider integrating other types of data sources, such as financial charts and market transaction data, resulting in limited effectiveness in multimodal financial analysis tasks.

[0003] Existing technologies have several shortcomings in multimodal data fusion and financial NLP models, including: insufficient modal integration: existing multimodal models such as VisualBERT and VL-BERT are mainly limited to the fusion of text and image modalities, failing to comprehensively cover the various data types commonly found in the financial field, such as financial statements, market data, and news information. Financial analysis involves a wider variety of data modalities, and single or dual-modal fusion methods are insufficient to meet complex analytical needs. Poor domain adaptability: general-purpose multimodal models lack a deep understanding of financial expertise, resulting in limited performance in financial tasks. For example, although FinancialBERT has been optimized for financial text, its multimodal fusion capability is still insufficient, failing to fully utilize key information in image and time-series data. Limited data fusion mechanisms: existing methods mostly use simple feature concatenation or basic attention mechanisms for data fusion, failing to fully explore the complex relationships and interactions between different modalities. This limited data fusion mechanism restricts the model's comprehensive understanding and analysis of multimodal data, affecting overall performance. Low processing efficiency: Faced with massive amounts of multimodal financial data, existing models suffer from bottlenecks in computational resources and processing speed, making it difficult to meet the demands of real-time financial analysis. This is especially true in scenarios such as high-frequency trading and market monitoring, where the model's response speed and processing efficiency urgently need improvement. Lack of hierarchical fusion structure: Existing multimodal fusion methods lack a hierarchical structure, hindering deep feature fusion and optimization at multiple levels. This limits the expressive power of fused features and the overall performance of the model, making it difficult to achieve a comprehensive understanding and efficient processing of complex financial data. In summary, while existing technologies have made some progress in multimodal data fusion and financial NLP models, significant shortcomings remain in modality integration, domain adaptability, data fusion mechanisms, processing efficiency, and hierarchical fusion structures. Therefore, a novel multimodal data fusion method is urgently needed to comprehensively improve the application effectiveness and performance of large-scale language models in the financial field. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a hierarchical adaptive multimodal fusion method and system for large-scale multimodal financial models, which can achieve efficient integration and deep fusion of multiple data modalities in the financial field, and improve the understanding and analysis capabilities of large-scale language models in application scenarios such as financial analysis, risk assessment, and market forecasting.

[0005] Technical solution: The present invention provides a hierarchical adaptive multimodal fusion method for large-scale multimodal financial models, the method comprising:

[0006] (1) Collect financial text data, including but not limited to financial statements, market analysis reports, and news information; collect financial image data, including but not limited to financial charts, market trend charts, and technical analysis charts; collect time series data, including but not limited to stock prices, trading volume, and index changes; and preprocess the financial text data, financial image data, and time series data respectively.

[0007] (2) Perform modal feature extraction, including text features, image features and temporal features, and generate text feature vectors, image feature vectors and temporal feature vectors;

[0008] (3) Design a fully connected layer followed by a Sigmoid activation function for each modality feature to generate a gated weight vector and obtain a weighted feature vector; use a multi-head attention mechanism to perform intermodal information interaction to form an enhanced feature vector;

[0009] (4) Construct a knowledge graph in the financial field, generate domain knowledge vectors, and concatenate the enhanced feature vectors with the domain knowledge vectors to generate a fused feature vector.

[0010] (5) Construct a multi-level adaptive multimodal fusion network, and train and optimize it to generate a comprehensive fusion feature vector.

[0011] Preferably, step (1) preprocesses the financial text data, including: using a word segmentation tool to segment the financial text data, removing stop words and irrelevant information, converting the text data to lowercase, performing stemming and word form restoration, and encoding all text data in UTF-8 format;

[0012] The preprocessing of the financial image data includes: adjusting all images of the financial image data to 224x224 pixels, normalizing the image pixel values ​​to the range of [0,1], and performing standardization processing. The image format is JPEG or PNG, and the image resolution is not less than 224x224 pixels.

[0013] The preprocessing of time series data includes: scaling the time series data to the range of [0,1] and smoothing the time series data using a moving average method.

[0014] Preferably, step (2) modal feature extraction is as follows: inputting the preprocessed financial text data into the BERT model, performing text encoding, and obtaining the CLS vector of the last hidden state of the BERT model as the text feature vector; inputting the preprocessed financial image data into the ResNet-50 model, performing image feature extraction, obtaining the global average pooling feature of the second-to-last layer of the ResNet-50 model, and generating the image feature vector; inputting the preprocessed time series data into a two-layer bidirectional LSTM network, performing temporal feature extraction, extracting the final hidden state of the LSTM network, and generating the temporal feature vector.

[0015] Preferably, step (3) includes: each modal feature vector is passed through its corresponding fully connected layer and Sigmoid activation function to output a gate weight vector with the same dimension as the feature vector, and the generated gate weight vector is multiplied element by element with the corresponding modal feature vector to obtain a weighted feature vector.

[0016] Preferably, step (3) employs a multi-head attention mechanism for intermodal information interaction, including: linearly transforming the feature vector of each modality through the corresponding query, key, and value weight matrix; calculating the dot product of the query and key, and performing scaling and Softmax normalization to obtain the attention weight; using the attention weight to perform weighted summation on the value vector to generate the attention output; and adding the attention output to the original weighted feature vector to form the feature vector after interaction enhancement.

[0017] Preferably, step (4) constructs a knowledge graph in the financial field, including nodes and edges, and adopts a two-layer GCN network model. The constructed knowledge graph is input into the GCN model, and after two layers of GCN processing, domain knowledge vectors are generated. The GCN model is trained through supervised learning and optimized using known relationships within the domain.

[0018] Preferably, the vector concatenation formula in step (4) is as follows:

[0019]

[0020] in, K∈R 512 , This indicates a vector concatenation operation.

[0021] Preferably, step (5) constructs a multi-level adaptive multimodal fusion network, which is designed as a three-layer HAMFN through hierarchical stacking. The first layer receives the initial modal features and generates the fusion feature vector of the first layer; the second layer receives the output of the first layer and further fuses the features to generate the fusion feature vector of the second layer; the third layer receives the output of the second layer and finally generates the comprehensive fusion feature vector.

[0022] Preferably, in the hierarchical stacking, residual connection and layer normalization are used in the fusion of each layer. The output feature vector of each layer is obtained by adding the fused feature vector of the current layer and the concatenated feature vector, and then processing it through layer normalization.

[0023] The hierarchical adaptive multimodal fusion system for large-scale multimodal financial models described in this invention includes:

[0024] Data collection and preprocessing module: used to collect financial text data, including but not limited to financial statements, market analysis reports, and news information; collect financial image data, including but not limited to financial charts, market trend charts, and technical analysis charts; collect time series data, including but not limited to stock prices, trading volumes, and index changes; and preprocess the financial text data, financial image data, and time series data respectively.

[0025] Feature extraction module: used to extract modal features, including text features, image features and temporal features, and generate text feature vectors, image feature vectors and temporal feature vectors;

[0026] Feature fusion module: Designs a fully connected layer followed by a Sigmoid activation function for each modality feature to generate a gated weight vector and obtain a weighted feature vector; Employs a multi-head attention mechanism for intermodal information interaction to form an enhanced feature vector.

[0027] Domain knowledge embedding module: used to construct a knowledge graph in the financial domain, generate domain knowledge vectors, and concatenate the enhanced feature vectors with the domain knowledge vectors to generate a fused feature vector;

[0028] Multimodal fusion network building module: used to build a multi-level adaptive multimodal fusion network, and to train and optimize it to generate a comprehensive fusion feature vector.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0030] (1) This invention adopts a hierarchical adaptive multimodal fusion network. Through dynamic weight allocation and cross-modal attention mechanism, feature fusion is performed at multiple levels to fully explore the complex relationships and interaction information between modalities and achieve more accurate data fusion. (2) A domain knowledge embedding mechanism is introduced. Through the embedding of domain knowledge graph, the model is endowed with a deep understanding of financial professional knowledge, which significantly improves the model's performance in application scenarios such as financial analysis and risk assessment, and meets the requirements of high precision and high efficiency. (3) Through dynamic weight allocation and optimized fusion network architecture, the speed and resource utilization of multimodal data processing are significantly improved, supporting real-time financial applications and meeting the needs of financial institutions under real-time requirements such as high-frequency trading and market monitoring. (4) Through cross-modal attention mechanism and hierarchical fusion structure, the expressive power of fusion features is enhanced, enabling the model to capture and understand key information in financial data more comprehensively and accurately, which significantly improves the prediction accuracy and reliability of the model in financial tasks. Attached Figure Description

[0031] Figure 1 This is a flowchart of the method described in this invention. Detailed Implementation

[0032] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0033] A hierarchical adaptive multimodal fusion method for large-scale multimodal financial models, the method comprising:

[0034] (1) Collect financial text data, including but not limited to financial statements, market analysis reports, and news information; collect financial image data, including but not limited to financial charts, market trend charts, and technical analysis charts; collect time series data, including but not limited to stock prices, trading volume, and index changes; and preprocess the financial text data, financial image data, and time series data respectively.

[0035] The text data preprocessing includes:

[0036] Word segmentation and noise reduction: Word segmentation tools, such as WordPiece, are used to segment financial text data such as company financial statements, market analysis reports, and news information to remove stop words and irrelevant information;

[0037] Standardization: Convert all text data to lowercase, perform stemming and lemmatization to reduce the impact of lexical diversity. Ensure all text data is encoded in UTF-8, and that the text length does not exceed the model input limit; if it does, truncate or segment it.

[0038] The image data preprocessing includes:

[0039] Image resizing: All financial-related image data, such as financial charts and market trend graphs, were uniformly resized to 224x224 pixels to meet the input requirements of the pre-trained ResNet model.

[0040] Normalization: Normalize the image pixel values ​​to the range [0,1] and perform standardization processing, such as a mean of 0.485 and a standard deviation of 0.229. Ensure the image format is JPEG or PNG and the image resolution is not less than 224x224 pixels.

[0041] The preprocessing of the time series includes:

[0042] Normalization: The Min-Max normalization method is used to scale financial time series data such as stock prices and trading volumes to the range of [0,1] to eliminate the influence between different units of measurement.

[0043] Smoothing: A moving average method with a window size of 5 is used to smooth the time series data and reduce noise interference. Ensure the time series data is arranged in chronological order and contains no missing values; if missing values ​​exist, use forward imputation or interpolation to complete them.

[0044] (2) Perform modal feature extraction, including text features, image features and temporal features, and generate text feature vectors, image feature vectors and temporal feature vectors;

[0045] The text feature extraction employs a pre-trained BERT model. Preprocessed text data is input into the BERT model, and the CLS vector of the last hidden state is obtained as the text feature vector f1∈R. 768 Use the same word segmentation method and input format as the BERT model to ensure that the feature vector dimension is 768.

[0046] The image feature extraction utilizes a pre-trained ResNet-50 model. Preprocessed image data is input into the ResNet-50 model, and the penultimate layer's global average pooling features are extracted to generate a visual feature vector f2∈R. 2048 .

[0047] The time series feature extraction employs a dual-layer LSTM network, with each layer containing 128 LSTM units. The preprocessed time series data is input into the dual-layer LSTM network, and the final hidden state is extracted as the temporal feature vector f3∈R. 256 The LSTM network uses a bidirectional mode with the Tanh activation function to ensure that the input sequence lengths are consistent; if they are inconsistent, padding or truncation is performed.

[0048] (3) Design a fully connected layer followed by a Sigmoid activation function for each modality feature to generate a gated weight vector and obtain a weighted feature vector; use a multi-head attention mechanism to perform intermodal information interaction to form an enhanced feature vector;

[0049] Dynamic weight allocation mechanism: For each modality feature (text feature, image feature, time series feature), a fully connected layer followed by a sigmoid activation function is designed to generate gating weights. Specifically, each modality feature vector passes through its corresponding fully connected layer and sigmoid activation function, outputting a gating weight vector with the same dimension as the feature vector. These weights are used to adjust the influence of each modality feature.

[0050] g i =σ(W i f i +b i )

[0051] Among them, g i For the gating weight, f i For each modal feature, For bias vectors, is the dimension of the modal feature vector.

[0052] Weighted feature generation: The generated gated weight vector is multiplied element-wise with the corresponding modal feature vector to obtain the weighted feature vector.

[0053] f′ i =g i ☉f i

[0054] Here, ⊙ represents element-wise multiplication.

[0055] Weighting operations can dynamically adjust the importance of each modality feature in the fusion process. The weight matrix and bias vector of each gating network are automatically learned through training, ensuring that the gating weights effectively reflect the importance of each modality feature. The generation of gating weights can flexibly adjust the weights of each modality feature according to different tasks and data characteristics, achieving accurate feature fusion.

[0056] Intermodal Enhancement Feature Generation: Intermodal Interaction Enhancement for Any Two Modal Features f i ′ and f j It employs a multi-head attention mechanism for information interaction.

[0057]

[0058] in, These are the weight matrices for the query, key, and value, respectively, d k is the dimension of the key vector.

[0059] The feature vector for each modality is first linearly transformed using the corresponding query, key, and value weight matrices. Then, attention weights are obtained by calculating the dot product of the query and key, followed by scaling and softmax normalization. Finally, these attention weights are used to perform a weighted summation of the value vectors to generate the attention output.

[0060] The attention output is added to the original weighted feature vector to form the interactively enhanced feature vector.

[0061] f″ i =f′ i +Attention(f′ i f′ j )

[0062] Among them, f i "The feature vector is enhanced by interaction. The cross-modal attention mechanism uses 8 attention heads, each with a dimension of 64, to ensure diverse feature interactions."

[0063] This helps preserve original features while incorporating information from other modalities, enhancing the expressive power of the features. The cross-modal attention mechanism employs eight attention heads, each with a dimension of 64, ensuring diverse feature interactions. The parameters (weight matrix) of the attention mechanism are automatically optimized through training to capture the complex relationships between different modalities.

[0064] (4) Construct a knowledge graph in the financial field, generate domain knowledge vectors, and concatenate the enhanced feature vectors with the domain knowledge vectors to generate a fused feature vector.

[0065] Embedded knowledge graph in the financial field: Constructing a knowledge graph covering financial terminology, indicators, industry relationships, etc. Nodes include companies, industries, financial indicators, etc., and edges represent relationships between entities, such as "belongs to", "publishes", "affects", etc. For example, company A "belongs to" industry B, industry B "publishes" report C, and financial indicator C "affects" company A's profitability.

[0066] A two-layer Graph Convolutional Network (GCN) is employed, with each layer containing 256 hidden units. The first-layer GCN is responsible for capturing the local structural information of nodes, while the second-layer GCN further integrates higher-level structural information.

[0067] The constructed knowledge graph is input into the GCN model. After two layers of GCN processing, domain knowledge vectors are generated. These domain knowledge vectors have 512 dimensions and are used to represent professional knowledge in the financial domain. The GCN model is trained using supervised learning and optimized using known relationships within the domain to ensure that the embedding vectors effectively represent professional knowledge in the financial domain. Appropriate loss functions, such as cross-entropy loss, are used during training to improve the accuracy and expressive power of the knowledge graph embedding.

[0068] Knowledge fusion: The enhanced feature vectors are concatenated with the domain knowledge vectors to generate the final fused feature vector.

[0069]

[0070] in, K∈R 512 , This indicates a vector concatenation operation.

[0071] First, the enhanced feature vectors from each modality are concatenated sequentially into a 2560-dimensional vector. Then, this vector is concatenated with a 512-dimensional domain knowledge vector to obtain a final 3072-dimensional fused feature vector. The concatenated vector is ensured to have a dimension of 3072 to accommodate subsequent multi-head attention processing. During the concatenation process, the order and alignment of the modal features and domain knowledge vectors are maintained to ensure semantic consistency of the fused features.

[0072] (5) Construct a multi-level adaptive multimodal fusion network, and train and optimize it to generate a comprehensive fusion feature vector.

[0073] Hierarchical Stacking: The dynamic weight allocation mechanism, inter-modal interaction enhancement module, and domain knowledge embedding mechanism are stacked in three layers, designed as a three-layer HAMFN (Hierarchical Adaptive Multi-modal Fusion Network), with each layer containing the above three sub-modules. The first layer receives the initial modal features and generates the first layer's fused feature vector. The second layer receives the output of the first layer, further fuses the features, and generates the second layer's fused feature vector. The third layer receives the output of the second layer and finally generates the comprehensive fused feature vector.

[0074] Each layer of the fusion network is trained independently to ensure that features are fully integrated and optimized at different levels.

[0075] By defining multiple financial tasks, including but not limited to risk assessment (classification task), stock price prediction (regression task), and market trend analysis (classification and regression tasks), the loss function is designed as follows:

[0076]

[0077] Among them, L t Let a be the loss function for the t-th task. t For task weights, L reg This represents the regularization loss. The weights 'a' for each task are... t The weights are dynamically adjusted based on the importance and difficulty of the task, and the optimal weight combination is determined through cross-validation.

[0078] Optimization through regularization:

[0079]

[0080] Where λ is the regularization coefficient, M is the number of modes, and λ is set to 0.01 to balance model complexity and generalization ability.

[0081] During the fusion process at each layer, residual connections and layer normalization are applied to ensure that features are fully integrated and optimized at different layers. The output feature vector of each layer is obtained by adding the fused feature vector of the current layer to the concatenated feature vector, followed by layer normalization. Layer normalization is used to stabilize the training process and improve the convergence speed and performance of the model. Residual connections help alleviate the gradient vanishing problem in deep networks, promoting the effective transfer and fusion of features.

[0082]

[0083] Where l represents the current layer number, is the output feature vector of the l-th layer.

[0084] The Adam optimizer was used, with the learning rate set to 1×10⁻⁶. -4 β1 = 0.9, β2 = 0.999. A cosine annealing learning rate scheduling strategy is adopted, with an initial learning rate of 1 × 10⁻⁶. -4 The minimum learning rate is 1×10. -6 The epoch length is 10 epochs. The learning rate is reset at the end of each epoch to avoid getting trapped in local optima.

[0085] This invention also discloses a hierarchical adaptive multimodal fusion system for large-scale multimodal financial models, comprising:

[0086] Data collection and preprocessing module: used to collect financial text data, including but not limited to financial statements, market analysis reports, and news information; collect financial image data, including but not limited to financial charts, market trend charts, and technical analysis charts; collect time series data, including but not limited to stock prices, trading volumes, and index changes; and preprocess the financial text data, financial image data, and time series data respectively.

[0087] Feature extraction module: used to extract modal features, including text features, image features and temporal features, and generate text feature vectors, image feature vectors and temporal feature vectors;

[0088] Feature fusion module: Designs a fully connected layer followed by a Sigmoid activation function for each modality feature to generate a gated weight vector and obtain a weighted feature vector; Employs a multi-head attention mechanism for intermodal information interaction to form an enhanced feature vector.

[0089] Domain knowledge embedding module: used to construct a knowledge graph in the financial domain, generate domain knowledge vectors, and concatenate the enhanced feature vectors with the domain knowledge vectors to generate a fused feature vector;

[0090] Multimodal fusion network building module: used to build a multi-level adaptive multimodal fusion network, and to train and optimize it to generate a comprehensive fusion feature vector.

Claims

1. A hierarchical adaptive multi-modal fusion method for a multi-modal financial large model, characterized in that, The method comprises: (1) collecting financial text data, including but not limited to financial statements, market analysis reports, news information; collecting financial image data, including but not limited to financial charts, market trend charts, technical analysis charts; collecting time series data, including but not limited to stock prices, trading volumes, index changes; respectively pre-processing the financial text data, the financial image data and the time series data; (2) modal feature extraction, including text features, image features and time series features, generating text feature vectors, image feature vectors and time series feature vectors; (3) designing a full connection layer followed by a Sigmoid activation function for each modal feature to generate a gating weight vector and obtain a weighted feature vector; using a multi-head attention mechanism for inter-modal information interaction to form an interaction-enhanced feature vector; (4) constructing a financial domain knowledge graph to generate a domain knowledge vector, and concatenating the interaction-enhanced feature vector with the domain knowledge vector to generate a fusion feature vector; (5) constructing a multi-level adaptive multi-modal fusion network for training and optimization to generate a comprehensive fusion feature vector.

2. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (1) of pre-processing the financial text data comprises: using a word segmentation tool to perform word segmentation processing on the financial text data, removing stop words and irrelevant information, converting the text data to lowercase, performing stem extraction and lemmatization, and all text data encoding formats are UTF-8; The pre-processing of the financial image data comprises: adjusting all images of the financial image data to 224x224 pixels, normalizing the image pixel values to the range of [0, 1], and performing standardization processing, the image format is JPEG or PNG, and the image resolution is not less than 224x224 pixels; The pre-processing of the time series data comprises: scaling the time series data to the range of [0, 1] and using the moving average method to smooth the time series data.

3. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (2) of modal feature extraction comprises: inputting the pre-processed financial text data into a BERT model for text encoding, obtaining the CLS vector of the last hidden state of the BERT model as the text feature vector; inputting the pre-processed financial image data into a ResNet-50 model for image feature extraction, obtaining the global average pooling features of the second last layer of the ResNet-50 model to generate the image feature vector; inputting the pre-processed time series data into a double-layer bidirectional LSTM network for time series feature extraction, extracting the final hidden state of the LSTM network to generate the time series feature vector.

4. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (3) comprises: each modal feature vector passes through its corresponding full connection layer and Sigmoid activation function to output a gating weight vector with the same dimension as the feature vector, the generated gating weight vector is multiplied element by element with the corresponding modal feature vector to obtain a weighted feature vector.

5. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (3) adopts a multi-head attention mechanism to interact inter-modal information, including: linearly transforming the feature vector of each modality through the corresponding query, key and value weight matrix, calculating the dot product of the query and the key, and performing scaling and Softmax normalization to obtain the attention weight; the attention weight is used to weight and sum the value vector to generate an attention output, and the attention output is added to the original weighted feature vector to form an enhanced feature vector after interaction.

6. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (4) constructs a financial field knowledge graph, including nodes and edges, and adopts a two-layer GCN network model to input the constructed knowledge graph into the GCN model, generate a field knowledge vector after two-layer GCN processing, and train the GCN model through supervised learning, and optimize by using known relationships in the field.

7. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The vector splicing formula of the step (4) is as follows: wherein K ∈ R 512 , denotes a vector concatenation operation.

8. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 1, characterized in that, The step (5) constructs a multi-level adaptive multi-modal fusion network, which is designed as a three-layer HAMFN through hierarchical stacking, the first layer receives the initial modality feature to generate a first-layer fusion feature vector; the second layer receives the output of the first layer to further fuse the features and generate a second-layer fusion feature vector; and the third layer receives the output of the second layer to finally generate a comprehensive fusion feature vector.

9. The hierarchical adaptive multi-modal fusion method for multi-modal financial large models according to claim 8, characterized in that, The hierarchical stacking adopts residual connection and layer normalization in the fusion of each layer, and the output feature vector of each layer is obtained by adding the fusion feature vector of the current layer to the splicing feature vector and then performing layer normalization.

10. A hierarchical adaptive multi-modal fusion system for a multi-modal financial large model, characterized in that, It includes: A data collection and preprocessing module is used to collect financial text data, including but not limited to financial statements, market analysis reports, news information, collect financial image data, including but not limited to financial charts, market trend charts, technical analysis charts, and collect time series data, including but not limited to stock prices, trading volumes, and index changes, and respectively preprocess the financial text data, the financial image data and the time series data; A feature extraction module is used to extract modality features, including text features, image features and time series features, to generate text feature vectors, image feature vectors and time series feature vectors; A feature fusion module is used to design a fully connected layer followed by a Sigmoid activation function for each modality feature to generate a gating weight vector and obtain a weighted feature vector; a multi-head attention mechanism is used to interact inter-modal information to form an enhanced feature vector after interaction; A domain knowledge embedding module is used to construct a financial field knowledge graph to generate a field knowledge vector, and the enhanced feature vector after interaction is spliced with the field knowledge vector to generate a fusion feature vector; A multi-modal fusion network construction module is used to construct a multi-level adaptive multi-modal fusion network to generate a comprehensive fusion feature vector through training and optimization. ​

Images