Multi-modal data processing method, related device and system

Abstract

The invention discloses a multi-modal data processing method, a related device and a system, and relates to the field of financial data processing and artificial intelligence combination. The dynamic evolution characteristics of the modal relationship between the multi-modal financial data in the process of updating the initial dynamic graph structure into the current dynamic graph structure are captured, the fusion weight of each modal data is adjusted based on the dynamic evolution characteristics, and the multi-modal financial data are deeply fused. The means of capturing the modal relationship among different modal financial data in real time and adjusting the financial data fusion weight can better fit the frequent fluctuation condition of the financial market; and a cross-modal attention interaction layer in the financial data processing model is adopted to further capture complementary information among modal data, and complementarity and difference among different modal data are deeply mined. In conclusion, the association relationship between the multi-modal financial data is deeply mined, and based on the association relationship, efficient and accurate processing of the multi-modal financial data is realized.

Description

Technical Field

[0001] This application relates to the field of financial data processing combined with artificial intelligence, and in particular to a multimodal data processing method, related apparatus and system. Background Technology

[0002] In the current field of financial data processing, single-modal data modeling is generally used to conduct in-depth analysis and pattern mining of single-type data in financial data in order to solve problems in specific business scenarios.

[0003] However, single-modal modeling has obvious limitations. It does not make sufficient use of other modal data such as corporate announcement texts and customer call audio, resulting in a one-sided information basis when conducting financial analysis and failing to capture the correlation between multiple modal data. Summary of the Invention

[0004] In view of the above problems, this application provides a multimodal data processing method, related apparatus, and system to achieve the goal of deeply mining the correlation between multimodal financial data and efficiently and accurately processing multimodal financial data. The specific solution is as follows:

[0005] The first aspect of this application provides a multimodal data processing method, including:

[0006] The process involves acquiring multimodal financial data to be processed, processing different types of modal data separately, and generating multimodal feature representation tensors corresponding to each type of modal data.

[0007] The modality recognition model is invoked to extract the first modality feature vector of the multimodal feature representation tensor and the first correlation between all modal data. Based on the first modality feature vector and the first correlation, the initial dynamic graph structure is updated to the current dynamic graph structure. The second correlation between the dynamic changes of each modality data during the update process is obtained. Based on the second correlation, structured feature representation information is generated to characterize the dynamic evolution trend of modal relationships between each modality data. The weight adjustment amount of each modality data is generated according to the structured feature representation information, and the fusion weight of the corresponding modality data is adjusted according to the weight adjustment amount to obtain a fusion dataset including multi-dimensional information. All modality data includes the initial multimodal financial data and multimodal financial data. The initial dynamic graph structure is constructed based on the initial multimodal financial data. The current dynamic graph structure is constructed based on the initial multimodal financial data and multimodal financial data.

[0008] The pre-trained financial data processing model is called to process the fused dataset to obtain the financial data processing results. The financial data processing model includes a cross-modal attention interaction layer, which includes a feature extraction sub-neural network constructed for different types of modal data. The feature extraction sub-neural network extracts the second modal feature vectors of different types of modal data in the fused dataset respectively, and uses a bidirectional attention mechanism to calculate the context-aware feature representation representing the relationship between the second modal feature vectors.

[0009] In one possible implementation, the first association relationship includes semantically relevant information and edge update weights. A modality recognition model is invoked to extract the first modality feature vector of the multimodal feature representation tensor, as well as the first association relationship between all modal data. Based on the first modality feature vector and the first association relationship, the initial dynamic graph structure is updated to the current dynamic graph structure, including:

[0010] The modality recognition model is invoked to extract the first modality feature vector from the multimodal feature representation tensor, and the first modality feature vector is used as a new modality data node;

[0011] The semantic information of each modal data node pair is calculated using the association calculation function; each modal data node pair includes a first node pair consisting of any two new modal data nodes, a second node pair consisting of any two initial modal data nodes in the initial dynamic graph structure, and a third node pair consisting of any one new modal data node and any one initial modal data node;

[0012] The update weights of the corresponding edges for each modality data node are calculated using a pre-set update balance coefficient and similarity metric function;

[0013] A weighted adjacency matrix is ​​generated based on semantically relevant information and edge update weights;

[0014] The graph neural network model is invoked to process the weighted adjacency matrix and generate the current dynamic graph structure.

[0015] In one possible implementation, a pre-trained financial data processing model is invoked to process the fused dataset, yielding financial data processing results, including:

[0016] The feature extraction sub-neural networks, which are constructed for different types of modal data in the fused dataset, are invoked to extract modal features for different types of modal data respectively, and the modal features are uniformly embedded into the second modal feature vector;

[0017] A bidirectional attention mechanism is used to process the second modality feature vector into a context-aware feature representation;

[0018] The context-aware feature representations are input into their respective buffer queues, and the context-aware feature representations are retrieved from the buffer queues using a rhythm scheduling function.

[0019] The intrinsic feature representation is extracted and fused from the context-aware feature representation to obtain the fused intrinsic feature representation. The fused intrinsic feature representation is then subjected to a nonlinear transformation to obtain the financial data processing result.

[0020] In one possible implementation, different types of modal data include text data, numerical data, image data, and audio data. The different types of modal data in the multimodal financial data are processed separately to generate multimodal feature representation tensors corresponding to each type of modal data, including:

[0021] When the modal data is text data, the text data is processed into a text feature matrix in a uniform format;

[0022] When the modal data is numerical data, the numerical data is processed into composite normalized data or into regularized data.

[0023] When the modal data is image data, the image data is sequentially processed by noise reduction, grayscale conversion and key visual feature extraction to obtain key visual features;

[0024] When the modal data is audio data, the audio data is processed into structured text;

[0025] The text feature matrix, composite normalized or regularized data, key visual features, and structured text are uniformly converted into corresponding multimodal feature representation tensors.

[0026] In one possible implementation, after calling a pre-trained financial data processing model to process the fused dataset and obtain the financial data processing results, the following is also included:

[0027] Financial data is standardized to obtain standardized financial data.

[0028] A data dictionary and data quality assessment system are established based on standardized financial processing data. The data dictionary and data quality assessment system are stored in the financial data warehouse, and the data quality assessment system is used to monitor the various financial data in the financial data warehouse.

[0029] A second aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the multimodal data processing method described in the first aspect or any implementation thereof.

[0030] A third aspect of this application provides a multimodal data processing device, including at least one processor and a memory connected to the processor, wherein:

[0031] Memory is used to store computer programs;

[0032] The processor is used to execute computer programs to enable the multimodal data processing device to implement the multimodal data processing method of the first aspect or any implementation thereof.

[0033] The fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multimodal data processing method described in the first aspect or any implementation thereof.

[0034] The fifth aspect of this application provides a multimodal data processing system, including a data acquisition module, a data preprocessing module, a data fusion module, a multimodal AI modeling module, and a data standardization and governance module;

[0035] The data acquisition module is used to collect multimodal financial data to be processed from different data sources and store the multimodal financial data in a temporary database;

[0036] The data preprocessing module is used to acquire multimodal financial data, perform customized processing on different types of modal data in the multimodal financial data, and after obtaining the customized processing results, convert the format of the customized processing results into a unified multimodal feature representation tensor.

[0037] The data fusion module is used to fuse multimodal feature representation tensors into initial multimodal data using a multimodal fusion algorithm to obtain a fused dataset. The multimodal fusion algorithm integrates a graph neural network model and an online learning algorithm. The graph neural network model is used to capture the dynamic evolution characteristics of modal relationships between the multimodal feature representation tensors and the initial multimodal data during the fusion process. The online learning algorithm is used to adjust the fusion weights of the modal data during the data fusion process based on the dynamic evolution characteristics of modal relationships to obtain the fused dataset.

[0038] The multimodal AI modeling module is used to process fused datasets using a pre-trained financial data processing model to obtain financial data processing results. The financial data processing model is a neural network model that processes the fused dataset layer by layer using a deep learning framework. The deep learning framework includes a cross-modal attention interaction layer, which constructs different feature extraction sub-neural networks for different types of modal data. Different feature extraction sub-neural networks are used to extract modal features of different types of modal data in the fused dataset. A bidirectional attention mechanism is used to calculate the context-aware feature representation of complementary information between the modal data in the fused dataset.

[0039] The data standardization and governance module is used to standardize the results of financial data processing, obtain standardized financial processing data, establish a data dictionary and a data quality assessment system based on the standardized financial processing data, and store the data dictionary and data quality assessment system in the financial data warehouse. The data quality assessment system is used to evaluate and monitor the financial data in the financial data warehouse.

[0040] In one possible implementation, the deep learning framework includes cross-modal attention interaction, network layers, fusion layers, and hidden layers, as well as a multimodal AI modeling module, specifically used for:

[0041] The cross-modal attention interaction layer is used to construct different feature extraction sub-neural networks for different types of modal data in the fused dataset. Different feature extraction sub-neural networks are used to extract modal features of different types of modal data in the fused dataset. The extracted modal features are uniformly embedded into a modal feature vector. A bidirectional attention mechanism is used to process the modal feature vector into a context-aware feature representation.

[0042] The network layer adopts an asynchronous input processing strategy, setting up different buffer queues and input rhythm control modules for different types of modal data in context-aware feature representation, and adjusting the order of inputting context-aware feature representations into the fusion layer according to the data characteristics of different types of modal data;

[0043] The fusion layer is used to extract and fuse intrinsic feature representations from context-aware feature representations using a self-supervised learning mechanism, resulting in a fused intrinsic feature representation.

[0044] After receiving the fused intrinsic feature representation, the hidden layer performs a nonlinear transformation and outputs the financial data processing results.

[0045] By employing the aforementioned technical solutions, the multimodal data processing method, related apparatus, and system provided in this application offer customized processing of multimodal financial data according to its type, ensuring the data quality of each modality. It captures the dynamic evolution characteristics of the modal relationships between multimodal financial data during the process of updating the initial dynamic graph structure to the current dynamic graph structure, and adjusts the fusion weights of each modality based on these dynamic evolution characteristics, thus deeply integrating the multimodal financial data. This method of capturing the modal relationships between different modal financial data in real time and adjusting the fusion weights better reflects the frequent fluctuations in the financial market. Furthermore, it utilizes a cross-modal attention interaction layer in a pre-trained financial data processing model to further capture complementary information between different modal data, deeply exploring the complementarity and differences between them. In summary, this application deeply explores the correlations between multimodal financial data, thereby achieving efficient and accurate processing of multimodal financial data. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A flowchart illustrating the multimodal data processing method provided in this application;

[0048] Figure 2 Example diagram of the preprocessing flow for the text data provided in this application;

[0049] Figure 3 A schematic diagram illustrating the working principle of the cross-modal attention interaction layer in the financial data processing model provided in this application;

[0050] Figure 4 A schematic diagram of the structure of the multimodal data processing device provided in this application;

[0051] Figure 5 This is a structural diagram of the multimodal data processing system provided in this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] In the field of financial data processing, single-modal modeling is generally used to process financial data.

[0056] Taking credit assessment as an example, traditional unimodal data modeling generally relies solely on financial numerical data. Its working principle mainly involves collecting financial statement information from enterprises or individuals, such as total assets and total liabilities in the balance sheet, and operating income and net profit in the profit statement. Then, statistical analysis methods are used to calculate a series of financial ratios, such as solvency indicators (debt-to-equity ratio, current ratio) and profitability indicators (net profit margin, return on equity). Based on these ratios, the creditworthiness is assessed by comparing them with preset credit scoring standards.

[0057] In the practical application of bank approval of personal housing loans, banks mainly refer to the applicant's financial data such as income and debt to assess the applicant's repayment ability and credit risk, and thus decide whether to grant the loan, the loan amount, and the interest rate.

[0058] However, single-modal modeling, which relies on only a single type of data, cannot capture the interrelationships of multi-dimensional information, thus leading to one-sided decision-making.

[0059] Specifically, unimodal models rely on only a single type of financial data, such as numerical or textual data, and cannot capture the relationships between multi-dimensional information. This may lead to the omission of key information, resulting in insufficient market dynamic analysis and blind spots in risk identification. Examples are used to illustrate the specific manifestations of insufficient market dynamic analysis and blind spots in risk identification.

[0060] Insufficient analysis of market dynamics: Volatility models built solely based on historical price data (numerical type) cannot combine news sentiment (textual type) or supply chain (visual type) to predict black swan events;

[0061] For areas with blind spots in risk identification: In anti-fraud scenarios, relying solely on transaction amount and frequency (structured data) while ignoring abnormal operation patterns in user behavior logs (such as multiple logins within a short period of time) or device fingerprint information (unstructured data).

[0062] In summary, unimodal financial data modeling has significant limitations. Unimodal models underutilize other modalities of data, such as corporate announcements and customer service call audio, resulting in biased information in financial analysis and an inability to comprehensively and deeply understand the true situation of the financial market and customers. For example, when assessing corporate credit risk, unimodal models focus only on numerical financial data, ignoring textual information such as strategic adjustments and major investment decisions potentially contained in corporate announcements, as well as information such as customer repayment willingness and potential risks reflected in customer call audio. The resulting assessment is inaccurate and fails to promptly identify potential credit risks, potentially leading to losses for financial institutions.

[0063] To address the aforementioned issues, this application proposes a multimodal data processing method, related apparatus, and system that can deeply integrate multimodal financial data, fully explore the potential value behind multimodal financial data, provide financial institutions with more accurate and comprehensive decision support, and achieve comprehensive and efficient processing of multimodal financial data.

[0064] Optional, see Figure 1 This application provides a flowchart illustrating a multimodal data processing method. Figure 1 As shown, the multimodal data processing method includes the following steps:

[0065] Step 101: Obtain the multimodal financial data to be processed, process the different types of modal data in the multimodal financial data separately, and generate multimodal feature representation tensors corresponding to each type of modal data.

[0066] It should be noted that the multimodal financial data to be processed is collected from multiple different data sources using multiple different data acquisition methods. These multiple data sources include, but are not limited to, bank trading systems, securities market data systems, and corporate financial statement systems; the multiple different data acquisition methods include, but are not limited to, API (Application Programming Interface) interfaces and web crawlers; the multimodal financial data includes different data types, including but not limited to text, numerical, image, and audio formats; specifically, multimodal financial data includes text-based corporate announcements, numerical transaction amounts, image-based scanned copies of invoices, and audio-based customer call records. The collected multimodal financial data to be processed will first be stored in a temporary database.

[0067] It should also be noted that the multimodal financial data to be processed is collected according to a preset collection time and at preset time intervals. For example, multimodal financial data is collected from multiple different data sources and using multiple different data collection methods at the top of each hour.

[0068] Furthermore, before collecting and processing multimodal financial data, the legitimacy and security of the data source need to be verified. For data sources such as bank trading systems and securities market data systems mentioned above, legitimate data collection permissions can be obtained by conducting security authentication and permission negotiation with the data source provider.

[0069] During the data collection process, customized data collection tasks can be generated based on the data source type and interface specifications.

[0070] For data sources with high real-time requirements, such as securities market data systems, a real-time streaming acquisition method is generally adopted. Specifically, message queue technologies such as Kafka can be used to receive securities market data in real time and quickly transmit it to a temporary database. For data sources with batch data, such as corporate financial statement systems, a timed batch acquisition method is adopted, setting the collection time point at the top of each hour. By writing Python scripts and calling the corresponding API interfaces, corporate financial statement data is collected in batches into a temporary database.

[0071] During the data collection process, it is necessary to verify the completeness and accuracy of the collected data. This involves checking the number of records, the completeness of data fields, and the logical rationality of the data to ensure that the collected data is accurate, complete, and meets the system requirements.

[0072] During data transmission, SSL / TLS (Secure Sockets Layer / Transport Layer Security) encryption protocols can be used to ensure the security of transmitted data and prevent data from being stolen or tampered with.

[0073] Next, we will perform customized processing on different types of modal data in multimodal financial data. Specifically, we can use a modality-aware processing mechanism to customize the processing of different types of modal data. Customizing the processing of multimodal financial data can improve the effectiveness of subsequent fusion and modeling.

[0074] The different types of modal data include text data, numerical data, image data, and audio data. This application employs different processing methods for each type of modal data.

[0075] Text data is processed into a unified format text feature matrix; numerical data is processed into composite normalized or regularized data; image data is processed into key visual features; and audio data is processed into structured text. Finally, a unified format conversion mechanism is used to convert the text feature matrix, composite normalized or regularized data, key visual features, and structured text into a multimodal feature representation tensor.

[0076] Next, we will introduce the process of customizing different types of modal data.

[0077] Text data:

[0078] The natural language processing toolchain in the deep semantic model is used to process text data into a text matrix in a unified format.

[0079] Specifically, the first step is to perform Chinese or multilingual word segmentation tasks to divide the original text sequence into word boundaries;

[0080] Subsequently, perform the word - part tagging operation, map words to their corresponding word - class tags, so as to identify the roles of different words in their respective semantic structures;

[0081] Use a stop - word list to remove semantically irrelevant or statistically redundant words, construct a compact and semantically rich word - vector representation, and finally generate a text feature matrix that conforms to a unified format.

[0082] Exemplarily, refer to Figure 2 , the example diagram of the pre - processing process of the text data provided by this application.

[0083] As Figure 2 shown, first perform Chinese or multi - language word segmentation on the original text data, then perform word - part tagging, stop - word removal, and construction of word - vector representation in sequence. Finally, generate a text feature matrix.

[0084] Specifically, use natural language processing technology to process text data. Specifically, the word - segmentation function of the NLTK (Natural Language Toolkit) library can be used to split the text into individual words.

[0085] For example, for the enterprise announcement text: "Our company held a general meeting of shareholders on October 1, 2024 to discuss the company's future development strategy", through the word - segmentation function of NLTK, this enterprise announcement text can be split into: "Our company", "on", "October 1, 2024", "held", "general meeting of shareholders", "discuss", "company", "future", "development", "strategy" and other words.

[0086] Then perform word - part tagging to clarify the word - class of each word. Specifically, the word - part tagging tool of NLTK can be used to tag the above - mentioned words as "noun", "preposition", "time phrase", "verb", "noun", "verb", "noun", "noun", "noun", "noun", etc.

[0087] Finally, remove stop - words. Use the stop - word list provided by NLTK to remove meaningless stop - words such as "on", "of", "etc.", and extract key information.

[0088] It should be further noted that a rule - based word - segmentation method can be used to replace the word - segmentation tool chain in the natural language processing tool chain of the deep semantic model mentioned above. Specifically, rule - based word - segmentation depends on a pre - set dictionary and segmentation rules for word - segmentation, and has a certain effect on financial data texts with stable specific - domain and professional vocabulary; word - part tagging can also adopt a statistical method, such as the Hidden Markov Model, which can judge the word - class by statistically analyzing the context information of words.

[0089] Numerical data:

[0090] A dual strategy of normalization and standardization is adopted to process numerical data into composite normalized and standardized data.

[0091] Specifically, employing a dual strategy of normalization and standardization can eliminate biases caused by inconsistencies in dimensions. Normalization can compress numerical data to the interval [0,1] using the minimum-maximum scaling formula, as shown below:

[0092] ;

[0093] in, The original data, The minimum value of the data. For the maximum value of the data, This is the normalized data.

[0094] Standardization uses a zero-mean, unit-variance transformation formula to make data exhibit standard normal distribution characteristics. The zero-mean, unit-variance transformation formula is as follows:

[0095] ;

[0096] in, The original data, The mean, Standard deviation This is the standardized data.

[0097] In summary, after normalization and standardization processes, numerical data are obtained as composite normalized and standardized data.

[0098] For example, normalization can be achieved using the Min-Max normalization method, which maps data to the [0,1] interval. Taking a set of transaction amount data [1000,2000,3000] as an example, assuming the minimum value is 1000 and the maximum value is 3000, after Min-Max normalization, the data becomes [0,0.5,1], eliminating the influence of units and making the data comparable.

[0099] It should also be noted that, in addition to using a dual strategy of normalization and standardization to process numerical data into composite normalized and standardized data, regularization methods can also be used to process numerical data into regularized data.

[0100] For example, L1 and L2 regularization can constrain parameters during model optimization, thereby avoiding overfitting, and can also adjust the distribution characteristics of the data to some extent. Additionally, the standardization method can specifically be a fractional-based standardization method, which can transform the data according to fractional values, and can better handle data with outliers.

[0101] Image data:

[0102] Image processing techniques are used to sequentially reduce noise in the image data, convert it to grayscale, and extract key visual features to obtain the key visual features.

[0103] Specifically, the image data is first denoised, which can be achieved by using a Gaussian filter to reduce background interference. Then, the denoised image is converted to grayscale, thus reducing computational complexity by unifying the channel dimensions. Finally, key visual features such as image edges and textures are extracted to represent the information structure. Furthermore, wavelet transform can be used instead of a Gaussian filter in the denoising process. Wavelet transform can analyze images at different frequency scales, enabling more accurate noise removal while preserving edge and detail information.

[0104] For example, the OpenCV library can be used to process image data. For corporate ID image data, noise reduction is performed first using a Gaussian filter algorithm, setting an appropriate Gaussian kernel size and standard deviation to remove noise interference from the image and make the image clearer. Then, the color image is converted to a grayscale image using the OpenCV cv2.cvtColor function to convert the RGB (Red, Green, Blue) color image to a grayscale image for subsequent extraction of key visual features.

[0105] The key visual features extracted can include not only traditional edge and texture features, but also higher-level image features extracted using pre-trained models in deep learning, such as VGG (Visual Geometry Group) and ResNet.

[0106] Audio data:

[0107] Audio data is processed into structured text using speech recognition models, acoustic models, and language models.

[0108] Specifically, a speech recognition model is used to transcribe the audio signal. Then, through joint reasoning using an acoustic model and a language model, the semantic content of the bank in the audio is accurately converted into structured text, which is then used as a supplementary modality in subsequent fusion processing.

[0109] For example, the Speech_Recognition library can be used for speech recognition, and the recognized speech content can be converted into text content; alternatively, speech recognition services such as the Google Web Speech API can be used to convert customer service call audio data into text; the converted text can be processed in the same way as the text data mentioned above, such as word segmentation, part-of-speech tagging, and removal of stop words, to generate structured text.

[0110] After different types of modal data are processed in a customized manner, a unified format conversion mechanism is finally used to convert the text feature matrix, composite normalized or regularized data, key visual features, and structured text into their respective multimodal feature representation tensors.

[0111] For example, the multimodal feature representation tensor constructed through a unified format conversion mechanism is as follows:

[0112] ;

[0113] in, The multimodal feature representation tensor representing the i-th modal data is given. This represents the total number of modes.

[0114] Optionally, different types of multimodal financial data to be processed can be collected from multiple different data sources using multiple different data collection methods. A modality-aware processing mechanism can be used to perform customized processing on different types of modal data to generate multimodal feature representation tensors corresponding to each type of modal data.

[0115] Step 102: Call the modality recognition model to extract the first modality feature vector of the multimodal feature representation tensor and the first correlation between all modal data. Based on the first modality feature vector and the first correlation, update the initial dynamic graph structure to the current dynamic graph structure. Obtain the second correlation between the dynamic changes of each modality data during the update process. Based on the second correlation, generate structured feature representation information to characterize the dynamic evolution trend of modal relationships between each modality data. Generate the weight adjustment amount of each modality data according to the structured feature representation information, and adjust the fusion weight of the corresponding modality data according to the weight adjustment amount to obtain a fusion dataset including multi-dimensional information.

[0116] It should be noted that this step involves fusing the newly acquired multimodal financial data with the initial financial data. During the fusion process, the correlation between the changing modal data is captured in real time, and the fusion weights of each modal data are adjusted in a timely manner to obtain a fused dataset that includes multidimensional information.

[0117] Traditional data fusion methods struggle to adapt to the dynamic nature of financial data. Financial markets are constantly changing, and when fluctuations occur, the relative importance of different modalities of financial data for objectives such as risk assessment changes rapidly. However, traditional data fusion methods cannot capture these changes in a timely manner and adjust the fusion weights between data points. For example, in the stock market, when a sudden and sharp decline occurs, news text data containing information about macroeconomic policy adjustments and negative industry news can significantly impact stock price trends and risk assessment. However, traditional algorithms cannot promptly adjust the weights of news text data, resulting in fused data that fails to effectively reflect market changes, reducing data validity and affecting financial institutions' decision-making.

[0118] The data fusion method of this application first uses a modality recognition model to extract the first modality feature vector and the first correlation relationship between all modality data from the multimodal feature representation tensor.

[0119] The first modality feature vector is a new data feature extracted from the multimodal feature representation tensor, and the first association relationship includes semantic information related to all modal data and edge update weights.

[0120] All modal data includes initial multimodal financial data and newly acquired multimodal financial data. The initial multimodal financial data, which is processed first, exists in the form of an initial dynamic graph structure before the multimodal financial data is integrated. The generation process of the initial dynamic graph structure is as follows:

[0121] First, based on the characteristics of the current financial business scenario, the modal data types of the initial multimodal financial data and the potential relationships between the various modal financial data are identified. A heterogeneous graph structure is then generated based on these different modal data types and relationships. In this heterogeneous graph structure, nodes represent modal data units such as text, numerical values, images, and audio, while edges represent the relationships between different modal data, such as time alignment, business logic coupling, or semantic relevance. Next, a graph neural network model is invoked to process the heterogeneous graph structure into an initial dynamic graph structure. Each node is assigned an attribute representation including its modal feature vector, and each edge is assigned an initial connection weight. This initial quantization weight is used to quantify the correlation strength between the two modal feature vectors connected by the edge.

[0122] Optionally, after the initial dynamic graph structure is generated, newly flowing multimodal financial data is continuously monitored in each time window. When new multimodal financial data flows in, a modality recognition model is used to extract the first modality feature vector from the newly flowing multimodal financial data, which exists as a tensor representing multimodal features. The first modality feature vector is used as the new modality data node in the current dynamic graph structure. Then, the first association relationship between each node in the initial dynamic graph structure and each new modality data node is calculated. The first association relationship includes semantically related information and edge update weights. Specifically, the semantically related information of each modality data node pair can be calculated using an association calculation function, and the edge update weights corresponding to each modality data node pair can be calculated using a pre-set update balance coefficient and a similarity measurement function. Then, a weighted adjacency matrix is ​​generated from the semantically related information and edge update weights, and the weighted adjacency matrix is ​​input into the graph neural network model to obtain the current dynamic graph structure.

[0123] It should be noted that each modal data node pair includes a first node pair consisting of any two new modal data nodes, a second node pair consisting of any two initial modal data nodes in the initial dynamic graph structure, and a third node pair consisting of any one new modal data node and any one initial modal data node. Simply put, the first association relationship includes the association relationship between all nodes in the initial dynamic graph structure and all any two node pairs among all nodes in the new modal data nodes.

[0124] Specifically, a modal recognition model is used to automatically extract modal feature vectors from the multimodal feature representation tensors. The modal feature vectors can be modal categories and embedding representations. The modal feature vectors are used as new modal data nodes. An association calculation function is used to calculate the association relationship between the new modal data nodes and any two node pairs among all nodes in the initial modal data nodes.

[0125] For example, suppose , The term "any two nodes" refers to any two nodes mentioned above. There are three scenarios for any two nodes: the first is where one node is the initial modal data node and the other is the new modal data node; the second is where both nodes are initial modal data nodes; and the third is where both nodes are new modal data nodes. An association computation function is used to measure the semantic relevance between the node pairs. For the first scenario, the semantic relevance between the initial and new modal data nodes is measured using the association computation function to update the edge weights. For the second scenario, when new multimodal financial data flows in or the financial scenario changes, it is necessary to remeasure the semantic relevance between any two initial modal data node pairs to adjust the edge weights, thereby reflecting the real-time changes in the relationships between nodes.

[0126] After calculating the semantic correlation between any two nodes, the edge update weights for each modal data node pair are calculated using the preset update balance coefficient and similarity metric function.

[0127] Specifically, the formula for calculating the edge update weights is as follows:

[0128] ;

[0129] in, Update the edge weights at the current moment. This represents a similarity metric function between any two pairs of nodes. To update the balance coefficient; the updated balance coefficient is a fixed value set in advance.

[0130] Finally, a weighted adjacency matrix is ​​generated using semantically relevant information and edge update weights. This weighted adjacency matrix can better capture the breadth and intensity of structural changes between data nodes of different modalities. The specific expression of the weighted adjacency matrix is ​​as follows:

[0131] ; ;

[0132] in, Indicates the number of modal data nodes. For a moment Distribution of connection strengths among all modal data nodes.

[0133] Optionally, a graph neural network model is invoked to process the weighted adjacency matrix, obtaining the current dynamic graph structure and outputting structured feature representations that characterize the dynamic evolution trend of modal relationships between various modal data. Then, an online learning algorithm is used to process the structured feature representations, generating weight adjustment amounts for each modal data during the fusion process. The fusion weights of the corresponding modal data are adjusted based on the weight adjustment amounts to obtain a fused dataset that includes multi-dimensional information.

[0134] The dynamic updating of the initial graph structure to the current dynamic graph structure serves as the basis for weight adjustments during the fusion of various modalities of financial data. The graph structure consists of nodes, edges, and edge weights. Generating the current dynamic graph structure means that the edge weights have been updated, directly altering the distribution of strong connectivity (association strength) in the graph. Online learning algorithms can capture the dynamic evolution trend of modal relationships between various modalities based on the updates to the association strength between modal nodes in the graph structure. Based on this, the fusion weights of node attributes and various modalities are continuously adjusted during the fusion process, thereby generating a multimodal fusion dataset that better meets the needs of financial scenarios. This provides more accurate and responsive feature inputs for subsequent multimodal data modeling.

[0135] Online learning algorithms can also quickly identify the abrupt changes in value of modal data that were originally underweighted during financial emergencies. For example, news text modalities contain important market sentiment information, and online learning algorithms dynamically increase the weight of news text modalities in the fusion process, improving the expressive power and timeliness of the final multimodal fusion data. Therefore, online learning algorithms can rapidly increase the weight of key modal data, enabling the fused dataset to reflect market dynamics more promptly and accurately.

[0136] The initial dynamic graph structure is constructed from initial modal data, specifically from different modal data and their relationships. When new multimodal financial data flows in, the modality recognition model extracts the data features of the multimodal financial data and measures their semantic relevance with the initial modal nodes in the initial dynamic graph structure to update the update weights of graph nodes and edges. Subsequently, the initial dynamic graph structure is dynamically updated to generate the current dynamic graph structure, specifically node features and a weighted adjacency matrix. These two are input into the dynamic graph neural network model, which uses the message passing mechanism in the dynamic graph neural network model to capture the dynamic relationship between modalities and outputs structured features that reflect the evolution of modal relationships. This provides a basis for the online learning algorithm to adjust the weights of each modality data, thereby achieving deep fusion of multimodal data.

[0137] Specifically, during the processing of the weighted adjacency matrix by the graph neural network model, a second dynamic correlation relationship between each modality of data is obtained. This second correlation relationship is used to characterize the dynamic evolution trend of the modal relationship between each modality of data. Then, based on the captured dynamic evolution trend of the modal relationship between each modality of data, a structured feature representation is generated, which can be the context-aware embedding of nodes and the dynamic distribution law of edge update weights. Then, the online learning algorithm adjusts the fusion weight of each modality of data in real time based on the structured feature representation of the dynamic evolution trend of the modal relationship between each modality of data output by the graph neural network model (specifically, the dynamic graph neural network model).

[0138] The following example illustrates this step:

[0139] First, a modality recognition model is used to extract the modality feature vectors of the multimodal feature representation tensor and the correlation between all modal data. All modal data includes the initial modal data and the newly acquired multimodal financial data. A dynamic graph structure is constructed using all modal data.

[0140] Taking the processing of corporate loan application data as an example, the numerical data from the company's financial statements, the text data of the loan application, the image data of the company's identification documents, and the audio data of the credit assessment of the company's responsible person are each treated as different nodes in the dynamic graph structure. The total assets and total liabilities from the financial statements are treated as numerical nodes, the description of the company's operating status in the loan application text is treated as text nodes, the key information in the company's identification document images is treated as image nodes, and the key audio content in the credit assessment audio is treated as audio nodes. The relationships between these nodes, such as the relationship between the data in the financial statements and the description of the company's operating status in the loan application text, are treated as edges in the dynamic graph structure.

[0141] By analyzing the modal characteristics of newly acquired multimodal financial data and their correlation with other modal data in real time, the attributes of graph nodes and the connection weights of graph edges are dynamically updated. In sudden financial events, the correlation between market data and corporate news text data may change. By analyzing the characteristics of new market data and news text data in real time, such as the correlation between descriptions of the impact of policy adjustments on enterprises in news texts and fluctuations in corporate stock prices in the market, the attributes of graph nodes and the connection weights of graph edges are dynamically adjusted to capture the changing correlations between data modalities in real time.

[0142] By combining online learning algorithms, the system can quickly adapt to new data and business scenarios, and adjust the weights of each modality in a timely manner. A multimodal fusion algorithm based on attention mechanisms, combined with dynamic graph neural networks and online learning algorithms, is employed to deeply fuse text, numerical, image, and audio data, generating a fused dataset containing multi-dimensional information. For example, when processing corporate loan application data, the system fuses numerical data from the company's financial statements, textual data from loan applications, image data from the company's identification documents, and audio data from the credit assessment of the company's responsible person, resulting in fused data that more comprehensively reflects the company's creditworthiness.

[0143] Step 103: Call the pre-trained financial data processing model to process the fused dataset and obtain the financial data processing results.

[0144] It should be noted that the pre-trained financial data processing model is a neural network model that includes multiple modal input layers. This model is a deep learning framework, including cross-modal attention interaction layers, network layers, fusion layers, and hidden layers.

[0145] Next, the specific implementation of this step will be explained according to the processing order of each layer in the pre-trained financial data processing model for the fused dataset.

[0146] The cross-modal attention interaction layer includes a feature extraction sub-neural network built for different types of modal data. The feature extraction sub-neural network is used to extract second modal features of different types of modal data in the fused dataset, embed the second modal features into a unified second modal feature vector, and finally use a bidirectional attention mechanism to calculate the context-aware feature representation between the feature vectors.

[0147] For example, see Figure 3 The working principle diagram of the cross-modal attention interaction layer in the financial data processing model provided in this application.

[0148] like Figure 3 As shown, feature extraction sub-neural networks are constructed for different types of modal data. Recurrent neural networks are used to process text data, and the processing results are then uniformly embedded into vector representations. Multilayer fully connected neural networks are used to process numerical data, and the processing results are then uniformly embedded into vector representations. Convolutional neural networks are used to process image data, and the processing results are then uniformly embedded into vector representations. Delayed neural networks are used to process audio data, and the processing results are then uniformly embedded into vector representations. A bidirectional attention mechanism is used to process the feature extraction sub-neural networks corresponding to different types of modal data to obtain context-aware feature representation fusion.

[0149] We construct feature extraction sub-neural networks for different types of modal data to fully capture the semantic and structural features of different modal data. Specifically, recurrent neural networks are used for sequence modeling of text data to capture the contextual dependencies between text data; multi-layer fully connected neural networks are used for modeling of numerical data to capture the nonlinear distribution patterns between numerical data; convolutional neural networks are used for modeling of image data to extract spatial structural features and edge texture information; and time-delay neural networks are used for modeling of audio data to capture the time delay characteristics and prosodic patterns of speech signals in audio data. It is easy to understand that the semantic and structural features of text data, the nonlinear distribution patterns of numerical data, the spatial structural features and edge texture information of image data, and the time delay characteristics and prosodic patterns of audio data constitute the second modal features mentioned above.

[0150] The second modality feature vector can also be called a higher-order feature vector. The higher-order feature vectors for text, numerical, image, and audio modal data can be represented as follows:

[0151] ;

[0152] Then, a bidirectional attention mechanism is used to process the second modality feature vector into a context-aware feature representation. The bidirectional attention mechanism can enhance the information interaction capability between modalities. Specifically, this mechanism guides feature focusing between different modalities by constructing a mutual attention matrix between modalities. The attention weight from modality x to modality y is defined as follows:

[0153] ;

[0154] Where W is the learnable attention weight matrix, and is the learnable parameter matrix for generating the mutual attention matrix; and Let represent the feature vectors of the meta-mode and the target mode, respectively. The denominator term represents the normalization factor for all target modes, ensuring that the attention weights satisfy the probability distribution property. Learnable attention weight matrix. It is the parameter matrix to be trained in the model, used to perform linear transformation on the vectors of the source mode and the target mode, and is an intermediate tool for calculating attention weights.

[0155] Based on attention weights, the context-aware feature representation of modality x focusing on other modalities is calculated. The calculation formula is as follows:

[0156] ;

[0157] Specifically, the mutual attention matrix consists of attention weights from multiple modes x to mode y. Each element represents the attention weight from the source mode x to the target mode y, expressed by the formula... The calculation shows that the correlation strength between the feature vectors of the source modality x and the feature vectors of the target modality y after transformation by the learnable weight matrix W is reflected. This is used to guide feature focusing between different modalities and realize cross-modal information interaction and semantic enhancement.

[0158] In this context, modality x focuses on the context-aware feature representation of other modalities. Specifically, it involves using a bidirectional attention mechanism to correlate the feature vector of modality x with the feature vectors of other modalities (such as y, z, etc.). After transforming the feature vectors using a learnable attention weight matrix W, the attention weights between modality x and other modalities are obtained through normalization. Based on these weights, information from other modalities that is semantically related to modality x is aggregated into the feature representation of modality x, forming a new feature vector that integrates complementary cross-modal information. This representation not only preserves the semantic and structural features of modality x itself but also incorporates contextual information related to x from other modalities, such as supplementary explanations of financial data in numerical modalities provided by corporate announcements in text modalities.

[0159] Therefore, context-aware feature representation integrates complementary information from other modalities and modality x, achieving deep semantic enhancement. For example, in the task of corporate credit risk assessment, the financial data of the numerical modality focuses on corporate announcements in the textual modality through an attention mechanism. This allows the model to not only identify a company's solvency but also sensitively capture potential clues about the impact of external market sentiment or policy changes on corporate credit, avoiding feature redundancy or information loss caused by modality fragmentation. This significantly improves the model's ability to characterize complex financial behaviors and its predictive accuracy.

[0160] In summary, the cross-modal attention interaction layer uses a learnable attention weight matrix to calculate the attention weights between modalities, thereby achieving cross-modal information fusion.

[0161] After processing the second modality feature vector into a context-aware feature representation using a bidirectional attention mechanism, the context-aware feature representation is input into the network layer.

[0162] The network layer adopts an asynchronous processing strategy, setting different buffer queues and input rhythm control modules for different types of modal data in the context-aware feature representation, and adjusting the order of inputting the context-aware feature representation to the fusion layer according to the data characteristics of different types of modal data.

[0163] The asynchronous input processing strategy is primarily used to coordinate the inconsistencies in modality information from different data sources across time and frequency dimensions. To achieve stable training rhythm and gradient balance, the network layer configures an independent buffer queue for each type of modality data to temporarily store its respective preprocessed feature tensors, ensuring data availability within the input time window. The input rhythm control module sets input intervals for each modality based on its specific data generation cycle and real-time requirements. and input batch size ,in, This represents the input period of the i-th modal data. This represents the number of samples input to this mode each time. The system uses a rhythm scheduling function to control the extraction of samples from the buffer queue of mode i at time step t. Each sample is fed into the neural network model. The expression for the rhythm scheduling function can be:

[0164] ;

[0165] Specifically, this neural network model can be a financial data prediction model or a financial data classification model. In the above process, the asynchronous processing strategy aims to coordinate the input rhythm and frequency of different modalities. An independent buffer queue is configured for each type of modality to temporarily store preprocessed feature tensors. The input rhythm control module sets the input interval and batch size for each modality. Finally, the rhythm scheduling function controls the extraction of corresponding samples from the buffer queue of modality i at time step t, allowing them to be processed by the neural network. This achieves dynamic input scheduling for modality awareness, ensuring gradient stability and convergence efficiency during model training.

[0166] The above-mentioned rhythm scheduling function is used to realize the dynamic input scheduling of modality awareness. This scheduling mechanism can avoid the input imbalance caused by the high-frequency continuous updates of securities market modalities and the low-frequency batch provision of corporate financial statement modalities, thereby reducing the risk of gradient fluctuation.

[0167] In actual training, to maintain the stability of the overall gradient propagation path, the network layers need to calculate the expected gradient of the loss function with respect to different modal inputs. The formula can be as follows:

[0168] ;

[0169] in, This represents the gradient of the loss function for the i-th sample in the j-th mode.

[0170] The above process balances the gradient distribution across modalities, ensuring the continuity of gradient convergence and the stability of model parameter updates during asynchronous cross-modal data flow, thus significantly improving overall training efficiency and final model performance.

[0171] Subsequently, context-aware feature representations are retrieved from the buffer queue according to the asynchronous input processing strategy described above, and then input into the fusion layer according to the input order and frequency determined by the asynchronous input strategy.

[0172] The fusion layer extracts and fuses the intrinsic feature representations from the context-aware feature representations to obtain the fused intrinsic feature representations;

[0173] Finally, the fused intrinsic feature representations are subjected to nonlinear transformation to obtain the processing results of multimodal financial data.

[0174] It should also be noted that before the modal data in the fusion dataset are input into the pre-trained financial data processing model, the modal data needs to be pre-trained using a self-supervised learning method. This will enable the model to fully explore the structural patterns and semantic representations inherent in the multimodal financial data and improve the processing performance.

[0175] Specifically, for the text modality, this application employs a masked language modeling task. By randomly masking portions of the original sentence and training the model to predict the masked words based on the context, it learns semantic associations and syntactic structure information. The text input sequence can be represented as follows:

[0176] ;

[0177] In this model, some positions are replaced with the mask symbol [MASK]. The goal is to maximize the predicted probability of the masked positions, and the loss function can be defined as:

[0178] ;

[0179] Where M represents the set of masked locations; This represents the remaining context sequence after removing the i-th word. This represents the model's conditional probability prediction of the word at that position.

[0180] Image modality employs image mask reconstruction and rotation angle prediction tasks. By randomly occluding local areas of the image or applying rotational perturbations to the image, the model is trained to recover the image structure or determine the rotation angle, thereby capturing spatial structure and global visual features.

[0181] The audio modality constructs a time-axis mask and context prediction task, enabling the model to reconstruct audio sequences or predict their semantic labels even after masking partial acoustic segments, thereby learning the distribution of acoustic features and the relationship between speech context. The output vector of each modality after self-supervised training is denoted as... Where i represents the sample number and m represents the model category, it is used as the initial input feature embedding when formally training the financial data processing model. It effectively bridges the differences in the distribution of the original data, enhances the model's ability to understand and integrate multimodal data, and improves the model's generalization ability and prediction accuracy for complex financial tasks.

[0182] During the self-supervised training phase, different modal data correspond to different models, and different models correspond to different self-supervised tasks. Specifically, text modality corresponds to masked language model, image modality corresponds to mask reconstruction and rotation prediction model, and audio modality corresponds to time axis mask prediction model. The parameters need to be optimized through loss function in order to learn the intrinsic feature representation of each modality data.

[0183] Optionally, feature extraction sub-neural networks constructed for different types of modal data in the fused dataset are invoked to extract modal features for different types of modal data respectively, and the modal features are uniformly embedded into a second modal feature vector. A bidirectional attention mechanism is used to process the second modal feature vector into a context-aware feature representation. The context-aware feature representation is input into its corresponding buffer queue. A rhythm scheduling function is used to retrieve the context-aware feature representation from the buffer queue. The intrinsic feature representation is extracted and fused from the context-aware feature representation to obtain the fused intrinsic feature representation. The fused intrinsic feature representation is then subjected to a nonlinear transformation to obtain the financial data processing result.

[0184] Specifically, the pre-trained financial data processing model employs a deep learning framework, constructing a neural network model with multiple modal input layers. For text data, a recurrent neural network (RNN) is used, leveraging its ability to process sequential data to capture contextual information within the text. When analyzing corporate announcement text, the RNN can learn the semantic relationships between preceding and following sentences. For numerical data, a fully connected neural network is used to effectively process and map numerical features. For image data, a convolutional neural network (CNN) is used, extracting key image features through structures such as convolutional and pooling layers. When processing corporate document images, the CNN can identify key information in the image, such as the company name and document number. For audio data, a time-series neural network is used, specifically designed to process the time-series features of audio data.

[0185] A cross-modal attention interaction layer is added to the neural network model to mine complementary information between features of different modalities through a bidirectional attention mechanism. When assessing corporate credit risk, the features of numerical financial data and the features of corporate announcements in the text modality are mutually influenced. Through this bidirectional attention mechanism, the feature vectors of financial data and corporate announcements learn from each other, preserving the direct reflection of solvency by financial data while capturing supplementary information such as market sentiment that may affect creditworthiness from corporate announcements, reducing feature redundancy and the omission of key information. An asynchronous input processing strategy is adopted, with independent buffer queues and input rhythm control modules for different modalities. Independent buffer queues temporarily store different modalities of data. The input rhythm control module sets different input time intervals and batch sizes based on the real-time requirements and data generation frequency of each modality, such as real-time streaming data from a securities market system and timed batch data from a corporate financial statement system, dynamically adjusting the input order and frequency of different modalities. For real-time streaming data from securities markets, a shorter input time interval is set, such as inputting data once per second; for timed batch data from corporate financial statements, a longer input time interval is set, such as inputting data once per day.

[0186] This study utilizes self-supervised learning methods to pre-train the model on various modalities before actual model training, learning the intrinsic feature representations of the data. For text data, self-supervised tasks such as masked language models are employed to randomly mask parts of the text, allowing the model to predict the masked words and thus learn the semantic features of the text data. For image data, self-supervised tasks such as image masking and rotation prediction are used to learn the visual features of the image data. These pre-trained intrinsic feature representations of each modality are used as the initial features for training the neural network model, improving the model's understanding and fusion capabilities of multimodal data, thereby enhancing its generalization ability and prediction accuracy. The outputs of each modality are integrated through a fusion layer and then subjected to multiple nonlinear transformations in hidden layers, ultimately outputting the prediction or classification results for financial data, such as predicting the probability of loan default for enterprises or classifying the risks of financial products.

[0187] In summary, the multimodal data processing method provided in this application customizes the processing of multimodal financial data according to its type, ensuring the data quality of each modality. It captures the dynamic evolution characteristics of the modal relationships between multimodal financial data during the update from the initial dynamic graph structure to the current dynamic graph structure, and adjusts the fusion weights of each modality based on these dynamic evolution characteristics, thus deeply integrating the multimodal financial data. This method of capturing the modal relationships between different modal financial data in real time and adjusting the fusion weights better reflects the frequent fluctuations in the financial market. Furthermore, it employs a cross-modal attention interaction layer in a pre-trained financial data processing model to further capture complementary information between different modalities, deeply exploring the complementarity and differences between them. In conclusion, this application deeply explores the correlations between multimodal financial data, thereby achieving efficient and accurate processing of multimodal financial data.

[0188] The multimodal data processing method provided in this application also includes the following steps:

[0189] Financial data is standardized to obtain standardized financial data; a data dictionary and a data quality assessment system are established based on the standardized financial data, and the data dictionary and data quality assessment system are stored in a financial data warehouse. The data quality assessment system is used to monitor the various financial data in the financial data warehouse.

[0190] Specifically, based on the regulatory requirements of the financial industry and the company's internal data standards, the data output from the modeling is standardized and governed. A data dictionary is established to clarify the definition, type, format, and value range of each data field. At the same time, a data quality assessment system is established to evaluate and monitor the integrity, accuracy, and consistency of the data. The standardized data after governance is stored in a formal financial data warehouse for use by the financial institution's business and analysis systems.

[0191] For example, based on the regulatory requirements of the financial industry and the company's internal data standards, the data output from the modeling is standardized and governed. A data dictionary is established, clearly defining the definition, type, format, and value range of each data field. For the "Transaction Amount" field, it is defined as the actual monetary amount in a financial transaction, with a numeric data type, a format retaining two decimal places, and a value range greater than 0. A data quality assessment system is established, setting multiple data quality indicators. Data integrity indicators are used to measure whether there are missing values ​​in the data, assessed by statistically analyzing the number and proportion of missing values ​​in the data set. Data accuracy indicators are used to verify whether the data matches the actual situation, verified by comparing with actual business records or authoritative data sources. Data quality reports are generated regularly, and visualization tools such as Tableau or Power BI are used to visualize and analyze data quality. The changing trends of indicators such as data integrity, accuracy, and consistency are displayed in chart form to facilitate timely identification and resolution of data quality issues. The standardized data after governance is stored in a formal financial data warehouse, using Hive data warehouse, leveraging its distributed storage and computing capabilities to store massive amounts of standardized financial data for use by the financial institution's business and analysis systems.

[0192] The beneficial effects achieved by this application will be explained from three aspects: technical performance improvement, business value realization, risk control, and industry development promotion.

[0193] In terms of technical performance improvement, this application significantly enhances the accuracy of data fusion compared to existing technologies. Traditional attention-based multimodal fusion algorithms struggle to adapt quickly to the dynamic nature of financial data. This application, however, introduces dynamic graph neural networks and online learning algorithms, enabling real-time capture of the correlations between data modalities and timely adjustment of the weights of each modality. In the event of sudden financial incidents, it can rapidly increase the weights of relevant modal data, making the fused dataset more effectively reflect market changes. Regarding modeling accuracy and efficiency, this application effectively addresses the problems of insufficient feature mining and unstable training in traditional modeling methods by designing a cross-modal feature collaborative learning mechanism, employing an asynchronous input processing strategy, and utilizing self-supervised learning methods for pre-training.

[0194] In terms of business value, it provides stronger support for financial institutions' business decisions. Through more accurate risk assessment and market forecasting, financial institutions can make more informed investment and loan approval decisions. In loan approval, based on the more comprehensive and accurate corporate credit assessment results provided in this application, financial institutions can more accurately assess loan risks, improve the accuracy of loan approval, and reduce non-performing loan rates.

[0195] In terms of risk control, it enables more timely and accurate identification of potential risks, allowing for proactive preventative measures and reduced losses. Analysis of multimodal data reveals risk clues that are difficult to detect using traditional methods; for example, analyzing customer service call audio data and transaction record text data can promptly identify financial fraud. Regarding customer service, this application helps build more accurate customer profiles, enabling personalized services and improving customer satisfaction. By integrating customer transaction record data, customer service call audio data, and customer feedback text data, it provides a deeper understanding of customer needs and preferences, allowing for the provision of financial products and services that better meet their requirements.

[0196] In terms of driving industry development, this application has played a positive role in promoting the digital transformation and technological innovation of the financial industry. It provides the financial industry with a new and more efficient model for financial data processing and analysis, prompting financial institutions to accelerate their digital transformation and enhance their overall competitiveness. This innovative technical solution and practical experience offer other financial institutions valuable insights and methods for processing multimodal financial data, building intelligent modeling and data governance systems, and contributing to the technological progress and innovative development of the entire financial industry.

[0197] This application also provides a computer program product including computer-readable instructions, which, when executed on a multimodal data processing device, cause the multimodal data processing device to implement any of the multimodal data processing methods provided in this application.

[0198] This application also provides a multimodal data processing device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the multimodal data processing device provided in this application. The multimodal data processing device in this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The multimodal data processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0199] like Figure 4As shown, the multimodal data processing device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the multimodal data processing device is powered on, the RAM 603 also stores various programs and data required for the operation of the multimodal data processing device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0200] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows the multimodal data processing device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 A multimodal data processing apparatus with various means is shown; however, it should be understood that implementation or possession of all the means shown is not required. More or fewer means may be implemented alternatively.

[0201] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the multimodal data processing methods provided in this application.

[0202] This application also provides a multimodal data processing system, which includes a data acquisition module, a data preprocessing module, a data fusion module, a multimodal AI modeling module, and a data standardization and governance module.

[0203] The data acquisition module is used to collect multimodal financial data to be processed from different data sources and store the multimodal financial data in a temporary database.

[0204] The data preprocessing module is used to acquire multimodal financial data, perform customized processing on different types of modal data in the multimodal financial data, and after obtaining the customized processing results, convert the format of the customized processing results into a unified multimodal feature representation tensor.

[0205] The data fusion module is used to fuse multimodal feature representation tensors into initial multimodal data using a multimodal fusion algorithm to obtain a fused dataset. The multimodal fusion algorithm integrates a graph neural network model and an online learning algorithm. The graph neural network model is used to capture the dynamic evolution characteristics of modal relationships between the multimodal feature representation tensors and the initial multimodal data during the fusion process. The online learning algorithm is used to adjust the fusion weights of the modal data during the data fusion process based on the dynamic evolution characteristics of modal relationships to obtain the fused dataset.

[0206] The multimodal AI modeling module is used to process fused datasets using a pre-trained financial data processing model to obtain financial data processing results. The financial data processing model is a neural network model that uses a deep learning framework to process the fused dataset layer by layer. The deep learning framework includes a cross-modal attention interaction layer, which constructs different feature extraction sub-neural networks for different types of modal data. Different feature extraction sub-neural networks are used to extract modal features of different types of modal data in the fused dataset. A bidirectional attention mechanism is used to calculate context-aware feature representations of complementary information between different modal data in the fused dataset. Specifically, the deep learning framework includes cross-modal attention interaction, network layers, fusion layers, and hidden layers. The cross-modal attention interaction layer is used to construct different feature extraction sub-neural networks for different types of modal data in the fused dataset. The feature extraction sub-neural network extracts modal features from different modalities in the fused dataset using different sub-neural networks. These extracted modal features are then uniformly embedded into a modal feature vector. A bidirectional attention mechanism is used to process this modal feature vector into a context-aware feature representation. The network layer employs an asynchronous input processing strategy, setting different buffer queues and input rhythm control modules for different modalities in the context-aware feature representation. The order in which the context-aware feature representations are input to the fusion layer is adjusted according to the characteristics of different modalities. The fusion layer uses a self-supervised learning mechanism to extract and fuse intrinsic feature representations from the context-aware feature representations, resulting in a fused intrinsic feature representation. The hidden layer receives the fused intrinsic feature representation and performs a non-linear transformation, outputting the financial data processing result.

[0207] The data standardization and governance module is used to standardize the results of financial data processing, obtain standardized financial processing data, establish a data dictionary and a data quality assessment system based on the standardized financial processing data, and store the data dictionary and data quality assessment system in the financial data warehouse. The data quality assessment system is used to evaluate and monitor the financial data in the financial data warehouse.

[0208] For example, see Figure 5 The structural composition diagram of the multimodal data processing system provided in this application.

[0209] like Figure 5 As shown, the multimodal data processing system mainly includes a data acquisition module, a data preprocessing module, a data fusion module, a multimodal AI modeling module, and a data standardization and governance module.

[0210] Specifically, the data acquisition module collects multimodal financial data to be processed from multiple different data sources at specified time intervals, such as every hour on the hour, using various multimodal financial data acquisition methods, and stores the multimodal financial data in a temporary database.

[0211] Then, the data preprocessing module immediately processes the multimodal data in the temporary database. For text data, natural language processing techniques are used for word segmentation, part-of-speech tagging, and stop word removal; for numerical data, normalization and standardization are performed to eliminate the influence of units; for image data, image processing techniques are used for noise reduction, grayscale conversion, and feature extraction; for audio data, speech recognition technology is used to convert it into text content. After processing, the preprocessed data in a unified format is transmitted to the data fusion module.

[0212] Then, after receiving the preprocessed data, the data fusion module uses an attention-based multimodal fusion algorithm to deeply fuse text, numerical data, images, audio, and other data based on the characteristics and relationships of different data modalities, generating a fused dataset containing multi-dimensional information. For example, when processing corporate loan application data, it fuses the company's financial statement numerical data, loan application text data, company document image data, and credit assessment audio data of the company's responsible person to obtain fused data that more comprehensively reflects the company's credit status.

[0213] Subsequently, the multimodal AI modeling module utilizes the fused dataset for intelligent modeling. This module employs a deep learning framework to construct a neural network model containing multiple modal input layers. For example, recurrent neural networks are used for text data, fully connected neural networks for numerical data, convolutional neural networks for image data, and time-delay neural networks for audio data. The outputs of each modality are integrated through a fusion layer, and then subjected to multiple nonlinear transformations in hidden layers to ultimately output the prediction or classification results of the financial data, such as predicting the probability of loan default for enterprises or classifying the risks of financial products. In this step, multimodal AI modeling enables in-depth analysis and modeling of financial data, representing an innovative approach to addressing the problem of insufficient utilization of single-modal information in existing financial data modeling.

[0214] Finally, the data standardization and governance module standardizes the data output from modeling based on the regulatory requirements of the financial industry and the company's internal data standards. It establishes a data dictionary to clarify the definition, type, format, and value range of each data field. At the same time, it establishes a data quality assessment system to evaluate and monitor the integrity, accuracy, and consistency of the data. The standardized data after governance is stored in the formal financial data warehouse for use by the financial institution's business and analysis systems.

[0215] The specific implementation details of each of the above modules have been elaborated in the section on multimodal data processing methods above, so they will not be repeated here.

[0216] In summary, the multimodal data processing system provided in this application establishes an efficient data governance system that can ensure the quality of multimodal financial data and meet the stringent regulatory requirements of the financial industry as well as the ever-growing business needs.

[0217] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0218] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0219] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0220] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A multimodal data processing method, characterized in that, include: Acquire multimodal financial data to be processed, process different types of modal data in the multimodal financial data separately, and generate multimodal feature representation tensors corresponding to each type of modal data; The modality recognition model is invoked to extract the first modality feature vector of the multimodal feature representation tensor and the first correlation relationship between all modal data. Based on the first modality feature vector and the first correlation relationship, the initial dynamic graph structure is updated to the current dynamic graph structure. The second correlation relationship of dynamic changes between each modality data during the update process is obtained. Based on the second correlation relationship, structured feature representation information is generated to characterize the dynamic evolution trend of modal relationships between each modality data. The weight adjustment amount of each modality data is generated according to the structured feature representation information, and the fusion weight of the corresponding modality data is adjusted according to the weight adjustment amount to obtain a fusion dataset including multidimensional information. All modality data includes the initial multimodal financial data and the multimodal financial data. The initial dynamic graph structure is constructed based on the initial multimodal financial data; The current dynamic graph structure is constructed based on the initial multimodal financial data and the multimodal financial data; The fused dataset is processed by calling a pre-trained financial data processing model to obtain the financial data processing results; The financial data processing model includes a cross-modal attention interaction layer, which includes a feature extraction sub-neural network constructed for the different types of modal data. The feature extraction sub-neural network extracts the second modal feature vectors of the different types of modal data in the fused dataset, and uses a bidirectional attention mechanism to calculate the context-aware feature representation representing the relationship between the second modal feature vectors.

2. The multimodal data processing method according to claim 1, characterized in that, The first association relationship includes semantically related information and edge update weights. The step of calling the modality recognition model to extract the first modality feature vector of the multimodal feature representation tensor and the first association relationship between all modal data, and updating the initial dynamic graph structure to the current dynamic graph structure based on the first modality feature vector and the first association relationship, includes: The modality recognition model is invoked to extract the first modality feature vector from the multimodal feature representation tensor, and the first modality feature vector is used as a new modality data node; The semantic related information of each modal data node pair is calculated using an association calculation function; each modal data node pair includes a first node pair consisting of any two new modal data nodes, a second node pair consisting of any two initial modal data nodes in the initial dynamic graph structure, and a third node pair consisting of any one new modal data node and any one initial modal data node; The edge update weights corresponding to each modal data node pair are calculated using a pre-set update balance coefficient and similarity metric function. A weighted adjacency matrix is ​​generated based on the semantic information and the edge update weights; The weighted adjacency matrix is ​​processed by a graph neural network model to generate the current dynamic graph structure.

3. The multimodal data processing method according to claim 1, characterized in that, The process of calling a pre-trained financial data processing model to process the fused dataset and obtain financial data processing results includes: The feature extraction sub-neural networks constructed for the different types of modal data in the fused dataset are invoked respectively to extract the modal features of the different types of modal data, and the modal features are uniformly embedded into the second modal feature vector; The second modality feature vector is processed into a context-aware feature representation using a bidirectional attention mechanism; The context-aware feature representations are input into their respective buffer queues, and the context-aware feature representations are retrieved from the buffer queues using a rhythm scheduling function; The intrinsic feature representation is extracted and fused from the context-aware feature representation to obtain the fused intrinsic feature representation. The fused intrinsic feature representation is then subjected to a nonlinear transformation to obtain the financial data processing result.

4. The multimodal data processing method according to claim 1, characterized in that, The different types of modal data include text data, numerical data, image data, and audio data. The process of processing the different types of modal data in the multimodal financial data to generate multimodal feature representation tensors corresponding to each type of modal data includes: When the modal data is text data, the text data is processed into a text feature matrix in a unified format; When the modal data is numerical data, the numerical data is processed into composite normalized data, or the numerical data is processed into regularized data. When the modal data is the image data, the image data is sequentially subjected to noise reduction, grayscale conversion, and key visual feature extraction to obtain the key visual features; When the modal data is audio data, the audio data is processed into structured text; The text feature matrix, the composite normalized or regularized data, the key visual features, and the structured text are uniformly converted into the corresponding multimodal feature representation tensors.

5. The multimodal data processing method according to claim 1, characterized in that, After processing the fused dataset using a pre-trained financial data processing model to obtain the financial data processing results, the process also includes: The financial data is standardized to obtain standardized financial processed data; A data dictionary and a data quality assessment system are established based on the standardized financial processing data. The data dictionary and data quality assessment system are stored in a financial data warehouse. The data quality assessment system is used to monitor the various financial data in the financial data warehouse.

6. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the multimodal data processing method as described in any one of claims 1 to 5.

7. A multimodal data processing device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the multimodal data processing device can implement the multimodal data processing method as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the multimodal data processing method as described in any one of claims 1 to 5.

9. A multimodal data processing system, characterized in that, It includes a data acquisition module, a data preprocessing module, a data fusion module, a multimodal AI modeling module, and a data standardization and governance module; The data acquisition module is used to collect multimodal financial data to be processed from different data sources and store the multimodal financial data in a temporary database; The data preprocessing module is used to acquire the multimodal financial data, perform customized processing on different types of modal data in the multimodal financial data, and after obtaining the customized processing results, convert the format of the customized processing results into a unified multimodal feature representation tensor. The data fusion module is used to fuse the multimodal feature representation tensor into the initial multimodal data using a multimodal fusion algorithm to obtain a fused dataset. The multimodal fusion algorithm integrates a graph neural network model and an online learning algorithm. The graph neural network model is used to capture the dynamic evolution characteristics of the modal relationships between the multimodal feature representation tensor and the initial multimodal data during the fusion process. The online learning algorithm is used to adjust the fusion weights of the modal data during the data fusion process based on the dynamic evolution characteristics of the modal relationships to obtain the fused dataset. The multimodal AI modeling module is used to process the fused dataset using a pre-trained financial data processing model to obtain financial data processing results. The financial data processing model is a neural network model that uses a deep learning framework to process the fused dataset layer by layer. The deep learning framework includes a cross-modal attention interaction layer, which constructs different feature extraction sub-neural networks for different types of modal data. The different feature extraction sub-neural networks are used to extract modal features of different types of modal data in the fused dataset. A bidirectional attention mechanism is used to calculate the context-aware feature representation of complementary information between the modal data in the fused dataset. The data standardization and governance module is used to standardize the results of financial data processing to obtain standardized financial processing data. Based on the standardized financial processing data, a data dictionary and a data quality assessment system are established, and the data dictionary and data quality assessment system are stored in the financial data warehouse. The data quality assessment system is used to evaluate and monitor the financial data in the financial data warehouse.

10. The multimodal data processing system according to claim 9, characterized in that, The deep learning framework includes cross-modal attention interaction, network layers, fusion layers, and hidden layers. The multimodal AI modeling module is specifically used for: The cross-modal attention interaction layer is used to construct different feature extraction sub-neural networks for different types of modal data in the fused dataset. The different feature extraction sub-neural networks are used to extract modal features of different types of modal data in the fused dataset. The extracted modal features are uniformly embedded into a modal feature vector. The bidirectional attention mechanism is used to process the modal feature vector into a context-aware feature representation. The network layer adopts an asynchronous input processing strategy, setting different buffer queues and input rhythm control modules for different types of modal data in the context-aware feature representation, and adjusting the order in which the context-aware feature representation is input into the fusion layer according to the data characteristics of the different types of modal data; The fusion layer is used to extract and fuse intrinsic feature representations from the context-aware feature representations using a self-supervised learning mechanism to obtain fused intrinsic feature representations. After receiving the fused intrinsic feature representation, the hidden layer performs a nonlinear transformation and outputs the financial data processing result.

Images