Multi-modal financial data fraud detection method, device, equipment, medium and program

Abstract

The invention discloses a multi-modal financial data fraud detection method, device and equipment and a medium, relates to the technical field of artificial intelligence, is suitable for the field of financial science and technology, and comprises the following steps: obtaining multi-modal financial data associated with a financial behavior of a target customer; according to the multi-modal financial data, mixed features corresponding to the target customer are generated, and the mixed features comprise customer inherent risk features and transaction behavior features of the target customer; inputting the mixed features of the target customer into a pre-trained anti-fraud model to obtain a customer risk portrait of the target customer; and when it is detected that the real-time transaction data corresponding to the target customer has a fraud risk according to the customer risk portrait, carrying out risk early warning on the real-time transaction data. According to the embodiment of the invention, through cross-dimensional fusion and processing of multi-modal financial data, systematic breakthrough of an anti-fraud technology in the field of financial science and technology is realized, and an intelligent solution with data compatibility and model generalization ability is provided for the field of financial science and technology risk control.

Description

Multimodal financial data fraud detection methods, devices, equipment, media and procedures Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and is applicable to the field of financial technology. In particular, it relates to a multimodal financial data fraud detection method, apparatus, equipment, medium, and program. Background Technology

[0002] Against the backdrop of the deep integration of fintech and artificial intelligence, banks and other financial institutions face the challenge of processing massive amounts of multimodal data in their operations. Currently, image modal data, text modal data, and time-series behavioral data generated in financial business scenarios are experiencing explosive growth. Efficiently processing this multi-source, heterogeneous data and extracting valuable information from it has become crucial for improving the intelligence level of financial services. Especially in core scenarios such as intelligent financial document recognition, real-time transaction anti-fraud, and customer identity verification, multimodal data fusion processing technology has become a key research focus in the industry.

[0003] However, traditional optical character recognition technology lacks cross-modal feature fusion capabilities, anti-fraud systems are limited to rule-based reasoning in a single modality, and biometric technology fails to combine multimodal behavioral features for dynamic verification. These shortcomings together make it difficult for banks to upgrade their technology from post-processing to real-time early warning and from single-modal analysis to multimodal collaborative reasoning in risk management, compliance review, and other businesses. There is an urgent need to break through existing technological bottlenecks through innovative applications of multimodal data fusion and deep learning technologies. Summary of the Invention

[0004] Based on this, the present invention provides a multimodal financial data fraud detection method, device, equipment, medium and program to solve the problems of insufficient multimodal data fusion capability, lagging anti-fraud identification and biometric security risks in traditional financial risk control.

[0005] In a first aspect, embodiments of the present invention provide a multimodal financial data fraud detection method, the method comprising:

[0006] Acquire targeted multimodal financial data that correlates with the financial behavior of target customers;

[0007] Based on the target multimodal financial data, target mixed features corresponding to the target customers are generated. The target mixed features include the target customers' inherent risk characteristics and transaction behavior characteristics.

[0008] The target customer's mixed features are input into a pre-trained anti-fraud model to obtain the target customer's risk profile, which includes the fraud probability of various transaction types.

[0009] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0010] When fraud risk is detected in real-time transaction data corresponding to the target customer based on the target customer's customer risk profile, a risk warning is issued for the real-time transaction data.

[0011] Secondly, embodiments of the present invention also provide a multimodal financial data fraud detection device, the device comprising:

[0012] The multimodal financial data acquisition module is used to acquire target multimodal financial data related to the financial behavior of target customers;

[0013] The hybrid feature generation module is used to generate target hybrid features corresponding to the target customer based on the target multimodal financial data. The target hybrid features include the target customer's inherent risk features and transaction behavior features.

[0014] The customer risk profile building module is used to input the target mixed features of the target customer into the pre-trained anti-fraud model to obtain the customer risk profile of the target customer, which includes the fraud probability of various transaction types.

[0015] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0016] The fraud risk warning module is used to issue a risk warning for the real-time transaction data when fraud risk is detected in the real-time transaction data corresponding to the target customer based on the target customer's customer risk profile.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform a multimodal financial data fraud detection method according to any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement a multimodal financial data fraud detection method according to any embodiment of the present invention.

[0022] Fifthly, a computer program product is also provided, the computer program product including a computer program, which, when executed by a processor, implements a multimodal financial data fraud detection method as described in any embodiment of the present invention.

[0023] This invention achieves a systemic breakthrough in anti-fraud technology within the fintech field through cross-dimensional fusion and intelligent processing of multimodal financial data. It integrates multi-source data such as images, text, and time series data, and solves the data fragmentation problem of traditional single-modal analysis through generative hybrid feature modeling. The anti-fraud model built based on a multimodal autoencoder forms a closed-loop technology of "pattern learning - anomaly detection - probability quantification." By transforming customer risk profiles into fraud probability matrices for various transaction types, and leveraging the unsupervised learning capability of the autoencoder and the synergistic effect of the supervised detection branch, it enables efficient risk warnings for real-time transaction data. Simultaneously, it provides interpretable support for risk decision-making through autoencoder reconstruction error analysis, offering an intelligent solution for fintech risk control that combines data compatibility and model generalization capabilities.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 is a flowchart of a multimodal financial data fraud detection method according to Embodiment 1 of the present invention;

[0027] Figure 2 is a flowchart of another multimodal financial data fraud detection method provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is a schematic diagram of the structure of a multimodal financial data fraud detection device according to Embodiment 3 of the present invention;

[0029] Figure 4 is a schematic diagram of the structure of an electronic device that implements a multimodal financial data fraud detection method according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 is a flowchart of a multimodal financial data fraud detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to multimodal data processing and risk control decisions, such as extraction of key information from financial documents, risk correlation analysis of transaction links, and local biometric verification. This method can be executed by a multimodal financial data fraud detection device, which can be implemented in hardware and / or software and can be configured in a bank's intelligent risk control system. As shown in Figure 1, the method includes:

[0034] S110. Obtain target multimodal financial data related to the financial behavior of target customers.

[0035] Target customer financial behavior correlation refers to limiting data to those directly related to a specific customer's transactions, operations, and other behaviors, excluding irrelevant public or non-transactional data. Essentially, it provides raw data for subsequent feature fusion and risk modeling. Its multimodal characteristics directly determine the information dimensions of the anti-fraud model. Financial risk control needs to achieve accurate positioning of fraud risks through multi-dimensional insights from multimodal data.

[0036] Optionally, the target multimodal financial data includes: image modal data, text modal data, time-series behavioral data, and externally related data;

[0037] Image modality refers to visual data such as check images, scanned copies of ID cards, bank card images, and handwritten signatures; text modality refers to textual information such as transaction notes, customer application forms, contract terms, and customer service communication records; time-series behavioral data refers to dynamic behavioral data such as transaction timestamp sequences, operation frequency (e.g., daily transfer count), and device login trajectory (e.g., IP address changes); external correlation data refers to cross-platform data such as credit reports, legal litigation records, correlation with black market lists, and third-party risk control scores. Image modality can verify the authenticity of transaction vouchers (e.g., consistency of signature handwriting); text modality can analyze the rationality of the transaction background (e.g., the matching degree between contract subject matter and transaction amount); time-series behavioral data can detect abnormal operation patterns (e.g., high-frequency cross-border transfers outside of working hours); and external correlation data can assess potential risk associations (e.g., the counterparty's level on black market lists).

[0038] S120. Based on the target multimodal financial data, generate target mixed features corresponding to the target customer, wherein the target mixed features include the target customer's inherent risk characteristics and transaction behavior characteristics.

[0039] Target hybrid features are fused features obtained by processing multimodal financial data of target customers. They include inherent customer risk characteristics (such as static attributes like identity and credit rating) and transaction behavior characteristics (such as dynamic transaction patterns like transaction time distribution and counterparty correlation). These features serve as input data for anti-fraud models, providing a basis for model analysis of customer risk. The essential attribute of target hybrid features is that they generate a unified-dimensional feature vector from multimodal financial data through cross-modal fusion technology, breaking down the modal barriers between image, text, and time-series data to form a three-dimensional feature representation that combines static risk and dynamic behavior. In this embodiment of the invention, tensor storage can be used to support batch computation of subsequent autoencoders.

[0040] When generating target hybrid features, feature-level desensitization is applied to customer inherent risk features and transaction behavior features. For example, after compression of customer inherent risk features through the bottleneck layer of the autoencoder, only contour features related to fraud detection are retained, filtering out identity recognition details (such as ID card addresses). Sensitive information in inherent risks (such as ID card numbers) is partially preserved, extracting only features related to fraud detection (such as edge features of ID card images, excluding specific text content). Text features automatically mask sensitive words such as names and ID card numbers, retaining only semantic keywords, such as transfer and cross-border. Device IPs in transaction behavior are converted to geographic region codes (such as "North China") to avoid locating specific devices.

[0041] S130. Input the target mixed features of the target customer into the pre-trained anti-fraud model to obtain the customer risk profile of the target customer, wherein the customer risk profile contains the fraud probability of various transaction types.

[0042] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0043] The pre-trained anti-fraud model is built on a multimodal autoencoder and consists of a feature extraction layer, a reconstruction layer, and a fraud detection branch. The feature extraction layer compresses and fuses multimodal features into a unified-dimensional feature vector; the reconstruction layer attempts to reconstruct the input features by learning normal transaction patterns; and the fraud detection branch classifies the features to determine if fraud risk exists. The customer risk profile is a structured representation that quantifies the probability of fraud in various transactions of a target customer. It covers the fraud probability of different transaction types (such as transfers, cash withdrawals, and cross-border transactions), and also includes information such as risk propagation paths (e.g., association weights with black market accounts) and dynamically updated risk levels (low / medium / high). It is a comprehensive presentation of the customer's risk status, providing core basis for banks' real-time transaction risk decisions and supporting banks in taking corresponding measures for transactions with different risk levels.

[0044] Furthermore, before inputting the target customer's mixed features into the pre-trained anti-fraud model, it may also include:

[0045] Acquire historical multimodal financial data correlated with the financial behavior of multiple reference clients;

[0046] Based on historical multimodal financial data, historical mixed features corresponding to each reference customer are generated, and labeled customer risk profiles are added to each historical mixed feature to obtain multiple training samples.

[0047] The multimodal autoencoder is trained using the training samples to obtain the anti-fraud model.

[0048] Historical multimodal financial data refers to images, text, time-series behavior, and externally related data associated with the historical financial behavior of multiple reference clients. Historical mixed features are fused features obtained by processing historical multimodal data according to the generation logic of the target mixed features. They include: the inherent risk characteristics of the reference clients and historical transaction behavior characteristics. The generation logic is consistent with that of the target mixed features, but historical mixed features are based on past data and used for model training, while the target mixed features are based on current data and used for real-time inference. Labeled client risk profiles refer to risk tags manually or through expert systems labeled with transaction scenarios corresponding to historical mixed features, bound to the historical mixed features in the form of structured tags.

[0049] The collected historical multimodal data is processed in the same way as the target mixed features, extracted and fused into historical mixed features, which include the customer's inherent risk characteristics and transaction behavior characteristics. Then, risk control experts or existing rules are used to label these historical mixed features to determine the risk profile corresponding to each transaction. For example, the fraud probability of a transfer transaction is labeled as 0.8, and the risk level is high. Finally, a training sample is formed by historical mixed features + labeled risk profile, such as [feature vector, {transfer fraud probability: 0.8, risk level: high}].

[0050] The generated training samples are input into a multimodal autoencoder, which includes a feature extraction layer, a reconstruction layer, and a fraud detection branch. During training, the feature extraction layer learns how to extract key features from the training samples; the reconstruction layer attempts to reconstruct the input features, minimizing the reconstruction error by continuously adjusting parameters, thereby learning patterns of normal transactions; the fraud detection branch, based on labeled risk profiles, learns how to classify input features and determine whether fraud exists using loss functions such as binary cross-entropy. After multiple rounds of training and parameter tuning, a practically applicable anti-fraud model is finally obtained.

[0051] The aforementioned training process essentially encodes the historical experience of bank risk control experts into the model. The historically labeled data is equivalent to the expert's risk assessment records of past transactions. By learning from these records, the model acquires risk identification logic similar to that of the expert. The dual architecture (reconstruction + detection) of the multimodal autoencoder retains the advantages of traditional risk control based on experience-based pattern recognition while also possessing data-driven pattern discovery capabilities. This training mechanism enables the anti-fraud model to quickly identify known fraud patterns and detect anomalies exceeding historical experience through reconstruction errors. Ultimately, this results in more accurate real-time risk profiles for target customers and more timely risk warnings.

[0052] Optionally, training the multimodal autoencoder using the training samples to obtain the anti-fraud model may include:

[0053] Each training sample is modally decomposed into text, image, temporal, and external correlation features, and input into the feature extraction layer of a multimodal autoencoder. A fusion feature vector with a unified dimension is generated by weighted summation.

[0054] Image modal data and abnormal temporal behavior data of typical fraud cases are extracted from the historical multimodal financial data and input into the generative adversarial network. Forged fraud samples are obtained by adjusting the adversarial loss of the generator and the discriminator.

[0055] The forged fraud samples are concatenated with the fused feature vector to form an adversarial training set containing real and synthetic samples;

[0056] The adversarial training set is input into the multimodal autoencoder. By minimizing the reconstruction loss and the fraud detection branch loss, the model parameters of the feature extraction layer, reconstruction layer, and fraud detection branch are updated synchronously to obtain the trained anti-fraud model.

[0057] The labeled training samples are split into four modalities: image, text, time series, and external correlation. These are then input into the feature extraction layer of a multimodal autoencoder. An attention mechanism assigns weights to features of different modalities, and the weighted sum generates a unified-dimensional fusion feature vector. This allows the model to simultaneously handle cross-modal risk signals, such as the combined risk of semantic anomalies in text notes and blurred image stamps. Images of typical fraud cases (e.g., counterfeit checks) and abnormal time series data (e.g., high-frequency late-night transfers) are selected from historical data and input into a generative adversarial network (GAN). A generator produces realistic counterfeit fraud samples, while a discriminator distinguishes between real and counterfeit samples. The two sides iteratively optimize until the generator outputs samples that are difficult to distinguish, enabling the model to learn potential fraud patterns in advance and improving its ability to identify zero-sample fraud. The features of fake fraud samples generated by a generative adversarial network are concatenated with the original fused feature vector (e.g., adding a 128-dimensional fake feature after the 512-dimensional vector) to form an adversarial training set containing real samples (70%) and synthetic samples (30%). This simulates the dynamic evolution of fraud methods in real-world scenarios; for example, when the model encounters fake samples during training, it can respond more quickly to new types of fraud. The adversarial training set is input into a multimodal autoencoder, simultaneously optimizing the reconstruction loss and the fraud detection branch loss. The parameters of the feature extraction layer, reconstruction layer, and detection branch are updated synchronously through backpropagation until the loss function converges. Through dual loss constraints, the model possesses both "normal pattern memory" (reconstruction layer) and "abnormal pattern recognition" (detection branch) capabilities. For example, samples with large reconstruction errors are preferentially marked as high-risk by the detection branch.

[0058] Generative adversarial networks are introduced to address the scarcity of fraud samples. Adversarial training sets are used to simulate the evolution of real risks. Dual loss functions are used to construct multidimensional risk judgment criteria. The training process of this invention uses a technical chain of "modal decomposition - adversarial enhancement - dual optimization" to upgrade the anti-fraud model from passively learning history to actively predicting the future. Ultimately, the model can identify "known fraud variants" and "unknown risk patterns" in real time in banking scenarios, forming a dynamic and adaptive risk control capability.

[0059] S140. When fraud risk is detected in real-time transaction data corresponding to the target customer based on the target customer's customer risk profile, a risk warning is issued for the real-time transaction data.

[0060] When the system determines that a target customer's real-time transaction data poses a fraud risk based on the customer's risk profile, it immediately triggers a risk warning. In this process, the customer risk profile is formed based on past data, encompassing key information such as the probability of fraud for various transaction types, serving as a benchmark for judgment. The system compares real-time transaction data with the profile and assesses risk through set rules or algorithms. Once the risk is determined to exceed the threshold, a warning is issued to bank risk control personnel or the customer via SMS, system notifications, etc., so that timely measures such as transaction interception and manual review can be taken to prevent financial losses and ensure transaction security.

[0061] Optionally, when fraud risk is detected in real-time transaction data corresponding to the target customer based on the target customer's customer risk profile, a risk warning may be issued for the real-time transaction data, which may include:

[0062] The target transaction type of the real-time transaction data is obtained, and the target fraud probability matching the target transaction type is extracted from the customer risk profile of the target customer.

[0063] If the probability of fraud is greater than or equal to a preset probability threshold, the real-time transaction data is determined to have a fraud risk, and a risk warning is issued for the real-time transaction data.

[0064] The system first identifies the type of real-time transaction (such as transfers and cross-border payments), then retrieves the fraud probability value for that type of transaction from the customer's risk profile (e.g., a transfer fraud probability of 0.85). This step achieves scenario-based risk assessment by accurately matching the transaction scenario. The extracted fraud probability is compared with a preset standard; if it exceeds the threshold, it is determined to be a high-risk transaction, and an early warning mechanism is immediately activated (e.g., transaction blocking, manual review). This step transforms risk judgment into executable operational instructions through quantitative standards, ensuring the efficiency and consistency of risk control decisions. This embodiment of the invention, through a standardized process of "transaction type - probability matching - threshold determination," transforms abstract risk profiles into specific transaction handling actions, achieving an automated closed loop from risk identification to response, ensuring both risk control accuracy and improving real-time handling efficiency.

[0065] This invention achieves a systemic breakthrough in anti-fraud technology within the fintech field through cross-dimensional fusion and intelligent processing of multimodal financial data. It integrates multi-source data such as images, text, and time series data, and solves the data fragmentation problem of traditional single-modal analysis through generative hybrid feature modeling. The anti-fraud model built based on a multimodal autoencoder forms a closed-loop technology of "pattern learning - anomaly detection - probability quantification." By transforming customer risk profiles into fraud probability matrices for various transaction types, and leveraging the unsupervised learning capability of the autoencoder and the synergistic effect of the supervised detection branch, it enables efficient risk warnings for real-time transaction data. Simultaneously, it provides interpretable support for risk decision-making through autoencoder reconstruction error analysis, offering an intelligent solution for fintech risk control that combines data compatibility and model generalization capabilities.

[0066] Example 2

[0067] Figure 2 is a flowchart of another multimodal financial data fraud detection method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on Embodiment 1. Specifically, as shown in Figure 2, the method includes:

[0068] S210. Obtain target multimodal financial data related to the financial behavior of target customers.

[0069] S220. The target multimodal financial data is modally aligned using a cross-modal alignment model to obtain aligned multimodal data.

[0070] Cross-modal alignment model processing refers to mapping heterogeneous data such as images, text, and time series to a unified semantic space through models such as deep learning, eliminating modal differences, bridging the semantic gap of multimodal data, and providing a foundation for subsequent fusion, such as aligning the amount in a check image with the amount in the text of a transaction system.

[0071] S230. The quantum annealing algorithm is used to perform semantic disambiguation and data augmentation on the aligned multimodal data to obtain standardized multimodal data.

[0072] Semantic disambiguation in quantum annealing algorithm optimization refers to identifying the accurate meaning of polysemous words in financial scenarios, while data augmentation refers to simulating different risk scenarios through quantum computing to generate more diverse transaction features.

[0073] Optionally, the aligned multimodal data is semantically disambiguated and data augmented using a quantum annealing algorithm to obtain standardized multimodal data, including:

[0074] The aligned multimodal data is processed using the quantum annealing algorithm to obtain disambiguation-optimized aligned multimodal data;

[0075] Generative adversarial network algorithm is used to process the disambiguation-optimized aligned multimodal data to obtain supplementary multimodal data samples;

[0076] The supplementary multimodal data samples and the disambiguated and optimized aligned multimodal data are concatenated for features. Feature enhancement vectors are generated through a domain-adaptive language model. After dimensional alignment processing, the enhanced multimodal data is output.

[0077] The enhanced multimodal data is processed by a fusion model of recurrent neural networks and graph neural networks to output a data association map. Standardized multimodal data containing abnormal pattern features is output through a cross-domain knowledge transfer framework.

[0078] Leveraging the probabilistic optimization properties of quantum annealing, semantic disambiguation is performed on cross-modal aligned data. For example, it resolves whether the word "frozen" in a text refers to "judicial freeze" or "system lock" in a financial context, eliminating semantic ambiguity in blurred seals in images. Based on the disambiguated aligned data, generative adversarial networks (GANs) are used to generate missing multimodal samples, such as synthesizing anomalous samples of "high-frequency nighttime transfers" based on the temporal characteristics of historical normal transactions, or simulating image features of counterfeit checks. These supplementary samples are then concatenated with the original disambiguated data, and feature enhancement vectors are generated using a domain-adaptive language model, focusing on professional semantics such as money laundering and identity theft. After dimensional alignment, enhanced data in a unified format is output. Recurrent neural networks are used to capture the temporal dependencies of multimodal data; graph neural networks are used to construct a customer-transaction-counterparty relationship graph to identify network topology features in group fraud; and a cross-domain knowledge transfer framework is combined to embed external risk control knowledge into data features, outputting standardized data containing anomalous patterns.

[0079] This invention constructs an "intelligent purification-enhancement-modeling" system for financial multimodal data through a technical chain of "quantum computing disambiguation + data augmentation + graph network correlation analysis". Its core value lies in transforming unstructured multimodal data into standardized features containing deep risk semantics, providing high-quality input for anti-fraud models, and ultimately achieving accurate identification and early warning of complex fraud patterns.

[0080] S240. Based on standardized multimodal data, generate target mixed features corresponding to the target customer. These target mixed features include the target customer's inherent risk characteristics and transaction behavior characteristics.

[0081] The processed, standardized data is then fused according to the framework of "inherent risk characteristics + transaction behavior characteristics" to generate a fixed-dimensional feature vector. This forms a unified risk feature representation, which is convenient for input into anti-fraud models for calculation. For example, a customer's historical credit records and current transaction frequency can be integrated into a single vector.

[0082] Furthermore, based on standardized multimodal data, target hybrid features corresponding to the target customer can be generated, which may include:

[0083] Standardized multimodal data is input into the generative fusion model to extract customer identity image features and signature image features from image modal data, customer basic information and account information from text modal data, transaction time features and transaction frequency features from time-series behavioral data, and customer credit rating features and litigation information features from external related data.

[0084] The extracted multimodal data is subjected to cross-modal generative alignment to output a feature tensor with uniform dimension, thereby constructing a unified feature space.

[0085] A causal reasoning graph is generated based on the unified feature space, and a local risk feature vector is obtained by analyzing the node relationships and edge weights in the causal reasoning graph.

[0086] The local risk feature vectors of each node in the causal reasoning graph are collected and aggregated using the federated knowledge distillation framework to obtain aggregated features.

[0087] The inherent risk features and transaction behavior features of customers are extracted from the aggregated features, and the extracted inherent risk features and transaction behavior features are combined to obtain the target mixed features.

[0088] Multimodal feature extraction mainly refers to extracting key features from four types of modal data through generative fusion models. Image modalities can include ID card texture features and signature handwriting topology; text modalities can include customer names, ID card numbers (after anonymization), and account opening information; temporal behavior can include transaction time distribution and operation interval patterns; external correlations can include credit ratings (such as AAA level) and litigation case numbers (anonymized), thereby achieving a preliminary abstraction from raw data to risk signals. For example, signature image features can be used for identity theft detection, and transaction time features can identify abnormal transactions at night.

[0089] Generative adversarial networks (GANs) are used to map features from different modalities to a unified dimension, addressing the inconsistency in dimensionality of heterogeneous data such as image pixels, text words, and time series, enabling direct calculation of correlations across modal features. The causal inference graph is a model representing the causal relationships of risk factors using a graph structure, where nodes are features (e.g., "low credit rating") and edges are causal weights (e.g., "low credit → fraud probability +20%)". Each institution locally generates local risk features for the causal graph. Based on federated learning feature aggregation technology, institutions share risk feature weights through distillation (knowledge transfer) without sharing the original data (e.g., Bank A contributes a cross-border transaction weight of 0.3, Bank B contributes an abnormal device weight of 0.2). Two types of features are separated from the aggregated features: inherent risk: static attributes that do not change with transactions (e.g., credit rating, account opening duration) and behavioral features: dynamic transaction patterns (e.g., current transfer amount deviates from the historical average). For example, when inherent risk is low but behavioral features are abnormal, a medium-risk warning is triggered to avoid indiscriminate misjudgment.

[0090] This invention utilizes a generative fusion model to deeply extract features from multi-source data, constructs a causal reasoning graph to analyze the logical relationships between risk elements, and employs a federated knowledge distillation framework to achieve secure aggregation of cross-institutional risk features. This accurately separates inherent customer risks from transaction behavior features, significantly improving the accuracy and comprehensiveness of financial data processing, providing refined feature inputs for anti-fraud models, and effectively enhancing the ability to identify and warn of complex financial risks.

[0091] S250. Input the target mixed features of the target customer into the pre-trained anti-fraud model to obtain the customer risk profile of the target customer, wherein the customer risk profile contains the fraud probability of various transaction types.

[0092] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0093] S260. When fraud risk is detected in real-time transaction data corresponding to the target customer based on the target customer's customer risk profile, a risk warning is issued for the real-time transaction data.

[0094] This invention focuses on the core needs of the financial system. By integrating multi-source data such as images and text, and employing a cross-modal alignment model and quantum annealing algorithm, it eliminates data modality differences and semantic ambiguity, enhances data sample diversity, and achieves standardized processing of multimodal data, laying a solid data foundation for subsequent risk analysis. The quantum annealing algorithm eliminates semantic ambiguity in financial data, and the combination of generative adversarial networks and fusion models enhances the data's ability to represent risk patterns, achieving standardization of multimodal data and extraction of anomalous features. Using generative fusion models and causal reasoning graphs, it separates inherent customer risk from transaction behavior characteristics, and achieves cross-institutional risk aggregation through federated knowledge distillation, providing accurate and interpretable feature support for financial anti-fraud.

[0095] Example 3

[0096] Figure 3 is a schematic diagram of a multimodal financial data fraud detection device provided in Embodiment 3 of the present invention. As shown in Figure 3, the device includes:

[0097] The multimodal financial data acquisition module 310 is used to acquire target multimodal financial data related to the financial behavior of target customers;

[0098] The hybrid feature generation module 320 is used to generate target hybrid features corresponding to the target customer based on the target multimodal financial data. The target hybrid features include the target customer's inherent risk features and transaction behavior features.

[0099] The customer risk profile construction module 330 is used to input the target mixed features of the target customer into the pre-trained anti-fraud model to obtain the customer risk profile of the target customer, wherein the customer risk profile contains the fraud probability of various transaction types.

[0100] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0101] The fraud risk warning module 340 is used to issue a risk warning to the real-time transaction data when a fraud risk is detected in the real-time transaction data corresponding to the target customer based on the target customer's customer risk profile.

[0102] This invention achieves a systemic breakthrough in anti-fraud technology within the fintech field through cross-dimensional fusion and intelligent processing of multimodal financial data. It integrates multi-source data such as images, text, and time series data, and solves the data fragmentation problem of traditional single-modal analysis through generative hybrid feature modeling. The anti-fraud model built based on a multimodal autoencoder forms a closed-loop technology of "pattern learning - anomaly detection - probability quantification." By transforming customer risk profiles into fraud probability matrices for various transaction types, and leveraging the unsupervised learning capability of the autoencoder and the synergistic effect of the supervised detection branch, it enables efficient risk warnings for real-time transaction data. Simultaneously, it provides interpretable support for risk decision-making through autoencoder reconstruction error analysis, offering an intelligent solution for fintech risk control that combines data compatibility and model generalization capabilities.

[0103] Optionally, based on the above embodiments, the target multimodal financial data in the multimodal financial data acquisition module 310 includes: image modal data, text modal data, time-series behavioral data, and external correlation data.

[0104] Optionally, based on the above embodiments, the hybrid feature generation module 320 may include:

[0105] A modal alignment unit is used to perform modal alignment on the target multimodal financial data using a cross-modal alignment model to obtain aligned multimodal data.

[0106] The quantum annealing processing unit is used to perform semantic disambiguation and data augmentation processing on the aligned multimodal data using the quantum annealing algorithm to obtain standardized multimodal data;

[0107] The hybrid feature generation unit is used to generate target hybrid features corresponding to the target customer based on standardized multimodal data.

[0108] Optionally, based on the above embodiments, the quantum annealing processing unit can be used to process the aligned multimodal data using a quantum annealing algorithm to obtain disambiguation-optimized aligned multimodal data;

[0109] Generative adversarial network algorithm is used to process the disambiguation-optimized aligned multimodal data to obtain supplementary multimodal data samples;

[0110] The supplementary multimodal data samples and the disambiguated and optimized aligned multimodal data are concatenated for features. Feature enhancement vectors are generated through a domain-adaptive language model. After dimensional alignment processing, the enhanced multimodal data is output.

[0111] The enhanced multimodal data is processed by a fusion model of recurrent neural networks and graph neural networks to output a data association map. Standardized multimodal data containing abnormal pattern features is output through a cross-domain knowledge transfer framework.

[0112] Optionally, based on the above embodiments, the hybrid feature generation unit can be used to input standardized multimodal data into a generative fusion model, extract customer identity image features and signature image features from image modal data, extract customer basic information and account information from text modal data, extract transaction time features and transaction frequency features from time-series behavioral data, and extract customer credit rating features and litigation information features from external related data.

[0113] The extracted multimodal data is subjected to cross-modal generative alignment to output a feature tensor with uniform dimension, thereby constructing a unified feature space.

[0114] A causal reasoning graph is generated based on the unified feature space, and a local risk feature vector is obtained by analyzing the node relationships and edge weights in the causal reasoning graph.

[0115] The local risk feature vectors of each node in the causal reasoning graph are collected and aggregated using the federated knowledge distillation framework to obtain aggregated features.

[0116] The inherent risk features and transaction behavior features of customers are extracted from the aggregated features, and the extracted inherent risk features and transaction behavior features are combined to obtain the target mixed features.

[0117] Optionally, based on the above embodiments, it may also include: an anti-fraud model training unit, used to acquire historical multimodal financial data associated with the financial behavior of multiple reference customers before inputting the target mixed features of the target customer into the pre-trained anti-fraud model;

[0118] Based on historical multimodal financial data, historical mixed features corresponding to each reference customer are generated, and labeled customer risk profiles are added to each historical mixed feature to obtain multiple training samples.

[0119] The multimodal autoencoder is trained using the training samples to obtain the anti-fraud model.

[0120] Optionally, based on the above embodiments, the anti-fraud model training unit can also be used to decompose each training sample into text, image, time series and external correlation features according to modality, input them into the feature extraction layer of the multimodal autoencoder, and generate a fusion feature vector of uniform dimension by weighted summation.

[0121] Image modal data and abnormal temporal behavior data of typical fraud cases are extracted from the historical multimodal financial data and input into the generative adversarial network. Forged fraud samples are obtained by adjusting the adversarial loss of the generator and the discriminator.

[0122] The forged fraud samples are concatenated with the fused feature vector to form an adversarial training set containing real and synthetic samples;

[0123] The adversarial training set is input into the multimodal autoencoder. By minimizing the reconstruction loss and the fraud detection branch loss, the model parameters of the feature extraction layer, reconstruction layer, and fraud detection branch are updated synchronously to obtain the trained anti-fraud model.

[0124] Optionally, based on the above embodiments, the fraud risk warning module 340 may include:

[0125] The fraud probability matching unit is used to obtain the target transaction type of the real-time transaction data and extract the target fraud probability that matches the target transaction type from the customer risk profile of the target customer.

[0126] The risk warning unit is used to determine that the real-time transaction data has a fraud risk if the target fraud probability is greater than or equal to a preset probability threshold, and to issue a risk warning for the real-time transaction data.

[0127] The multimodal financial data fraud detection device provided in this embodiment of the invention can execute the multimodal financial data fraud detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0128] It should be noted that the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0129] Example 4

[0130] Figure 4 illustrates a schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0131] As shown in Figure 4, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a multimodal financial data fraud detection method.

[0134] In other words: acquiring target multimodal financial data related to the financial behavior of target customers;

[0135] Based on the target multimodal financial data, target mixed features corresponding to the target customers are generated. The target mixed features include the target customers' inherent risk characteristics and transaction behavior characteristics.

[0136] The target customer's mixed features are input into a pre-trained anti-fraud model to obtain the target customer's risk profile, which includes the fraud probability of various transaction types.

[0137] The anti-fraud model is obtained by training a multimodal autoencoder that includes a feature extraction layer, a reconstruction layer, and a fraud detection branch.

[0138] When fraud risk is detected in real-time transaction data corresponding to the target customer based on the target customer's customer risk profile, a risk warning is issued for the real-time transaction data.

[0139] In some embodiments, a multimodal financial data fraud detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal financial data fraud detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a multimodal financial data fraud detection method by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0145] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0146] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal financial data fraud detection method, characterized in that, The method includes: acquiring target multimodal financial data associated with the financial behavior of target customers; generating target mixed features corresponding to target customers based on the target multimodal financial data, wherein the target mixed features include the target customer's inherent risk features and transaction behavior features; inputting the target mixed features of target customers into a pre-trained anti-fraud model to obtain a customer risk profile of target customers, wherein the customer risk profile includes the fraud probability of various transaction types; wherein the anti-fraud model is obtained by training a multimodal autoencoder containing a feature extraction layer, a reconstruction layer, and a fraud detection branch; and issuing a risk warning for the real-time transaction data when fraud risk is detected in the real-time transaction data corresponding to the target customer based on the customer risk profile of the target customer.

2. The method according to claim 1, characterized in that, The target multimodal financial data includes: image modal data, text modal data, time-series behavioral data, and external correlation data. Based on the target multimodal financial data, target hybrid features corresponding to the target customer are generated, including: using a cross-modal alignment model to perform modal alignment on the target multimodal financial data to obtain aligned multimodal data; using a quantum annealing algorithm to perform semantic disambiguation and data augmentation processing on the aligned multimodal data to obtain standardized multimodal data; and generating target hybrid features corresponding to the target customer based on the standardized multimodal data.

3. The method according to claim 2, characterized in that, The aligned multimodal data is semantically disambiguated and data augmented using a quantum annealing algorithm to obtain standardized multimodal data. This process includes: processing the aligned multimodal data using quantum annealing to obtain disambiguated and optimized aligned multimodal data; processing the disambiguated and optimized aligned multimodal data using a generative adversarial network (GAN) algorithm to obtain supplementary multimodal data samples; concatenating the supplementary multimodal data samples with the disambiguated and optimized aligned multimodal data using feature concatenation, generating feature augmentation vectors through a domain-adaptive language model, and outputting augmented multimodal data after dimensional alignment processing; and outputting a data association graph from the augmented multimodal data using a recurrent neural network (RNN) and graph neural network (GNN) fusion model, and outputting standardized multimodal data containing anomalous pattern features through a cross-domain knowledge transfer framework.

4. The method according to claim 2, characterized in that, Based on standardized multimodal data, target hybrid features corresponding to the target customer are generated, including: inputting standardized multimodal data into a generative fusion model to extract customer identity image features and signature image features from image modality data, extract customer basic information and account information from text modality data, extract transaction time features and transaction frequency features from time-series behavior data, and extract customer credit rating features and litigation information features from external correlation data; performing cross-modal generative alignment on the extracted multimodal data to output a feature tensor with unified dimensions, and constructing a unified feature space; generating a causal reasoning graph based on the unified feature space, and obtaining local risk feature vectors by analyzing the node relationships and edge weights in the causal reasoning graph; collecting and aggregating the local risk feature vectors of each node in the causal reasoning graph through a federated knowledge distillation framework to obtain aggregated features; extracting customer inherent risk features and transaction behavior features from the aggregated features, and combining the extracted customer inherent risk features and transaction behavior features to obtain target hybrid features.

5. The method according to any one of claims 1-4, characterized in that, Before inputting the target mixed features of the target customer into the pre-trained anti-fraud model, the method further includes: acquiring historical multimodal financial data associated with the financial behavior of multiple reference customers; generating historical mixed features corresponding to each reference customer based on each historical multimodal financial data, and adding labeled customer risk profiles to each historical mixed feature to obtain multiple training samples; and using each of the training samples to train the multimodal autoencoder to obtain the anti-fraud model.

6. The method according to claim 5, characterized in that, The anti-fraud model is obtained by training the multimodal autoencoder using the training samples described above, including: decomposing each training sample modally into text, image, temporal, and external correlation features, inputting them into the feature extraction layer of the multimodal autoencoder, and generating a fusion feature vector of uniform dimension using weighted summation; extracting image modality data and abnormal temporal behavior data of typical fraud cases from the historical multimodal financial data and inputting them into a generative adversarial network, and obtaining forged fraud samples by adjusting the adversarial losses of the generator and discriminator; concatenating the forged fraud samples with the fusion feature vector to form an adversarial training set containing real samples and synthetic samples; inputting the adversarial training set into the multimodal autoencoder, and simultaneously updating the model parameters of the feature extraction layer, reconstruction layer, and fraud detection branch by minimizing the reconstruction loss and fraud detection branch loss, to obtain the trained anti-fraud model.

7. The method according to claim 1, characterized in that, When a fraud risk is detected in real-time transaction data corresponding to a target customer based on the target customer's customer risk profile, a risk warning is issued for the real-time transaction data, including: obtaining the target transaction type of the real-time transaction data, and extracting the target fraud probability matching the target transaction type from the target customer's customer risk profile; if the target fraud probability is greater than or equal to a preset probability threshold, it is determined that the real-time transaction data has a fraud risk, and a risk warning is issued for the real-time transaction data.

8. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a multimodal financial data fraud detection method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the multimodal financial data fraud detection method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a multimodal financial data fraud detection method according to any one of claims 1-7.

Images