Financial fraud detection system and method based on multi-source heterogeneous data fusion

Abstract

The invention discloses a financial fraud detection system and method based on multi-source heterogeneous data fusion, and the system comprises a data collection module which collects data; the data preprocessing module is used for processing data; the data fusion module is used for carrying out feature selection and carrying out data fusion based on a data source weight vector; the fraud detection and early warning module is used for training by adopting a machine learning algorithm to obtain a fraud detection model; the method comprises the steps of collecting data, preprocessing the collected data, carrying out feature selection on the preprocessed data, carrying out data fusion based on a data source weight vector, obtaining a fraud detection model, and carrying out fraud detection. Therefore, through different types of data sources, the support vector machine algorithm is combined with the feature selection technology and the weight-based data fusion method, higher detection precision and reliability are achieved, acquisition functions are customized for different data sources, the high performance of the system is ensured, the application range is widened, and the practicability is improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a financial fraud detection system and method for multi-source heterogeneous data fusion. Background Art

[0002] With the rapid development of information technology, the amount of data is growing exponentially. The amount of data generated and collected by various enterprises and organizations every day is huge and diverse. In the financial field, enterprises need to process data from multiple different channels, including but not limited to internal structured financial statements, semi-structured business documents, and external unstructured market information and public opinion data. These multi-source heterogeneous data contain rich information on enterprise operations and risks, but also bring challenges to traditional data analysis methods.

[0003] Traditional financial fraud detection mainly relies on manual review of structured financial data or rule-based automated systems. This method is not only inefficient, but also difficult to adapt to complex and changeable fraud behavior patterns. In addition, traditional methods often ignore the important clues contained in semi-structured and unstructured data, which may lead to incomplete and inaccurate identification of potential fraudulent behaviors. With the advancement of machine learning and artificial intelligence technology, it has become possible to use advanced algorithms to analyze massive multi-source heterogeneous data, thereby providing new ideas and technical means for more efficient and accurate detection of financial fraud. Therefore, it is necessary to develop a financial fraud detection system that can effectively integrate and analyze multi-source heterogeneous data. Summary of the invention

[0004] The present invention aims to solve the technical problems in the above-mentioned technology at least to some extent.

[0005] To this end, the first aspect of the present invention discloses a financial fraud detection system for multi-source heterogeneous data fusion, comprising:

[0006] A data collection module, used for collecting data from a plurality of different types of data source sets, wherein the data collection module is provided with data quality constraints;

[0007] Data preprocessing module, which is used for cleaning and standardizing structured data, extracting key information and structuring semi-structured data, and processing text and extracting feature vectors for unstructured data;

[0008] The data fusion module is used to perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector;

[0009] The fraud detection and early warning module is used to train a fraud detection model using a machine learning algorithm, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds a preset threshold.

[0010] In addition, the financial fraud detection system for multi-source heterogeneous data fusion disclosed in the present invention may also have the following additional technical features:

[0011] In one embodiment of the present invention, the data collection module is provided with a collection function defined for each data source, and the collection function depends on the time, indicating an operation of collecting data from the corresponding data source at a specific time.

[0012] In one embodiment of the present invention, the data preprocessing module is provided with:

[0013] Cleaning functions are used to remove redundancy and errors in structured data;

[0014] Normalization functions, used to convert structured data into a uniform format;

[0015] Extract key information function, used to extract key information from semi-structured data;

[0016] Structural transformation function, used to transform key information extracted from semi-structured data into structured data;

[0017] Text processing functions, used to convert unstructured data into text data;

[0018] The feature vector extraction function is used to extract feature vectors from text data.

[0019] In one embodiment of the present invention, the data fusion module is provided with a structured data feature selection function, a semi-structured data feature selection function and an unstructured data feature selection function, which are used to extract feature vectors from the preprocessed data, and perform data fusion according to the data source weight vector to obtain a fused feature vector.

[0020] In one embodiment of the present invention, in the fraud detection and early warning module, the fraud detection model is trained using a support vector machine algorithm, the training function input is the fused data features, and the output is the trained fraud detection model.

[0021] The second aspect of the present invention discloses a financial fraud detection method based on multi-source heterogeneous data fusion, comprising the following steps:

[0022] S1. Collect data from multiple different types of data sources and monitor data quality;

[0023] S2. Preprocess the collected data, including cleaning and standardization of structured data, key information extraction and structured conversion of semi-structured data, and text processing and feature vector extraction of unstructured data;

[0024] S3. Perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector;

[0025] S4. Use machine learning algorithms to train a fraud detection model, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds the preset threshold.

[0026] In addition, the financial fraud detection method based on multi-source heterogeneous data fusion disclosed in the present invention may also have the following additional technical features:

[0027] In one embodiment of the present invention, in step S1 , a collection function is defined for each data source, and data is collected from the corresponding data source at a specific time.

[0028] In one embodiment of the present invention, in step S2, preprocessing operations are performed on structured data, semi-structured data and unstructured data respectively.

[0029] In one embodiment of the present invention, in step S3, data fusion is performed according to the data source weight vector to obtain a fused feature vector.

[0030] In one embodiment of the present invention, in step S4, a support vector machine algorithm is used to train a fraud detection model, and a determination is made as to whether to trigger an early warning based on a fraud possibility threshold.

[0031] In summary, the financial fraud detection system and method for multi-source heterogeneous data fusion disclosed in the present invention can use different types of data sources, utilize support vector machine algorithms combined with feature selection technology and weight-based data fusion methods, have higher detection accuracy and reliability, and customized acquisition functions for different data sources, thereby ensuring the high performance of the system and enhancing the scope of application and practicality.

[0032] Additional contents and advantages of the present invention will be given in the following description or may be understood through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The technical solutions and beneficial effects of the present invention will become apparent and easily understood from the following contents in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 It is a system block diagram of the financial fraud detection system of multi-source heterogeneous data fusion of the present invention;

[0035] Figure 2 The flowchart of the financial fraud detection method of multi-source heterogeneous data fusion of the present invention. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0037] The financial fraud detection system and method for multi-source heterogeneous data fusion disclosed by the present invention will be described below with reference to the accompanying drawings.

[0038] like Figure 1 and Figure 2 As shown, a financial fraud detection system based on multi-source heterogeneous data fusion includes:

[0039] A data collection module is used to collect data from a plurality of different types of data source sets. The data collection module is provided with data quality constraints;

[0040] Data preprocessing module, which is used for cleaning and standardizing structured data, extracting key information and structuring semi-structured data, and processing text and extracting feature vectors for unstructured data;

[0041] The data fusion module is used to perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector;

[0042] The fraud detection and early warning module is used to train a fraud detection model using a machine learning algorithm, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds a preset threshold.

[0043] In the data collection module, a collection function defined for each data source is provided. The collection function depends on the time, and represents the operation of collecting data from the corresponding data source at a specific time.

[0044] In the data preprocessing module, the following settings are set:

[0045] Cleaning functions are used to remove redundancy and errors in structured data;

[0046] Normalization functions, used to convert structured data into a uniform format;

[0047] Extract key information function, used to extract key information from semi-structured data;

[0048] Structural transformation function, used to transform key information extracted from semi-structured data into structured data;

[0049] Text processing functions, used to convert unstructured data into text data;

[0050] The feature vector extraction function is used to extract feature vectors from text data.

[0051] In the data fusion module, there are structured data feature selection functions, semi-structured data feature selection functions and unstructured data feature selection functions, which are used to extract feature vectors from the preprocessed data and perform data fusion according to the data source weight vector to obtain the fused feature vector.

[0052] In the fraud detection and early warning module, the fraud detection model is trained using the support vector machine algorithm. The training function input is the fused data features, and the output is the trained fraud detection model.

[0053] A financial fraud detection method based on multi-source heterogeneous data fusion includes the following steps:

[0054] S1. Collect data from multiple different types of data sources and monitor data quality;

[0055] S2. Preprocess the collected data, including cleaning and standardization of structured data, key information extraction and structured conversion of semi-structured data, and text processing and feature vector extraction of unstructured data;

[0056] S3. Perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector;

[0057] S4. Use machine learning algorithms to train a fraud detection model, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds the preset threshold.

[0058] In step S1, a collection function is defined for each data source, and data is collected from the corresponding data source at a specific time.

[0059] In step S2, preprocessing operations are performed on structured data, semi-structured data and unstructured data respectively.

[0060] In step S3, data fusion is performed according to the data source weight vector to obtain a fused feature vector.

[0061] In step S4, a support vector machine algorithm is used to train a fraud detection model, and a judgment is made as to whether to trigger an early warning based on a fraud possibility threshold.

[0062] In the embodiment of the present invention, let the data source set be D={D1, D2, ..., D n}, where D i Represents different types of data sources. In this embodiment, it is assumed that D1 is a financial database, including structured data such as balance sheets and income statements; D2 is an internal document library of the enterprise, including semi-structured data such as meeting minutes and internal audit reports; D3 is an online news media data source, including unstructured data such as news reports about the enterprise and public opinion information.

[0063] For each data source D i , define the acquisition function C i (t), which depends on time t, indicating that at time t from D i The operation of collecting data, the collected data is recorded as X i (t), and satisfy the data quality constraint Q[X i (t)]≥τ i , τ i Data source D i The set quality thresholds cover dimensions such as completeness and accuracy, and are monitored in real time through data verification algorithms to ensure the reliability of incoming data.

[0064] Specifically, for data source D1, the collection function C1(t) is an operation based on the database query language. For example, if we want to collect financial data within a certain time period, assuming the time period is [t1, t2], the collection function can be expressed as:

[0065]

[0066] For data source D2, the collection function C2(t) is a search function that accepts keywords and time as parameters and returns a list of document paths that meet the conditions. The collection function can be expressed as C2(t)=read(search(keyword=”audit”, time=t));

[0067] For data source D3, the collection function C3(t) can be a web crawler tool, and the collection function can be expressed as C3(t)=init(parameters={keyword:"XXX company", time: t, website:["website1,..."]});

[0068] For structured data X s (t), let the structured data set be X s , the cleaning function is The cleaned data set is The normalization function is The standardized data set is

[0069] For example, now collect a financial data matrix X s , which represents the sales and cost data of a company over a period of time, as shown below: Here, the first column represents sales, and the second column represents cost. The data in the second row and the first column are missing and are represented by NaN.

[0070] Cleaning function used Fill missing values ​​and clean functions Specifically For the above data matrix X s , the missing value filling calculation for the 2nd row and 1st column is as follows. The mean of the column (sales column) where the missing value is located is So the filled data matrix becomes:

[0071] Assume that the cleaned financial data matrix The standardized data matrix is

[0072] For semi-structured data X ss (t), let the semi-structured data set be X ss , extract key information function as E ss (X ss ), the data set after extracting key information is The structured conversion function is , the data set after structured transformation is

[0073] Specifically, key information extraction can be regarded as a mapping. For example, for a semi-structured data X ss (t) can be a meeting minutes document containing the following text: "At the first quarter meeting of 2023, the company decided to increase marketing expenses to 500,000 yuan." Using natural language processing techniques, such as named entity recognition (NER) or keyword extraction algorithms, key information E can be identified. ss (X ss ) = {"Date":"First Quarter of 2023","Decision":"Increase Marketing Expenses","Amount":"500,000 Yuan"}, this information can be converted into a structured table format

[0074] For unstructured dataX us (t), let the unstructured data set be X us , let the text processing function be P us (Xus ), the data set after text processing is The function for extracting feature vectors is The data set after extracting the feature vector is

[0075] Specifically, for a semi-structured data X us (t), for example, for a news report containing the following text "The company has recently faced some legal challenges, which may affect its future profitability.", first perform text processing, including removing stop words, word segmentation, and then apply TF-IDF or Word2Vec model to generate feature vectors. Suppose we choose TF-IDF method and get the following vocabulary and its corresponding TF-IDF value, specifically "TF-IDF(X us (t)) = {"Company": 0.2, "Law": 0.8, "Challenges": 0.7, "Future": 0.3, "Profit": 0.5}", this news can be represented as a feature vector

[0076] Suppose the structured data feature selection function is Get the eigenvector F s .

[0077] Suppose the semi-structured data feature selection function is Get the eigenvector F ss .

[0078] Suppose the unstructured data feature selection function is Get the eigenvector F us .

[0079] Assume that the data source weight vector is ω=[ω1,ω2,…,ω n ],satisfy The fused feature vector is Among them, F1 is the feature vector F s , F2 is the eigenvector F ss , F3 is the eigenvector F us .

[0080] Specifically, assuming that the weight vector ω=[ω1,ω2,…,ω n ]=[0.5,0.3,0.2], the fused feature vector F us Calculated as F by weighted summation us =[0.95,0.4,1.02];

[0081] Let the fraud detection model be M(F), which is trained using a machine learning algorithm, such as the support vector machine (SVM) algorithm, and let the training function be TM , then M=T M (F), where the input is the fused data feature F.

[0082] Specifically, for each eigenvector F us , assign a label y to each sample, where 0 indicates normal behavior and 1 indicates fraudulent behavior.

[0083] Choose SVM as the classifier and decide whether to use a linear kernel or a nonlinear kernel (such as the RBF kernel). The RBF kernel is usually chosen if the data is not linearly separable.

[0084] Set the regularization parameter C and kernel function parameters, and these parameters can be optimized through cross-validation.

[0085] Create an SVM model and pass the prepared feature vector F us and the corresponding label y are input into the SVM model for training.

[0086] Evaluate model performance on the test dataset. Common evaluation indicators include accuracy, recall, F1 score, etc. If the performance is not ideal, you can return to adjust the model parameters or improve the data preprocessing steps.

[0087] Assume that the fraud possibility threshold is θ, and when M(F)>θ, an early warning is triggered. Assume that the early warning function is 1 indicates that an alert is triggered, and 0 indicates that an alert is not triggered.

[0088] Specifically, assuming that the fraud possibility threshold θ=0.7, when the fraud probability p predicted by the model exceeds this threshold, the warning function W(F) is triggered.

[0089] In summary, the financial fraud detection system and method for multi-source heterogeneous data fusion disclosed in the present invention can use different types of data sources, utilize support vector machine algorithms combined with feature selection technology and weight-based data fusion methods, have higher detection accuracy and reliability, and customized acquisition functions for different data sources, thereby ensuring the high performance of the system and enhancing the scope of application and practicality.

[0090] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A financial fraud detection system based on multi-source heterogeneous data fusion, characterized in that: include: A data collection module, used for collecting data from a plurality of different types of data source sets, wherein the data collection module is provided with data quality constraints; Data preprocessing module, which is used for cleaning and standardizing structured data, extracting key information and structuring semi-structured data, and processing text and extracting feature vectors for unstructured data; The data fusion module is used to perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector; The fraud detection and early warning module is used to train a fraud detection model using a machine learning algorithm, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds a preset threshold.

2. The financial fraud detection system based on multi-source heterogeneous data fusion as claimed in claim 1, characterized in that: In the data collection module, a collection function defined for each data source is provided. The collection function depends on the time and represents the operation of collecting data from the corresponding data source at a specific time.

3. The financial fraud detection system based on multi-source heterogeneous data fusion as claimed in claim 1, characterized in that: The data preprocessing module is provided with: Cleaning functions are used to remove redundancy and errors in structured data; Normalization functions, used to convert structured data into a uniform format; Extract key information function, used to extract key information from semi-structured data; Structural transformation function, used to transform key information extracted from semi-structured data into structured data; Text processing functions, used to convert unstructured data into text data; The feature vector extraction function is used to extract feature vectors from text data.

4. The financial fraud detection system based on multi-source heterogeneous data fusion as claimed in claim 1, characterized in that: The data fusion module is provided with a structured data feature selection function, a semi-structured data feature selection function and an unstructured data feature selection function, which are used to extract feature vectors from the preprocessed data and perform data fusion according to the data source weight vector to obtain a fused feature vector.

5. The financial fraud detection system based on multi-source heterogeneous data fusion as claimed in claim 1, characterized in that: In the fraud detection and early warning module, the fraud detection model is trained using a support vector machine algorithm, the training function input is the fused data features, and the output is the trained fraud detection model.

6. A financial fraud detection method based on multi-source heterogeneous data fusion, characterized in that: The steps include: S1. Collect data from multiple different types of data sources and monitor data quality; S2. Preprocess the collected data, including cleaning and standardization of structured data, key information extraction and structured conversion of semi-structured data, and text processing and feature vector extraction of unstructured data; S3. Perform feature selection on the preprocessed structured data, semi-structured data and unstructured data, and perform data fusion based on the data source weight vector to obtain a fused feature vector; S4. Use machine learning algorithms to train a fraud detection model, input the fused data features for fraud detection, and trigger an early warning when the possibility of fraud exceeds the preset threshold.

7. The financial fraud detection method based on multi-source heterogeneous data fusion as claimed in claim 6, characterized in that: In step S1 , a collection function is defined for each data source, and data is collected from the corresponding data source at a specific time.

8. The financial fraud detection method based on multi-source heterogeneous data fusion as claimed in claim 6, characterized in that: In step S2, preprocessing operations are performed on structured data, semi-structured data and unstructured data respectively.

9. The financial fraud detection method based on multi-source heterogeneous data fusion as claimed in claim 6, characterized in that: In step S3, data fusion is performed according to the data source weight vector to obtain a fused feature vector.

10. The financial fraud detection method based on multi-source heterogeneous data fusion as claimed in claim 6, characterized in that: In step S4, a support vector machine algorithm is used to train a fraud detection model, and a judgment is made as to whether to trigger an early warning based on a fraud possibility threshold.

Images