A financial integrated data management system and method based on cloud computing

Abstract

The present invention discloses a financial integrated data management system and method based on cloud computing, which belongs to the field of artificial intelligence technology. In the present invention, the user sets the needs of the enterprise in financial data management, obtains the financial data of the enterprise, performs data preprocessing, merges duplicate data, and forms a financial data set; designs the overall architecture of the system, deploys it to the cloud computing platform, and transmits the financial data set to the cloud computing platform; sets the threshold of different data in the financial data set, and when it exceeds the set warning threshold, confirms that the data is abnormal; uses the isolation forest to detect abnormal data in the financial data, analyzes the historical change trend of the abnormal financial data, and identifies abnormal patterns; analyzes the abnormal data through the association between the abnormal financial data and the normal financial data in the same time period, as well as the connection between all financial data; uses the decision tree algorithm to analyze the source data that causes the abnormal financial data, generates an abnormal report and notifies.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a financial integrated data management system and method based on cloud computing. Background Art

[0002] In modern enterprises, financial data management is a crucial link. The financial data of an enterprise includes accounting data, budget data, financial statement data, transaction data, and cost data. These data are not only large and complex, but also often come from different data sources and have different formats, making them difficult to process. With the development of cloud computing technology, more and more enterprises are beginning to migrate their financial data to cloud platforms in order to utilize the powerful processing and storage capabilities of cloud computing to improve the efficiency and accuracy of financial data management.

[0003] Existing financial data management systems often face the problem of diverse data sources and non-uniform formats. Integrating and standardizing these data requires a lot of manpower and time, which is inefficient and prone to errors. Existing systems use simple rules and threshold settings to detect abnormal financial data, which cannot fully cope with complex financial anomalies. False positives and false negatives are serious, affecting the effectiveness of financial risk management. Existing systems rely on manual data analysis and decision-making, lack intelligent algorithm support, resulting in insufficient accuracy and timeliness of analysis results. Summary of the invention

[0004] The purpose of the present invention is to provide a financial integrated data management system and method based on cloud computing to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A financial integrated data management method based on cloud computing, the method steps are as follows:

[0007] S100, the user sets the enterprise's requirements for financial data management, obtains the enterprise's financial data from various data sources, cleans and standardizes the data, handles missing values, outliers and duplicate data, integrates data from different sources, merges duplicate data, and forms a financial data set;

[0008] S200. Select AWS as the cloud computing platform, design the overall architecture of the system, which includes data storage, computing and network architecture, and plan data lake, data warehouse and real-time data processing functions; deploy the designed system architecture to the cloud computing platform, implement backup and recovery strategies, monitor system performance in real time, and transmit financial data sets to the cloud computing platform;

[0009] S300. According to the enterprise's needs for financial data management, thresholds for different data in the financial data set are set. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal; the abnormal data in the financial data is detected using the isolation forest; based on the abnormal data detected by the warning threshold and the isolation forest, the historical change trend of the abnormal financial data is analyzed to identify abnormal patterns;

[0010] S400. Analyze abnormal data by the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data; combine the normal financial data threshold and use the decision tree algorithm to analyze and obtain the source data that causes the abnormal financial data; generate a detailed abnormal report for the detected abnormal financial data, and notify relevant financial personnel to handle it.

[0011] According to step S100, the financial data includes accounting data, budget data, financial statement data, transaction data and cost data, and the data sources include ERP system, banking system, procurement system and sales system; accounting data obtains accounting data from the ERP system through API, database connection or file import, budget data obtains budget preparation, budget execution and budget adjustment data from the ERP system through API or file import, financial statement data obtains balance sheet, income statement and cash flow statement data from the ERP system through API, database connection or file import, transaction data obtains sales orders, purchase orders, payment records and collection records from the sales system, procurement system and banking system through API, database connection or file import, and cost data obtains production costs, operating costs and sales costs from the ERP system through API, database connection or file import.

[0012] According to step S100, the missing values ​​of the financial data are filled with the mean and median, and the dehashing algorithm is used to detect and merge duplicate financial data to ensure that all data follows a unified format and unified data units; the table structure and field definition of each type of data are determined, and the data from different data sources are merged and integrated, and a financial data set is created based on the integrated data.

[0013] According to step S200, Amazon S3 is used as a data lake to store various raw financial data and processed data, and different S3 storage buckets are set to store raw data, intermediate processed data, and final data sets respectively; Amazon Redshift is used as a data warehouse to store structured financial data, and the Redshift table structure is designed; AWS Glue is used to perform ETL operations to transform and load data from raw data sources into the data lake and data warehouse, where ETL stands for extraction, transformation, and loading; Glue jobs and crawlers are configured to implement data cleaning, transformation, and loading;

[0014] Use Amazon Kinesis to process real-time data, monitor real-time data streams and analyze them; use Kinesis Data Streams and Kinesis Data Analytics to collect and analyze real-time data streams; use Amazon VPC to create an isolated network environment to ensure the security of data transmission and storage, configure subnets, routing tables and gateways, set security groups and network ACLs, and control access rights; use AWS CloudWatch to monitor system performance, set monitoring indicators and alarm rules, collect and analyze log data through CloudWatch Logs, and ensure the operating status of the system; use the version control function of AWS Backup to regularly back up data, and configure Redshift snapshots and RDS backups;

[0015] Configure and deploy S3 buckets, Redshift clusters, Glue jobs, and EC2 instances on AWS, set up VPC, subnets, routing tables, and security groups, configure automatic backup policies, set up monitoring indicators in AWS CloudWatch, and configure alarm rules; use AWS DataSync to transfer local financial data sets to S3 buckets on the AWS cloud platform, and use GlueETL jobs to process the data transferred to S3 and load it into the Redshift data warehouse.

[0016] Step S300 includes:

[0017] S301. According to the enterprise's needs for financial data management, set warning thresholds for different data in the financial data set. When a certain data exceeds the set warning threshold, confirm that the data is abnormal;

[0018] S302, using the isolation forest algorithm to detect abnormal data in the financial data. The isolation forest isolates data points by constructing random trees. The easier it is to isolate a data point of the financial data, the more likely it is to be an abnormal point. The isolation forest calculates an abnormality score by the average path length of multiple trees.

[0019] S303. Based on the abnormal data detected by the early warning threshold and the isolation forest, the historical change trend and pattern of the abnormal financial data are analyzed by moving average and exponential smoothing, and the abnormal pattern is identified by using K-means.

[0020] According to step S301, when setting the threshold, it is determined based on the statistical analysis of historical data; the financial data set is set to , using the mean and standard deviation Setting the threshold:

[0021] , ;

[0022] Where n represents the total number of data points in the financial dataset;

[0023] Set upper threshold and lower threshold :

[0024] , ;

[0025] Here, k is a constant used when setting the threshold.

[0026] According to step S302, the isolation forest constructs t random trees, each tree The algorithm splits by randomly selecting a feature and a feature value until each data point is completely isolated; for each data point x in the financial dataset, the isolation forest calculates the path length in each tree. , and take the average path length , the formula is as follows:

[0027] ;

[0028] in, It is used to measure the degree of isolation of abnormal data points in financial data sets, thereby identifying abnormal financial data;

[0029] Anomaly score for Isolation Forest The calculation formula is as follows:

[0030] ;

[0031] in, is a normalization constant that adjusts the path length and affects the calculation of the anomaly score, Used to measure the degree of abnormality of a data point; The calculation formula is as follows:

[0032] ;

[0033] in, represents the n-1th harmonic number; The calculation formula is as follows:

[0034] ;

[0035] where g is used to traverse from 1 to n-1; if the anomaly score If it is close to 1, the data point x is considered to be an outlier.

[0036] According to step S303, the moving average is used to identify the trend of financial data, and the formula is as follows:

[0037] ;

[0038] in, is the moving average of time, m is the size of the moving window, It's time Financial data; for an abnormal transaction amount, by calculating the moving average, users can observe its changing trend over time;

[0039] Moving average is used to smooth financial data and help identify long-term trends in the data. By observing the changes in the moving average, you can find the upward or downward trend of the data and identify potential abnormal patterns. For example, by calculating the moving average of the abnormal transaction amount over a period of time, you can observe its trend over time and find out whether the abnormal transactions show a pattern of continuous growth or sudden surge.

[0040] Exponential smoothing takes into account all past data points, giving greater weight to newer data points, and the formula is as follows:

[0041] ;

[0042] in, is the exponential smoothing value at time, is the smoothing coefficient, defined by the user, in the range , It is the financial data point at time; exponential smoothing can reflect the short-term trend of financial data;

[0043] Exponential smoothing takes into account all past data points, but gives greater weight to newer data points, so it can better reflect the short-term trend of financial data. For anomaly detection of financial data, exponential smoothing can capture short-term fluctuations in abnormal data, thereby discovering abnormal patterns more timely.

[0044] K-means clustering is used to divide the financial data set into w clusters, where the data points in each cluster have high similarity. K-means clustering is optimized by minimizing the intra-cluster sum of squares.

[0045] During the execution of K-means clustering, w initial cluster centers are randomly selected, each data point in the financial data set is assigned to the nearest cluster center, and the new centroid of each cluster is calculated until the cluster center no longer changes or the predetermined number of iterations is reached; the Euclidean distance is used The distance between each data point x and the cluster center is measured as follows:

[0046]

[0047] in, is the data point x The value of the dimension, is the cluster center No. The value of the dimension, D is the dimension of the data point, indicating the number of features contained in each data point;

[0048] By calculating the Euclidean distance between a data point and the cluster center, financial data points can be assigned to the nearest cluster center and abnormal patterns with similar characteristics can be identified. For example, data points in a cluster may all have similar abnormal transaction amounts and frequencies, suggesting that there may be the same abnormal cause.

[0049] Update cluster center , the formula is as follows:

[0050] ;

[0051] in, is the set of data points in cluster j, is the number of data points in cluster j; K-means clustering can be used to identify abnormal patterns in financial data.

[0052] By continuously updating the cluster centers, the results of K-means clustering can be optimized, making the data points in each cluster more similar and the abnormal patterns more obvious. For financial data, K-means clustering can group abnormal data points and help identify different types of abnormal patterns, thereby performing anomaly detection and analysis more accurately.

[0053] According to step S400, normal financial data within the same time period as the current abnormal data is extracted through historical data stored in the system to ensure that the data contains the same features; the abnormal data is compared with the normal data of the same time period, and the differences are analyzed to check whether the abnormal transactions are significantly different in transaction amount, transaction time and transaction type, and the significant differences are judged by specific thresholds set by the user; the position and relevance of the abnormal data in the entire financial data set are analyzed to check whether there are other similar abnormal transactions, or whether there are specific time or type patterns;

[0054] The financial data set is divided into a training set and a test set, wherein the training set includes normal financial data and labeled abnormal data, and a decision tree model is constructed using the training data set. The model generates decision rules based on the characteristics of the financial data set to distinguish normal data from abnormal data, wherein the characteristics of the financial data set include transaction amount, transaction time, and transaction type; the detected abnormal financial data is input into the decision tree model to analyze and determine the specific characteristics and sources that cause the abnormality;

[0055] Based on the output of the decision tree model, the characteristics, sources and possible causes of abnormal data are summarized. The output content will point out certain high-risk transaction types or abnormal activities within a specific time period; create a detailed exception report, including the following: specific details of the abnormal data, the time and location of the anomaly detection, the source and cause of the anomaly obtained through analysis, and the recommended treatment measures and prevention suggestions; the report uses charts, tables and text descriptions to display the analysis results, and the system automatically generates and sends the exception report to relevant financial personnel.

[0056] A financial integrated data management system based on cloud computing, comprising:

[0057] Data acquisition and preprocessing module: including: data acquisition unit, data preprocessing unit and data integration unit; the data acquisition unit is responsible for obtaining the financial data management requirements of the enterprise set by the user, and obtaining the financial data of the enterprise from various data sources; the data preprocessing unit is responsible for cleaning and standardizing data, processing missing values, outliers and duplicate data; the data integration unit is responsible for integrating data from different sources, merging duplicate data, and forming a financial data set;

[0058] System deployment and operation and maintenance module: including: system architecture design unit, system deployment unit, backup and recovery unit and system monitoring unit; among them, the system architecture design unit is responsible for designing the overall architecture of the system, which includes data storage, computing and network architecture, planning data lake, data warehouse and real-time data processing functions, the system deployment unit is responsible for deploying the designed system architecture to the cloud computing platform, transmitting the financial data set to the cloud computing platform, the backup and recovery unit is responsible for implementing the backup and recovery strategy, and the system monitoring unit is responsible for real-time monitoring of system performance;

[0059] Anomaly detection and analysis module: including: threshold setting unit, anomaly detection unit and historical trend analysis unit; wherein, the threshold setting unit is responsible for setting the thresholds of different data in the financial data set according to the enterprise's needs in financial data management. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal. The anomaly detection unit is responsible for using the isolation forest to detect abnormal data in the financial data. The historical trend analysis unit is responsible for analyzing the historical change trend of abnormal financial data and identifying abnormal patterns based on the abnormal data detected by the warning threshold and the isolation forest.

[0060] Exception processing and reporting module: including: data association analysis unit, decision tree analysis unit, exception report generation unit and exception notification unit; among them, the data association analysis unit is responsible for analyzing abnormal data through the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data. The decision tree analysis unit is responsible for combining the normal financial data threshold and using the decision tree algorithm to analyze the source data that causes the abnormal financial data. The exception report generation unit is responsible for generating detailed exception reports for the detected abnormal financial data. The exception notification unit is responsible for notifying relevant financial personnel to handle it.

[0061] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0062] The present invention provides an efficient method for data acquisition, cleaning, standardization and integration, which can automatically acquire financial data from various data sources and convert the data into a unified format through cleaning and standardization processing, greatly reducing manpower input and improving the efficiency and accuracy of data integration.

[0063] The present invention uses the isolation forest algorithm for anomaly detection, and effectively improves the accuracy and sensitivity of anomaly detection by constructing random trees to isolate data points. At the same time, combined with the warning threshold and historical trend analysis, it can more accurately identify and locate abnormal financial data.

[0064] The present invention provides intelligent financial data analysis and decision support functions by using decision tree algorithm and K-means clustering algorithm. The system can analyze the historical trend of abnormal data, identify abnormal patterns, and automatically generate abnormal reports to help enterprises understand their financial status more accurately and make timely and correct decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 It is a schematic diagram of the steps of a financial integrated data management method based on cloud computing of the present invention;

[0067] Figure 2 It is a system structure diagram of a financial integrated data management system based on cloud computing of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0070] According to one embodiment of the present invention, Figure 1 A step diagram of a financial integrated data management method based on cloud computing is shown in the figure. A financial integrated data management method based on cloud computing, the method steps are as follows:

[0071] S100, the user sets the enterprise's requirements for financial data management, obtains the enterprise's financial data from various data sources, cleans and standardizes the data, handles missing values, outliers and duplicate data, integrates data from different sources, merges duplicate data, and forms a financial data set;

[0072] S200. Select AWS as the cloud computing platform, design the overall architecture of the system, which includes data storage, computing and network architecture, and plan data lake, data warehouse and real-time data processing functions; deploy the designed system architecture to the cloud computing platform, implement backup and recovery strategies, monitor system performance in real time, and transmit financial data sets to the cloud computing platform;

[0073] S300. According to the enterprise's needs for financial data management, thresholds for different data in the financial data set are set. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal; the abnormal data in the financial data is detected using the isolation forest; based on the abnormal data detected by the warning threshold and the isolation forest, the historical change trend of the abnormal financial data is analyzed to identify abnormal patterns;

[0074] S400. Analyze abnormal data by the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data; combine the normal financial data threshold and use the decision tree algorithm to analyze and obtain the source data that causes the abnormal financial data; generate a detailed abnormal report for the detected abnormal financial data, and notify relevant financial personnel to handle it.

[0075] According to step S100, the financial data includes accounting data, budget data, financial statement data, transaction data and cost data, and the data sources include ERP system, banking system, procurement system and sales system; accounting data obtains accounting data from the ERP system through API, database connection or file import, budget data obtains budget preparation, budget execution and budget adjustment data from the ERP system through API or file import, financial statement data obtains balance sheet, income statement and cash flow statement data from the ERP system through API, database connection or file import, transaction data obtains sales orders, purchase orders, payment records and collection records from the sales system, procurement system and banking system through API, database connection or file import, and cost data obtains production costs, operating costs and sales costs from the ERP system through API, database connection or file import.

[0076] According to step S100, the missing values ​​of the financial data are filled with the mean and median, and the dehashing algorithm is used to detect and merge duplicate financial data to ensure that all data follows a unified format and unified data units; the table structure and field definition of each type of data are determined, and the data from different data sources are merged and integrated, and a financial data set is created based on the integrated data.

[0077] According to step S200, Amazon S3 is used as a data lake to store various raw financial data and processed data, and different S3 storage buckets are set to store raw data, intermediate processed data, and final data sets respectively; Amazon Redshift is used as a data warehouse to store structured financial data, and the Redshift table structure is designed; AWS Glue is used to perform ETL operations to transform and load data from raw data sources into the data lake and data warehouse, where ETL stands for extraction, transformation, and loading; Glue jobs and crawlers are configured to implement data cleaning, transformation, and loading;

[0078] Use Amazon Kinesis to process real-time data, monitor real-time data streams and analyze them; use Kinesis Data Streams and Kinesis Data Analytics to collect and analyze real-time data streams; use Amazon VPC to create an isolated network environment to ensure the security of data transmission and storage, configure subnets, routing tables and gateways, set security groups and network ACLs, and control access rights; use AWS CloudWatch to monitor system performance, set monitoring indicators and alarm rules, collect and analyze log data through CloudWatch Logs, and ensure the operating status of the system; use the version control function of AWS Backup to regularly back up data, and configure Redshift snapshots and RDS backups;

[0079] Configure and deploy S3 buckets, Redshift clusters, Glue jobs, and EC2 instances on AWS, set up VPC, subnets, routing tables, and security groups, configure automatic backup policies, set up monitoring indicators in AWS CloudWatch, and configure alarm rules; use AWS DataSync to transfer local financial data sets to S3 buckets on the AWS cloud platform, and use GlueETL jobs to process the data transferred to S3 and load it into the Redshift data warehouse.

[0080] Step S300 includes:

[0081] S301. According to the enterprise's needs for financial data management, set warning thresholds for different data in the financial data set. When a certain data exceeds the set warning threshold, confirm that the data is abnormal;

[0082] S302, using the isolation forest algorithm to detect abnormal data in the financial data. The isolation forest isolates data points by constructing random trees. The easier it is to isolate a data point of the financial data, the more likely it is to be an abnormal point. The isolation forest calculates an abnormality score by the average path length of multiple trees.

[0083] S303. Based on the abnormal data detected by the early warning threshold and the isolation forest, the historical change trend and pattern of the abnormal financial data are analyzed by moving average and exponential smoothing, and the abnormal pattern is identified by using K-means.

[0084] According to step S301, when setting the threshold, it is determined based on the statistical analysis of historical data; the financial data set is set to , using the mean and standard deviation Setting the threshold:

[0085] , ;

[0086] Where n represents the total number of data points in the financial dataset;

[0087] Set upper threshold and lower threshold :

[0088] 181.06, ;

[0089] Here, k is a constant used when setting the threshold, and is set to 2 in this embodiment.

[0090] Therefore, the warning threshold is [108.94, 181.06], and any value above 181.06 or below 108.94 will be considered abnormal.

[0091] According to step S302, the isolation forest constructs t random trees, t is set to 100, and each tree The algorithm splits by randomly selecting a feature and a feature value until each data point is completely isolated; for each data point x in the financial dataset, the isolation forest calculates the path length in each tree. , and take the average path length , the formula is as follows:

[0092] ;

[0093] in, It is used to measure the degree of isolation of abnormal data points in financial data sets, thereby identifying abnormal financial data;

[0094] In this embodiment, n=10, represents the n-1th harmonic number; The calculation formula is as follows:

[0095] ;

[0096] where g is used to traverse from 1 to n-1; if the anomaly score If it is close to 1, the data point x is considered to be an outlier.

[0097] is a normalization constant that adjusts the path length and affects the calculation of the anomaly score, Used to measure the degree of abnormality of a data point; The calculation formula is as follows:

[0098] ;

[0099] Anomaly score for Isolation Forest The calculation formula is as follows:

[0100] .

[0101] Since the anomaly score is close to 1, the data point is considered to be anomaly. Therefore, in this embodiment, the isolation forest algorithm is used to identify data points with higher anomaly scores.

[0102] According to step S303, the moving average is used to identify the trend of financial data, and the formula is as follows:

[0103] ;

[0104] in, is the moving average of time, m is the size of the moving window, It's time Financial data; for a certain abnormal transaction amount, by calculating the moving average, the user can observe its changing trend over time; in this embodiment, m=3;

[0105] For the 5th data point:

[0106] ;

[0107] Exponential smoothing takes into account all past data points, giving greater weight to newer data points, and the formula is as follows:

[0108] ;

[0109] in, is the exponential smoothing value at time, is the smoothing coefficient, which is defined by the user as 0.5 and has a range of , It is a financial data point at time; exponential smoothing can reflect the short-term trend of financial data; there is an initial exponential smoothing value , then for the second data point:

[0110] ;

[0111] For the third data point:

[0112] ;

[0113] Exponential smoothing takes into account all past data points, but gives greater weight to newer data points, so it can better reflect the short-term trend of financial data. For anomaly detection of financial data, exponential smoothing can capture short-term fluctuations in abnormal data, thereby discovering abnormal patterns more timely.

[0114] K-means clustering is used to partition the financial dataset into w = 3 clusters. The data points in each cluster have high similarity. K-means clustering is optimized by minimizing the intra-cluster sum of squares.

[0115] During the execution of K-means clustering, three initial cluster centers are randomly selected, each data point in the financial data set is assigned to the nearest cluster center, and the new centroid of each cluster is calculated until the cluster center no longer changes or the predetermined number of iterations is reached; the Euclidean distance is used The distance between each data point x and the cluster center is measured as follows:

[0116]

[0117] in, is the data point x The value of the dimension, is the cluster center No. The value of the dimension, D is the dimension of the data point, which is set to 2 in this embodiment, indicating the number of features contained in each data point;

[0118] By calculating the Euclidean distance between a data point and the cluster center, financial data points can be assigned to the nearest cluster center and abnormal patterns with similar characteristics can be identified. For example, data points in a cluster may all have similar abnormal transaction amounts and frequencies, suggesting that there may be the same abnormal cause.

[0119] Update cluster center , the formula is as follows:

[0120] ;

[0121] in, is the set of data points in cluster j, is the number of data points in cluster j; K-means clustering can be used to identify abnormal patterns in financial data.

[0122] By continuously updating the cluster centers, the results of K-means clustering can be optimized, making the data points in each cluster more similar and the abnormal patterns more obvious. For financial data, K-means clustering can group abnormal data points and help identify different types of abnormal patterns, thereby performing anomaly detection and analysis more accurately.

[0123] According to step S400, normal financial data within the same time period as the current abnormal data is extracted through historical data stored in the system to ensure that the data contains the same features; the abnormal data is compared with the normal data of the same time period, and the differences are analyzed to check whether the abnormal transactions are significantly different in transaction amount, transaction time and transaction type, and the significant differences are judged by specific thresholds set by the user; the position and relevance of the abnormal data in the entire financial data set are analyzed to check whether there are other similar abnormal transactions, or whether there are specific time or type patterns;

[0124] The financial data set is divided into a training set and a test set, wherein the training set includes normal financial data and labeled abnormal data, and a decision tree model is constructed using the training data set. The model generates decision rules based on the characteristics of the financial data set to distinguish normal data from abnormal data, wherein the characteristics of the financial data set include transaction amount, transaction time, and transaction type; the detected abnormal financial data is input into the decision tree model to analyze and determine the specific characteristics and sources that cause the abnormality;

[0125] Based on the output of the decision tree model, the characteristics, sources and possible causes of abnormal data are summarized. The output content will point out certain high-risk transaction types or abnormal activities within a specific time period; create a detailed exception report, including the following: specific details of the abnormal data, the time and location of the anomaly detection, the source and cause of the anomaly obtained through analysis, and the recommended treatment measures and prevention suggestions; the report uses charts, tables and text descriptions to display the analysis results, and the system automatically generates and sends the exception report to relevant financial personnel.

[0126] According to another embodiment of the present invention, Figure 2 As shown in the system structure diagram of a financial integrated data management system based on cloud computing, a financial integrated data management system based on cloud computing includes:

[0127] Data acquisition and preprocessing module: including: data acquisition unit, data preprocessing unit and data integration unit; the data acquisition unit is responsible for obtaining the financial data management requirements of the enterprise set by the user, and obtaining the financial data of the enterprise from various data sources; the data preprocessing unit is responsible for cleaning and standardizing data, processing missing values, outliers and duplicate data; the data integration unit is responsible for integrating data from different sources, merging duplicate data, and forming a financial data set;

[0128] System deployment and operation and maintenance module: including: system architecture design unit, system deployment unit, backup and recovery unit and system monitoring unit; among them, the system architecture design unit is responsible for designing the overall architecture of the system, which includes data storage, computing and network architecture, planning data lake, data warehouse and real-time data processing functions, the system deployment unit is responsible for deploying the designed system architecture to the cloud computing platform, transmitting the financial data set to the cloud computing platform, the backup and recovery unit is responsible for implementing the backup and recovery strategy, and the system monitoring unit is responsible for real-time monitoring of system performance;

[0129] Anomaly detection and analysis module: including: threshold setting unit, anomaly detection unit and historical trend analysis unit; wherein, the threshold setting unit is responsible for setting the thresholds of different data in the financial data set according to the enterprise's needs in financial data management. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal. The anomaly detection unit is responsible for using the isolation forest to detect abnormal data in the financial data. The historical trend analysis unit is responsible for analyzing the historical change trend of abnormal financial data and identifying abnormal patterns based on the abnormal data detected by the warning threshold and the isolation forest.

[0130] Exception processing and reporting module: including: data association analysis unit, decision tree analysis unit, exception report generation unit and exception notification unit; among them, the data association analysis unit is responsible for analyzing abnormal data through the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data. The decision tree analysis unit is responsible for combining the normal financial data threshold and using the decision tree algorithm to analyze the source data that causes the abnormal financial data. The exception report generation unit is responsible for generating detailed exception reports for the detected abnormal financial data. The exception notification unit is responsible for notifying relevant financial personnel to handle it.

[0131] In this embodiment, the data association analysis unit obtains the following results:

[0132] Abnormal data points: 210; normal data in the same time period: 120, 130, 125, 140, 145, 150, 135,160, 170, 175;

[0133] The decision tree analysis unit uses the decision tree algorithm to analyze the source data that causes abnormal financial data. We use the following decision tree rules: Feature 1: transaction amount, Feature 2: transaction frequency;

[0134] By training the decision tree model, it is found that transactions with transaction amounts higher than 180 and abnormal transaction frequencies are abnormal transactions.

[0135] Generate a detailed exception report, which contains the following general contents:

[0136] Abnormal data points: 210

[0137] Testing time: June 12, 2024

[0138] Abnormal type: Exceeding the upper threshold

[0139] Historical data comparison: Data points: 210, normal range: 108.94 - 181.06;

[0140] Analysis of abnormal reasons: Through decision tree algorithm analysis, the abnormal reasons may be abnormally high transaction amounts and high transaction frequencies;

[0141] Historical trend: average transaction amount in the last month: 145; standard deviation of transaction amount in the last month: 18.03; recent transaction amount gradually increased

[0142] Correlation data analysis: Normal data in the same time period: 120, 130, 125, 140, 145, 150, 135, 160, 170, 175; The abnormal transaction amount is significantly different from the normal transaction amount in the same time period;

[0143] The abnormal notification unit notifies relevant financial personnel for processing.

[0144] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0145] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A financial integrated data management method based on cloud computing, characterized in that: The method steps are as follows: S100, the user sets the enterprise's requirements for financial data management, obtains the enterprise's financial data from various data sources, cleans and standardizes the data, handles missing values, outliers and duplicate data, integrates data from different sources, merges duplicate data, and forms a financial data set; S200. Select AWS as the cloud computing platform, design the overall architecture of the system, which includes data storage, computing and network architecture, and plan data lake, data warehouse and real-time data processing functions; deploy the designed system architecture to the cloud computing platform, implement backup and recovery strategies, monitor system performance in real time, and transmit financial data sets to the cloud computing platform; S300. Setting thresholds for different data in the financial data set according to the enterprise's needs for financial data management. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal. Detecting outliers in financial data using Isolation Forests; Analyze the historical trend of abnormal financial data and identify abnormal patterns based on abnormal data detected by early warning thresholds and isolation forests; Step S300 includes: S301. According to the enterprise's needs for financial data management, set warning thresholds for different data in the financial data set. When a certain data exceeds the set warning threshold, confirm that the data is abnormal; S302, using the isolation forest algorithm to detect abnormal data in the financial data. The isolation forest isolates data points by constructing random trees. The easier it is to isolate a data point of the financial data, the more likely it is to be an abnormal point. The isolation forest calculates an abnormality score by the average path length of multiple trees. S303, based on the abnormal data detected by the early warning threshold and the isolation forest, the historical change trend and pattern of the abnormal financial data are analyzed by moving average and exponential smoothing, and the abnormal pattern is identified by using K-means; According to step S303, the moving average is used to identify the trend of financial data, and the formula is as follows: ； in, is the moving average of time, m is the size of the moving window, It's time Financial data; for an abnormal transaction amount, by calculating the moving average, users can observe its changing trend over time; Exponential smoothing takes into account all past data points, giving greater weight to newer data points, and the formula is as follows: ； in, is the exponential smoothing value at time, is the smoothing coefficient, defined by the user, in the range , It is the financial data point at time; exponential smoothing can reflect the short-term trend of financial data; K-means clustering is used to divide the financial data set into w clusters, where the data points in each cluster have high similarity. K-means clustering is optimized by minimizing the intra-cluster sum of squares. During the execution of K-means clustering, w initial cluster centers are randomly selected, each data point in the financial data set is assigned to the nearest cluster center, and the new centroid of each cluster is calculated until the cluster center no longer changes or the predetermined number of iterations is reached; the Euclidean distance is used The distance between each data point x and the cluster center is measured as follows: in, is the data point x The value of the dimension, is the cluster center No. The value of the dimension, D is the dimension of the data point, indicating the number of features contained in each data point; Update cluster center , the formula is as follows: ； in, is the set of data points in cluster j, is the number of data points in cluster j; K-means clustering can be used to identify abnormal patterns in financial data; S400. Analyze abnormal data by the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data; combine the normal financial data threshold and use the decision tree algorithm to analyze and obtain the source data that causes the abnormal financial data; generate a detailed abnormal report for the detected abnormal financial data, and notify relevant financial personnel to handle it.

2. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S100, the financial data includes accounting data, budget data, financial statement data, transaction data and cost data, and the data sources include ERP system, banking system, procurement system and sales system; accounting data obtains accounting data from the ERP system through API, database connection or file import, budget data obtains budget preparation, budget execution and budget adjustment data from the ERP system through API or file import, financial statement data obtains balance sheet, income statement and cash flow statement data from the ERP system through API, database connection or file import, transaction data obtains sales orders, purchase orders, payment records and collection records from the sales system, procurement system and banking system through API, database connection or file import, and cost data obtains production costs, operating costs and sales costs from the ERP system through API, database connection or file import.

3. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S100, the missing values ​​of the financial data are filled with the mean and median, and the dehashing algorithm is used to detect and merge duplicate financial data to ensure that all data follows a unified format and unified data units; the table structure and field definition of each type of data are determined, and the data from different data sources are merged and integrated, and a financial data set is created based on the integrated data.

4. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S200, Amazon S3 is used as a data lake to store various raw financial data and processed data, and different S3 storage buckets are set to store raw data, intermediate processed data, and final data sets respectively; Amazon Redshift is used as a data warehouse to store structured financial data, and the Redshift table structure is designed; AWS Glue is used to perform ETL operations to transform and load data from the raw data source into the data lake and data warehouse, where ETL stands for extraction, transformation, and loading; Glue jobs and crawlers are configured to implement data cleaning, transformation, and loading; Use Amazon Kinesis to process real-time data, monitor real-time data streams and analyze them; use Kinesis Data Streams and Kinesis Data Analytics to collect and analyze real-time data streams; use Amazon VPC to create an isolated network environment to ensure the security of data transmission and storage, configure subnets, routing tables and gateways, set security groups and network ACLs, and control access rights; use AWS CloudWatch to monitor system performance, set monitoring indicators and alarm rules, collect and analyze log data through CloudWatch Logs, and ensure the operating status of the system; use the version control function of AWS Backup to regularly back up data, and configure Redshift snapshots and RDS backups; Configure and deploy S3 buckets, Redshift clusters, Glue jobs, and EC2 instances on AWS, set up VPC, subnets, routing tables, and security groups, configure automatic backup policies, set up monitoring indicators in AWS CloudWatch, and configure alarm rules; use AWS DataSync to transfer local financial data sets to S3 buckets on the AWS cloud platform, and use Glue ETL jobs to process the data transferred to S3 and load it into the Redshift data warehouse.

5. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S301, when setting the threshold, it is determined based on the statistical analysis of historical data; the financial data set is set to , using the mean and standard deviation Setting the threshold: ， ； Where n represents the total number of data points in the financial dataset; Set upper threshold and lower threshold : ， ； Here, k is a constant used when setting the threshold.

6. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S302, the isolation forest constructs t random trees, each tree The algorithm splits by randomly selecting a feature and a feature value until each data point is completely isolated; for each data point x in the financial dataset, the isolation forest calculates the path length in each tree. , and take the average path length , the formula is as follows: ； in, It is used to measure the degree of isolation of abnormal data points in financial data sets, thereby identifying abnormal financial data; Anomaly score for Isolation Forest The calculation formula is as follows: ； in, is a normalization constant that adjusts the path length and affects the calculation of the anomaly score, Used to measure the degree of abnormality of a data point; The calculation formula is as follows: ； in, represents the n-1th harmonic number; The calculation formula is as follows: ； where g is used to traverse from 1 to n-1; if the anomaly score If it is close to 1, the data point x is considered to be an outlier.

7. The financial integrated data management method based on cloud computing according to claim 1, characterized in that: According to step S400, normal financial data within the same time period as the current abnormal data is extracted through historical data stored in the system to ensure that the data contains the same features; the abnormal data is compared with the normal data of the same time period, and the differences are analyzed to check whether the abnormal transactions are significantly different in transaction amount, transaction time and transaction type, and the significant differences are judged by specific thresholds set by the user; the position and relevance of the abnormal data in the entire financial data set are analyzed to check whether there are other similar abnormal transactions, or whether there are specific time or type patterns; The financial data set is divided into a training set and a test set, wherein the training set includes normal financial data and labeled abnormal data, and a decision tree model is constructed using the training data set. The model generates decision rules based on the characteristics of the financial data set to distinguish normal data from abnormal data, wherein the characteristics of the financial data set include transaction amount, transaction time, and transaction type; the detected abnormal financial data is input into the decision tree model to analyze and determine the specific characteristics and sources that cause the abnormality; Based on the output of the decision tree model, the characteristics, sources and possible causes of abnormal data are summarized. The output content will point out certain high-risk transaction types or abnormal activities within a specific time period; create a detailed exception report, including the following: specific details of the abnormal data, the time and location of the anomaly detection, the source and cause of the anomaly obtained through analysis, and the recommended treatment measures and prevention suggestions; the report uses charts, tables and text descriptions to display the analysis results, and the system automatically generates and sends the exception report to relevant financial personnel.

8. A financial integrated data management system based on cloud computing, using a financial integrated data management method based on cloud computing according to any one of claims 1 to 7, characterized in that: include: Data acquisition and preprocessing module: including: data acquisition unit, data preprocessing unit and data integration unit; the data acquisition unit is responsible for obtaining the financial data management requirements of the enterprise set by the user, and obtaining the financial data of the enterprise from various data sources; the data preprocessing unit is responsible for cleaning and standardizing data, processing missing values, outliers and duplicate data; the data integration unit is responsible for integrating data from different sources, merging duplicate data, and forming a financial data set; System deployment and operation and maintenance module: including: system architecture design unit, system deployment unit, backup and recovery unit and system monitoring unit; among them, the system architecture design unit is responsible for designing the overall architecture of the system, which includes data storage, computing and network architecture, planning data lake, data warehouse and real-time data processing functions, the system deployment unit is responsible for deploying the designed system architecture to the cloud computing platform, transmitting the financial data set to the cloud computing platform, the backup and recovery unit is responsible for implementing the backup and recovery strategy, and the system monitoring unit is responsible for real-time monitoring of system performance; Anomaly detection and analysis module: including: threshold setting unit, anomaly detection unit and historical trend analysis unit; wherein, the threshold setting unit is responsible for setting the thresholds of different data in the financial data set according to the enterprise's needs in financial data management. When a certain data exceeds the set warning threshold, the system confirms that the data is abnormal. The anomaly detection unit is responsible for using the isolation forest to detect abnormal data in the financial data. The historical trend analysis unit is responsible for analyzing the historical change trend of abnormal financial data and identifying abnormal patterns based on the abnormal data detected by the warning threshold and the isolation forest. Exception processing and reporting module: including: data association analysis unit, decision tree analysis unit, exception report generation unit and exception notification unit; among them, the data association analysis unit is responsible for analyzing abnormal data through the association between abnormal financial data and normal financial data in the same time period, as well as the connection between all financial data. The decision tree analysis unit is responsible for combining the normal financial data threshold and using the decision tree algorithm to analyze the source data that causes the abnormal financial data. The exception report generation unit is responsible for generating detailed exception reports for the detected abnormal financial data. The exception notification unit is responsible for notifying relevant financial personnel to handle it.

Images