Real-time transaction monitoring and analysis system and method of financial big data platform

Through distributed data acquisition system and edge computing technology, combined with machine learning and deep learning algorithms, real-time acquisition and high concurrency processing of financial data are achieved, real-time and performance bottlenecks of existing systems are solved, and transaction risk identification and Market volatility response capability.

CN119067780AInactive Publication Date: 2024-12-03HEBEI FINANCE UNIV +1
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411200212.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing financial transaction monitoring system is difficult to realize real-time data processing when facing high-frequency trading and algorithmic trading, resulting in transaction delays and risk identification lag. There are performance bottlenecks when processing large-scale diversified financial data, which cannot provide efficient Real-time monitoring service.

Method used

Use distributed data acquisition system and edge computing technology to collect and initial processing of financial data, combine machine learning-based preprocessing algorithms, big data stream processing frameworks and deep learning algorithms to analyze transaction data in real time, and use reinforcement learning algorithms to optimize the system , realize high concurrent processing and real-time risk identification.

Benefits of technology

It significantly reduces data transmission delay, improves real-time response capabilities to market volatility and transaction risks, enhances data processing accuracy and system performance, and ensures efficient monitoring and analysis capabilities in complex market environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067780A_ABST
    Figure CN119067780A_ABST
Patent Text Reader

Abstract

The invention discloses a real-time transaction monitoring and analysis system and method for a financial big data platform, and relates to the technical field of financial transaction monitoring, and the method comprises the following steps: 1, building a distributed data collection system, carrying out the real-time collection of financial data from different transaction sources, and enabling the distributed data collection system to carry out the real-time collection of the financial data through employing an edge calculation technology, the acquisition nodes are distributed near a plurality of transaction terminals. Through a distributed data acquisition system and an edge computing technology, real-time acquisition and primary processing of financial data are realized, data transmission delay is remarkably reduced, the real-time response capability of the system to market fluctuation and transaction risks is improved, and the real-time performance of the system is improved by combining a preprocessing algorithm based on machine learning, a big data stream processing framework and a deep learning algorithm. The data processing accuracy and the system performance are improved, the high-concurrency processing requirement of large-scale data can be met, the system is continuously optimized through the reinforcement learning algorithm, and the adaptability to the complex market environment is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of financial transaction monitoring, and in particular to a real-time transaction monitoring and analysis system and method for a financial big data platform. Background Art

[0002] With the continuous development of the financial market, the transaction volume and transaction speed have increased significantly, and the complexity and scale of financial data have also increased. Traditional financial transaction monitoring systems are often based on batch processing mode, analyzing transaction data at certain time intervals. Although this method can provide certain transaction analysis capabilities, it is often unable to timely discover and respond to sudden market fluctuations and potential transaction risks due to the inability to process and monitor transaction data in real time. In addition, with the popularization of high-frequency trading and algorithmic trading, the financial market's requirements for data processing speed and accuracy are constantly increasing. Existing transaction monitoring systems are difficult to meet these requirements, which can easily cause problems such as transaction delays and risk identification lags.

[0003] At present, some financial institutions have begun to use big data technology for transaction monitoring and analysis. However, these systems usually lack real-time and accuracy, and cannot identify anomalies or potential risks at the moment of transaction. At the same time, due to the limitation of data processing capacity, these systems are prone to data processing bottlenecks when facing large-scale and diverse financial data, resulting in system performance degradation and inability to provide efficient real-time monitoring services. Therefore, how to design a monitoring and analysis system that can process large-scale financial data in real time in a high-concurrency environment and can promptly identify potential transaction risks has become an important challenge facing the current financial industry.

[0004] The prior art has the following deficiencies:

[0005] The existing financial transaction monitoring system has two main deficiencies: First, it lacks real-time performance, making it difficult to detect and respond to sudden market fluctuations and potential transaction risks in a timely manner. This delay may lead to abnormal market fluctuations or financial fraud not being detected and handled in a timely manner, thus posing a threat to the stability of financial institutions and markets. Second, when faced with large-scale and diverse financial data, the system performance has bottlenecks and limited processing capabilities, making it difficult to provide efficient real-time monitoring services while ensuring analytical accuracy. These problems have seriously restricted the application and effectiveness of financial big data platforms in real-time transaction monitoring, and also limited the ability of financial institutions to respond to market changes in a timely manner.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a real-time transaction monitoring and analysis system and method for a financial big data platform. Through a distributed data acquisition system and edge computing technology, real-time acquisition and preliminary processing of financial data are realized, data transmission delays are significantly reduced, and the system's real-time response capability to market fluctuations and transaction risks is improved. Combined with a preprocessing algorithm based on machine learning, a big data stream processing framework, and a deep learning algorithm, the accuracy of data processing and system performance are improved, and it can cope with the high-concurrency processing requirements of large-scale data. The system is continuously optimized through a reinforcement learning algorithm, thereby enhancing the adaptability to complex market environments, so as to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a real-time transaction monitoring and analysis method of a financial big data platform, comprising the following steps:

[0009] Step 1: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays.

[0010] Step 2: Preprocess the collected financial data using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers;

[0011] Step 3: Use the big data stream processing framework to perform real-time streaming processing on the preprocessed data;

[0012] Step 4: Based on real-time streaming processing, deep learning algorithms are used to analyze transaction data in real time to identify potential market fluctuations and transaction anomalies;

[0013] Step 5: Display the analysis results through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users and provide a multi-dimensional interactive interface;

[0014] Step 6: Generate transaction risk warnings in real time based on the analysis results, and automatically execute preset risk control measures through smart contract technology;

[0015] Step 7: Continuously monitor and optimize the operating status of the entire system. By introducing reinforcement learning algorithms, dynamically adjust data processing strategies and model parameters according to actual operating conditions to ensure that the system maintains efficient real-time monitoring and analysis capabilities in the face of a changing market environment.

[0016] Preferably, the distributed data collection system utilizes edge computing technology to distribute the collection nodes near multiple transaction terminals. The specific steps are as follows:

[0017] S1.1. Deploy edge computing nodes near multiple transaction terminals to form a distributed collection network;

[0018] S1.2. After deploying edge computing nodes, the system will implement local data processing and caching functions on each node. The edge computing nodes directly connect to the transaction terminals to obtain transaction data in real time and perform preliminary processing locally.

[0019] S1.3, by establishing a distributed transmission protocol, the processed data is transmitted from each edge computing node to the central server or other required nodes for further processing;

[0020] S1.4. Dynamically manage and optimize edge computing nodes through the central management platform, monitor the operating status, resource usage and data transmission efficiency of each node in real time, and dynamically adjust the node configuration and data collection strategy according to the fluctuation of transaction volume and node load.

[0021] Preferably, the collected financial data is preprocessed using an automated data preprocessing algorithm based on machine learning, and the specific steps for excluding irrelevant data and outliers are as follows:

[0022] S2.1. Clean the collected financial data and standardize its format;

[0023] S2.2. After data cleaning is completed, irrelevant data in the data set is filtered out. Irrelevant data refers to redundant data that has no direct relationship with the transaction monitoring and analysis objectives;

[0024] S2.3, use machine learning algorithms to detect and remove outliers in the data;

[0025] S2.4. The remaining data will be normalized and feature engineering will be performed to improve the analysis performance of the data.

[0026] Preferably, in step three, by building a high-concurrency data processing channel, it is ensured that the system can process data at the moment of input, avoiding delays caused by data accumulation.

[0027] Preferably, the specific steps of using a large data stream processing framework to perform real-time streaming processing on the preprocessed data are as follows:

[0028] S4.1. Build an efficient data stream input pipeline to introduce pre-processed financial data into the big data stream processing framework in real time;

[0029] S4.2. Partition the input real-time data stream to achieve parallel computing. Partitioning is to divide the data stream into multiple independent sub-streams according to certain key fields of the data, thereby allowing multiple processors to process different sub-streams in parallel.

[0030] S4.3. In order to realize the time series analysis and trend prediction of financial data, the system needs to perform time window operation on the streaming data. The window operation aggregates and analyzes the data according to time periods by defining sliding windows or tumbling windows.

[0031] S4.4. After partitioning and windowing operations, perform real-time analysis and anomaly detection on the processed data stream.

[0032] Preferably, in step six, when abnormal trading behavior is detected, the relevant accounts are automatically frozen or the trading strategy is adjusted to reduce financial risks.

[0033] The real-time transaction monitoring and analysis system of the financial big data platform includes distributed data acquisition module, data preprocessing module, real-time stream processing module, deep learning analysis module, visualization display module, smart contract risk control module and system monitoring and optimization module;

[0034] Distributed data collection module: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays;

[0035] Data preprocessing module: preprocesses the collected financial data, using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers;

[0036] Real-time stream processing module: Use the big data stream processing framework to perform real-time stream processing on pre-processed data;

[0037] Deep learning analysis module: Based on real-time streaming processing, deep learning algorithms are used to analyze transaction data in real time to identify potential market fluctuations and transaction anomalies;

[0038] Visualization display module: The analysis results are displayed through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users, and provides a multi-dimensional interactive interface;

[0039] Smart contract risk control module: Based on the analysis results, it generates transaction risk warnings in real time and automatically executes preset risk control measures through smart contract technology;

[0040] System monitoring and optimization module: Continuously monitor and optimize the operating status of the entire system. By introducing reinforcement learning algorithms, the data processing strategy and model parameters are dynamically adjusted according to the actual operating conditions to ensure that the system maintains efficient real-time monitoring and analysis capabilities in the face of a changing market environment.

[0041] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0042] The present invention realizes real-time collection of financial data from different transaction sources by establishing a distributed data collection system and utilizing edge computing technology, significantly reducing the delay in data transmission. Since the collection nodes are distributed near multiple transaction terminals, the system can quickly and directly acquire data and perform preliminary processing through edge computing technology. This real-time performance greatly improves the system's response speed to sudden market fluctuations and potential transaction risks, enabling financial institutions to monitor market dynamics and take rapid countermeasures at the moment of transaction, thereby effectively reducing the risk of abnormal market fluctuations or financial fraud not being discovered in a timely manner due to delays.

[0043] The present invention adopts an automated data preprocessing algorithm based on machine learning, and uses a big data stream processing framework and a deep learning algorithm to efficiently preprocess and analyze large-scale, diversified financial data in real time. This not only improves the accuracy of data processing, can effectively eliminate irrelevant data and outliers, but also enhances the overall performance of the system, enabling it to cope with the high concurrency processing requirements of large-scale data. In addition, the use of deep learning algorithms for real-time analysis of transaction data can more accurately identify potential market fluctuations and transaction anomalies. At the same time, the system is continuously optimized through reinforcement learning algorithms to ensure that in the face of a constantly changing market environment, the system can still maintain efficient real-time monitoring and analysis capabilities. This enables financial institutions to improve their adaptability to market changes while ensuring analysis accuracy, and effectively respond to the challenges brought about by a complex market environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0045] Figure 1 The present invention is a method flow chart of the real-time transaction monitoring and analysis method of the financial big data platform.

[0046] Figure 2 It is a module schematic diagram of the real-time transaction monitoring and analysis system of the financial big data platform of the present invention. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0048] The present invention provides Figure 1 The real-time transaction monitoring and analysis method of the financial big data platform shown includes the following steps:

[0049] Step 1: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays.

[0050] The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals. The specific steps are as follows:

[0051] S1.1. Deploy edge computing nodes near multiple transaction terminals to form a distributed collection network;

[0052] Each edge computing node is equipped with a powerful processor and storage device, capable of independently performing data collection, preliminary processing, and local storage tasks. These nodes are usually physically close to the trading terminal to minimize data transmission delays. When deploying, it is necessary to consider the geographical location of the trading terminal, network conditions, and the allocation of computing resources to ensure that more nodes are deployed in areas with high transaction volume to share the load of data collection and improve the efficiency and stability of the entire system.

[0053] S1.2. After deploying edge computing nodes, the system will implement local data processing and caching functions on each node. The edge computing nodes directly connect to the transaction terminals to obtain transaction data in real time and perform preliminary processing locally.

[0054] Preliminary processing such as data cleaning, compression and format conversion. This process can be processed quickly using the low-latency local network environment, thus avoiding the delay problem of uploading data to the central server. In addition, the edge node also has a cache function, which can temporarily store data when the network connection is unstable, ensuring that the data will not be lost or delayed in uploading to the central system.

[0055] S1.3, by establishing a distributed transmission protocol, the processed data is transmitted from each edge computing node to the central server or other required nodes for further processing;

[0056] This process usually uses a transmission protocol based on blockchain or P2P (peer-to-peer) technology to ensure the security and consistency of data transmission. During the transmission process, each edge node will select the best path based on the network conditions to transmit the data to the target node while ensuring the real-time and integrity of the data. In addition, in order to reduce the network bandwidth usage, the system will compress the data and use incremental synchronization technology to only transmit the data that has changed since the last synchronization.

[0057] S1.4. Dynamically manage and optimize edge computing nodes through the central management platform, monitor the operating status, resource usage and data transmission efficiency of each node in real time, and dynamically adjust the node configuration and data collection strategy according to the fluctuation of transaction volume and node load;

[0058] For example, during peak trading hours, the system can automatically increase the number of edge nodes in hot spots or adjust the processing priority of existing nodes to cope with high-concurrency data processing needs. At the same time, the platform will also pre-schedule nodes and optimize resources based on historical data and prediction models to ensure that the entire distributed data collection system is always in the best operating state.

[0059] Step 2: Preprocess the collected financial data using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers, thereby improving the accuracy and efficiency of subsequent data analysis;

[0060] The collected financial data is preprocessed using an automated data preprocessing algorithm based on machine learning. The specific steps to exclude irrelevant data and outliers are as follows:

[0061] S2.1. Clean the collected financial data and standardize its format;

[0062] Since financial data usually comes from different trading terminals and systems, the format and quality may vary greatly. In this step, the system will automatically identify and correct missing values, duplicate values, and erroneous data in the data. For example, missing data can be filled by interpolation, mean filling, or a prediction model based on historical data; duplicate data can be identified and deleted by hashing or other deduplication algorithms. At the same time, the data format needs to be standardized to convert data from different sources into a unified structure and format for subsequent processing and analysis.

[0063] S2.2. After data cleaning is completed, irrelevant data in the data set is filtered out. Irrelevant data refers to redundant data that has no direct relationship with the transaction monitoring and analysis objectives;

[0064] For example, account information, user logs or other non-transaction related data that are not relevant to the current analysis task. To achieve this goal, the system uses machine learning algorithms based on feature selection, such as LASSO regression, decision trees or feature screening methods based on mutual information. These algorithms can automatically evaluate the importance of each data feature and filter out data that contributes little or is useless to transaction analysis, thereby reducing the dimension of the data and improving the processing speed and efficiency of the system.

[0065] S2.3, use machine learning algorithms to detect and remove outliers in the data;

[0066] Outliers usually refer to values ​​that deviate significantly from the normal range in data distribution. These values ​​may be caused by incorrect transaction records, system failures, or other reasons. In this step, commonly used anomaly detection algorithms include Isolation Forest, Z-Score method, and density-based clustering algorithms (such as DBSCAN). These algorithms can effectively identify and remove outliers by analyzing data distribution, ensuring that subsequent data analysis is more accurate and reliable. At the same time, for the results of anomaly detection, the system can also combine artificial intelligence judgment to further improve the accuracy of outlier identification.

[0067] S2.4, normalize the remaining data and perform feature engineering to improve the data analysis performance;

[0068] Normalization refers to converting the numerical range of data into a unified scale, such as mapping all data to a range of 0 to 1 to eliminate the dimensional differences between different features. This step is usually performed using methods such as Min-Max scaling, Z-Score standardization, or Log transformation. After that, the system will apply feature engineering techniques to generate new features based on the original data to better describe the inherent patterns of the transaction data. For example, by constructing lagged variables of time series, generating polynomial features, or applying PCA (principal component analysis) for feature dimensionality reduction, the performance and analysis accuracy of subsequent machine learning models can be improved.

[0069] Step 3: Use a big data stream processing framework (such as Apache Flink or Apache Kafka) to perform real-time streaming processing on the preprocessed data;

[0070] In step three, by building a high-concurrency data processing channel, we ensure that the system can process data at the moment of input, avoiding delays caused by data accumulation.

[0071] Step 4: Based on real-time streaming processing, deep learning algorithms (such as LSTM or Transformer models) are used to analyze trading data in real time to identify potential market fluctuations and trading anomalies.

[0072] By learning and modeling historical data, the algorithm is able to discover patterns in real-time trading data and predict future trends.

[0073] The specific steps of using the big data stream processing framework to perform real-time streaming processing on preprocessed data are as follows:

[0074] S4.1. Build an efficient data stream input pipeline to introduce pre-processed financial data into the big data stream processing framework in real time;

[0075] This step usually relies on a message queue system (such as Apache Kafka or RabbitMQ) to push the financial data generated by each data source into the stream processing system in the form of messages. In order to ensure the real-time and consistency of data, the input pipeline adopts a distributed architecture that can handle concurrent inputs from multiple data sources at the same time, and package and partition the data in sequence to ensure that the incoming data can be reasonably distributed to subsequent processing nodes.

[0076] S4.2. Partition the input real-time data stream to achieve parallel computing. Partitioning is to divide the data stream into multiple independent sub-streams based on certain key fields of the data (such as transaction identifier, account number, etc.), thereby allowing multiple processors to process different sub-streams in parallel;

[0077] This step usually uses an in-memory computing engine (such as Apache Flink's KeyedStream or ApacheSpark's DStream), which can ensure efficient data processing under high concurrency conditions. Through partitioning, the system not only improves processing efficiency, but also implements targeted data analysis and processing strategies based on different partition contents.

[0078] S4.3. In order to realize the time series analysis and trend prediction of financial data, the system needs to perform time window operation on the streaming data. The window operation aggregates and analyzes the data according to time periods by defining sliding windows or tumbling windows.

[0079] For example, you can set a one-minute sliding window and perform an aggregation calculation on the data in the past minute every 30 seconds. This operation can effectively capture short-term fluctuations in the financial market and generate real-time statistical results. Windowing operations rely on built-in functions in the stream processing framework, which can handle data aggregation tasks in different time periods and push the results to subsequent processing units at the end of the window.

[0080] S4.4, after partitioning and windowing operations, real-time analysis and anomaly detection are performed on the processed data stream;

[0081] This step usually uses stream-based analysis algorithms, such as sliding mean, weighted moving average (WMA), or more complex machine learning models to analyze trends, fluctuations, and anomalies in trading data in real time. The anomaly detection module combines historical data and real-time input, using algorithms such as Z-Score or K-Means clustering to identify possible abnormal trading behaviors and trigger risk warnings. Through real-time analysis and detection, the system can make decisions at the moment the data arrives, thereby effectively preventing financial risks and providing decision support.

[0082] Step 5: Display the analysis results through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users and provide a multi-dimensional interactive interface;

[0083] Step 6: Generate transaction risk warnings in real time based on the analysis results, and automatically execute preset risk control measures through smart contract technology;

[0084] In step six, when abnormal trading behavior is detected, the relevant accounts are automatically frozen or the trading strategy is adjusted to reduce financial risks.

[0085] Step 7: Continuously monitor and optimize the operating status of the entire system. By introducing reinforcement learning algorithms, dynamically adjust data processing strategies and model parameters according to actual operating conditions to ensure that the system can maintain efficient real-time monitoring and analysis capabilities when facing a changing market environment.

[0086] The present invention realizes real-time collection of financial data from different transaction sources by establishing a distributed data collection system and utilizing edge computing technology, significantly reducing the delay in data transmission. Since the collection nodes are distributed near multiple transaction terminals, the system can quickly and directly acquire data and perform preliminary processing through edge computing technology. This real-time performance greatly improves the system's response speed to sudden market fluctuations and potential transaction risks, enabling financial institutions to monitor market dynamics and take rapid countermeasures at the moment of transaction, thereby effectively reducing the risk of abnormal market fluctuations or financial fraud not being discovered in a timely manner due to delays.

[0087] The present invention adopts an automated data preprocessing algorithm based on machine learning, and uses a big data stream processing framework and a deep learning algorithm to efficiently preprocess and analyze large-scale, diversified financial data in real time. This not only improves the accuracy of data processing, can effectively eliminate irrelevant data and outliers, but also enhances the overall performance of the system, enabling it to cope with the high concurrency processing requirements of large-scale data. In addition, the use of deep learning algorithms for real-time analysis of transaction data can more accurately identify potential market fluctuations and transaction anomalies. At the same time, the system is continuously optimized through reinforcement learning algorithms to ensure that in the face of a constantly changing market environment, the system can still maintain efficient real-time monitoring and analysis capabilities. This enables financial institutions to improve their adaptability to market changes while ensuring analysis accuracy, and effectively respond to the challenges brought about by a complex market environment.

[0088] The present invention provides Figure 2 The real-time transaction monitoring and analysis system of the financial big data platform shown includes a distributed data acquisition module, a data preprocessing module, a real-time stream processing module, a deep learning analysis module, a visualization display module, a smart contract risk control module, and a system monitoring and optimization module;

[0089] Distributed data collection module: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays;

[0090] Data preprocessing module: preprocesses the collected financial data, using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers;

[0091] Real-time stream processing module: Use the big data stream processing framework to perform real-time stream processing on pre-processed data;

[0092] Deep learning analysis module: Based on real-time streaming processing, deep learning algorithms are used to analyze transaction data in real time to identify potential market fluctuations and transaction anomalies;

[0093] Visualization display module: The analysis results are displayed through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users, and provides a multi-dimensional interactive interface;

[0094] Smart contract risk control module: Based on the analysis results, it generates transaction risk warnings in real time and automatically executes preset risk control measures through smart contract technology;

[0095] System monitoring and optimization module: Continuously monitor and optimize the operation status of the entire system. By introducing reinforcement learning algorithms, the data processing strategy and model parameters are dynamically adjusted according to the actual operation status to ensure that the system maintains efficient real-time monitoring and analysis capabilities in the face of a changing market environment;

[0096] The real-time transaction monitoring and analysis method of the financial big data platform provided in the embodiment of the present invention is realized by the real-time transaction monitoring and analysis system of the above-mentioned financial big data platform. The specific methods and processes of the real-time transaction monitoring and analysis system of the financial big data platform are detailed in the embodiment of the real-time transaction monitoring and analysis method of the above-mentioned financial big data platform, which will not be repeated here.

[0097] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A real-time transaction monitoring and analysis method for a financial big data platform, characterized in that: The following steps are involved: Step 1: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays. Step 2: Preprocess the collected financial data using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers; Step 3: Use the big data stream processing framework to perform real-time streaming processing on the preprocessed data; Step 4: Based on real-time streaming processing, deep learning algorithms are used to analyze transaction data in real time to identify potential market fluctuations and transaction anomalies; Step 5: Display the analysis results through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users and provide a multi-dimensional interactive interface; Step 6: Generate transaction risk warnings in real time based on the analysis results, and automatically execute preset risk control measures through smart contract technology; Step 7: Continuously monitor and optimize the operating status of the entire system. By introducing reinforcement learning algorithms, dynamically adjust data processing strategies and model parameters according to actual operating conditions to ensure that the system maintains efficient real-time monitoring and analysis capabilities in the face of a changing market environment.

2. The real-time transaction monitoring and analysis method of the financial big data platform according to claim 1 is characterized in that: The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals. The specific steps are as follows: S1.

1. Deploy edge computing nodes near multiple transaction terminals to form a distributed collection network; S1.

2. After deploying edge computing nodes, the system will implement local data processing and caching functions on each node. The edge computing nodes directly connect to the transaction terminals to obtain transaction data in real time and perform preliminary processing locally. S1.3, by establishing a distributed transmission protocol, the processed data is transmitted from each edge computing node to the central server or other required nodes for further processing; S1.

4. Dynamically manage and optimize edge computing nodes through the central management platform, monitor the operating status, resource usage and data transmission efficiency of each node in real time, and dynamically adjust the node configuration and data collection strategy according to the fluctuation of transaction volume and node load.

3. The real-time transaction monitoring and analysis method of the financial big data platform according to claim 1 is characterized in that: The collected financial data is preprocessed using an automated data preprocessing algorithm based on machine learning. The specific steps to exclude irrelevant data and outliers are as follows: S2.

1. Clean the collected financial data and standardize its format; S2.

2. After data cleaning is completed, irrelevant data in the data set is filtered out. Irrelevant data refers to redundant data that has no direct relationship with the transaction monitoring and analysis objectives; S2.3, use machine learning algorithms to detect and remove outliers in the data; S2.

4. The remaining data will be normalized and feature engineering will be performed to improve the analysis performance of the data.

4. The real-time transaction monitoring and analysis method of the financial big data platform according to claim 1 is characterized in that: In step three, by building a high-concurrency data processing channel, we ensure that the system can process data at the moment of input, avoiding delays caused by data accumulation.

5. The real-time transaction monitoring and analysis method of the financial big data platform according to claim 1, characterized in that: The specific steps of using the big data stream processing framework to perform real-time streaming processing on preprocessed data are as follows: S4.

1. Build an efficient data stream input pipeline to introduce pre-processed financial data into the big data stream processing framework in real time; S4.

2. Partition the input real-time data stream to achieve parallel computing. Partitioning is to divide the data stream into multiple independent sub-streams according to certain key fields of the data, thereby allowing multiple processors to process different sub-streams in parallel. S4.

3. In order to realize the time series analysis and trend prediction of financial data, the system needs to perform time window operation on the streaming data. The window operation aggregates and analyzes the data according to time periods by defining sliding windows or tumbling windows. S4.

4. After partitioning and windowing operations, perform real-time analysis and anomaly detection on the processed data stream.

6. The real-time transaction monitoring and analysis method of the financial big data platform according to claim 1, characterized in that: In step six, when abnormal trading behavior is detected, the relevant accounts are automatically frozen or the trading strategy is adjusted to reduce financial risks.

7. A real-time transaction monitoring and analysis system for a financial big data platform, used to implement the real-time transaction monitoring and analysis method for a financial big data platform as described in any one of claims 1 to 6, characterized in that: It includes distributed data acquisition module, data preprocessing module, real-time stream processing module, deep learning analysis module, visualization display module, smart contract risk control module and system monitoring and optimization module; Distributed data collection module: Establish a distributed data collection system to collect financial data from different transaction sources in real time. The distributed data collection system uses edge computing technology to distribute collection nodes near multiple transaction terminals to reduce data transmission delays; Data preprocessing module: preprocesses the collected financial data, using an automated data preprocessing algorithm based on machine learning to exclude irrelevant data and outliers; Real-time stream processing module: Use the big data stream processing framework to perform real-time stream processing on pre-processed data; Deep learning analysis module: Based on real-time streaming processing, deep learning algorithms are used to analyze transaction data in real time to identify potential market fluctuations and transaction anomalies; Visualization display module: The analysis results are displayed through the visualization module. The visualization module uses a dynamic dashboard to intuitively display transaction data, risk warnings and market trends to users, and provides a multi-dimensional interactive interface; Smart contract risk control module: Based on the analysis results, it generates transaction risk warnings in real time and automatically executes preset risk control measures through smart contract technology; System monitoring and optimization module: Continuously monitor and optimize the operating status of the entire system. By introducing reinforcement learning algorithms, the data processing strategy and model parameters are dynamically adjusted according to the actual operating conditions to ensure that the system maintains efficient real-time monitoring and analysis capabilities in the face of a changing market environment.

Citation Information

Cited By

  • Computer network security transaction optimization system

    CN119624457A

  • Business auditing fusion auditing system based on big data

    CN120297977A

  • A Big Data-Based Business Audit Integration System

    CN120297977B

  • Financial risk monitoring system and monitoring method based on big data

    CN120338947A

  • Financial data anomaly detection method and device, equipment and medium

    CN120705774A