Transaction data monitoring and early warning method based on bypass flow and real-time counting warehouse

Abstract

The invention relates to the technical field of transaction data monitoring and early warning, and discloses a transaction data monitoring and early warning method based on bypass flow and real-time data warehouse, which comprises the following steps: copying a flow data packet of a port in network equipment to a bypass port based on a bypass mirroring technology, decoding the flow data packet to obtain a flow detailed list corresponding to the flow data packet, and sending the flow detailed list to the bypass port; synchronizing the change transaction data in the kafka message queue and the flow detailed list, and forming a data set by the synchronized data; executing a dynamic priority calculation strategy for any data in the data set, calculating the priority of the data in real time, and dynamically adjusting the storage position of the data according to the priority; and executing the abnormity monitoring alarm strategy, obtaining an abnormal flow detailed list, and giving an alarm for the abnormal flow detailed list. The system load can be dynamically adapted while the high-risk data is preferentially captured, and the monitoring efficiency and the resource utilization rate are improved. And the monitoring sensitivity and the early warning accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of transaction data monitoring and early warning, and in particular to a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse. Background Art

[0002] In the financial and e-commerce industries, transaction security is of vital importance. With the continuous development of digitalization and Internet technology, the scale and frequency of transactions have increased significantly, which has brought new challenges and complexities to transaction security. The key to transaction security lies in protecting the confidentiality, integrity and availability of transaction data. Confidentiality ensures that only authorized personnel can access transaction data and prevent the leakage of sensitive information. Integrity ensures that transaction data is not tampered with during transmission and storage, while availability ensures that related systems and data are available when needed.

[0003] Existing transaction data monitoring systems often have the following problems: Data delay: Traditional monitoring systems rely on database logs or post-audits, which leads to delays in data updates and cannot meet the needs of real-time monitoring. Incomplete monitoring: Traditional monitoring can only monitor application service logs and database data, and it is difficult to analyze request data in switch network traffic packets. Inefficient anomaly detection: Traditional monitoring analysis is inefficient when processing large amounts of data, and it is difficult to quickly identify abnormal behavior.

[0004] The present invention proposes a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse to solve the problems existing in the prior art. Summary of the invention

[0005] The present invention provides a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse, which promotes solving the problems mentioned in the above background technology.

[0006] In a first aspect, the present application provides a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse, which adopts the following technical scheme: a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse, comprising:

[0007] S1, based on the bypass mirroring technology, copying the traffic data packets of the port in the network device to the bypass port, wherein the bypass port is used to store the traffic data packets obtained using the bypass mirroring technology;

[0008] By applying bypass mirroring technology, you can copy and capture data packets of port traffic in network devices without affecting normal network traffic. This method provides a non-invasive means of monitoring transaction data without negatively affecting network performance. By saving traffic packets through the bypass port, the system can continuously and in real time obtain information about network activities, which is very important for identifying potential security threats and abnormal trading behavior. Bypass mirroring technology can ensure that the captured data packets are accurate and provide reliable basic data for subsequent analysis and processing. This technology is particularly suitable for high-frequency trading environments because it supports real-time monitoring and processing of large amounts of data.

[0009] S2. Decode the traffic data packet based on the parallel processing mechanism of the real-time data analysis engine, obtain the traffic details corresponding to the traffic data packet, and push the traffic details to the Kafka message queue;

[0010] By using a high-performance real-time data analysis engine, traffic packets can be efficiently processed and decoded. The parallel processing mechanism improves the speed and efficiency of data processing, allowing the system to respond to changes in network traffic in a timely manner. This not only helps to generate detailed traffic records, but also quickly identify and mark abnormal traffic patterns. Pushing traffic details to the Kafka message queue ensures that data can be quickly consumed and further processed. This approach enhances the overall responsiveness of the system, reduces data latency, and provides fast and effective support for subsequent risk assessment and anomaly detection.

[0011] S3, obtaining transaction data in the transaction system database, executing the risk classification strategy, and classifying the transaction data into three categories, namely, low-risk data, medium-risk data, and high-risk data;

[0012] Execute the dynamic window adjustment strategy, adjust the time window of the CDC technology, capture the changed transaction data in the transaction system database, and record it as the changed transaction data;

[0013] Send the change transaction data to the kafka message queue;

[0014] S4. Use the Flink real-time computing engine to synchronize the changed transaction data and traffic details in the Kafka message queue, and form a data set with the synchronized data.

[0015] By using the Flink real-time computing engine, the system can establish a comprehensive and consistent data view by synchronizing the transaction data and the traffic details into a data set. This real-time synchronization ensures the accuracy and timeliness of the data, making subsequent analysis and query more reliable. Saving the data set to the StarRocks online query platform database provides an efficient query and analysis environment that can quickly respond to complex query requirements. The dynamic priority calculation strategy adjusts the storage location according to the importance of the data, optimizes the utilization efficiency of storage resources, and improves the overall performance of the system.

[0016] Save the data set to the StarRocks online query platform database;

[0017] For any data in the data set, a dynamic priority calculation strategy is executed to calculate the data priority in real time and dynamically adjust the data storage location according to the priority;

[0018] S5. Execute the abnormal monitoring and alarm strategy for the data set stored in the StarRocks online query platform database, obtain the abnormal traffic details, and issue an alarm for the abnormal traffic details;

[0019] Preferably, the transaction data in the transaction system database is obtained, and a risk classification strategy is executed to classify the transaction data into three categories, including:

[0020] For each transaction data in the transaction system database:

[0021] Extract the risk feature value in the transaction data and record it as , , where n is the number of risk characteristic values;

[0022] Calculating risk scores for transaction data ,in, is the weight of the risk characteristic value;

[0023] Setting a high risk threshold and medium risk threshold ;

[0024] like , then the transaction data is high-risk data;

[0025] like , then the transaction data is medium-risk data;

[0026] like , then the transaction data is low-risk data.

[0027] By extracting transaction data from the transaction system database, the system can perform risk analysis on each transaction. The risk classification strategy divides transaction data into three risk levels: low, medium, and high, allowing the system to manage different types of transactions more targeted. This process helps identify potential high-risk transactions and take appropriate preventive measures to reduce financial losses and reputation risks. The dynamic window adjustment strategy is combined with change data capture to ensure that the system can monitor changes in transaction data in the database in real time. This approach improves the system's sensitivity to risks and enhances the initiative of risk control.

[0028] Preferably, the executing a dynamic window adjustment strategy, adjusting the time window of the CDC technology, and capturing data changes in the transaction system database includes:

[0029] Get the current time window ;

[0030] Get the total number of transaction data changes in the current time window ;

[0031] Get the total number of times The total number of medium and high risk data changes ;

[0032] Get the transaction volume in the current time window ;

[0033] Get the time window as The historical maximum transaction volume ;

[0034] Calculate the adjusted time window ,in, is the risk adjustment factor, is the transaction volume adjustment coefficient.

[0035] By taking the number of high-risk changes as a sensitive indicator of the potential risk of the current system, the adjustment of the dynamic time window is based on the fact that when the number of high-risk transaction changes increases, the potential risk faced by the system is higher. At this time, the time window needs to be shortened to capture changes faster. When the number of high-risk transaction changes is low, the time window can be appropriately delayed to reduce system overhead and save costs. At the same time, reference to the transaction volume is also necessary. Since the higher the transaction volume, the more times the system may change, the greater the risk. Therefore, the higher the transaction volume, the more time windows need to be shortened adaptively. Conversely, the time windows can be appropriately extended.

[0036] Preferably, for any data in the data set, executing a dynamic priority calculation strategy, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority, includes:

[0037] When data is stored in the StarRocks online query platform database:

[0038] Get the time when the data is saved to the StarRocks online query platform database ;

[0039] Get the current time ;

[0040] Get the total number of times p that the data is queried in the time interval from the save time to the current time;

[0041] Get the time when the data was last queried ;

[0042] Set the priority of the data at the current moment ,in, is the adjustment amplitude of the control priority;

[0043] Setting Priority Thresholds ;

[0044] Compare the priority of the data to the priority threshold:

[0045] like , the data will be saved in the StarRocks online query platform database;

[0046] like , then store the data in distributed data.

[0047] Preferably, for any data in the data set, executing a dynamic priority calculation strategy, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority, includes:

[0048] Dynamically adjust priority thresholds , specifically:

[0049] Get the current time The priority of all data in the data set, calculate the mean of all priorities ;

[0050] Calculate the current time The standard deviation of the priorities of all data in the data set ;

[0051] Calculate the current time Priority threshold ,in, is the standard quantile value under normal distribution, which is determined by the proportion of data stored in the StarRocks online query platform database.

[0052] By adjusting priorities based on normal distribution, the system can achieve more reasonable standardized processing. This approach allows the system to more accurately identify and prioritize data that really needs attention by dynamically adapting to changes in data distribution. Through this adjustment, resource allocation efficiency is improved because low-priority data can be stored in lower-cost media, while high-priority data enjoys more efficient storage resources. By enhancing anomaly detection capabilities, the system can sensitively identify abnormal points that deviate from the center of the normal distribution, thereby promptly discovering and responding to potential abnormal trading behaviors. By providing a scientific basis for setting data priorities, the system's decision-making process becomes more reasonable and reliable. This approach not only optimizes resource utilization, but also lays the foundation for the efficient operation of the system in complex trading environments. By combining these advantages, priority adjustment based on normal distribution can greatly improve the overall performance and security of the system.

[0053] Preferably, the method of executing an abnormal monitoring and alarming strategy for a data set stored in a StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list includes:

[0054] Get any changed transaction data in the data set, traverse all traffic details in the data set, and match the traffic details of all changed transaction data;

[0055] Determine whether there is a flow detail bill without change transaction data;

[0056] If all traffic details have change transaction data, the traffic details are normal;

[0057] If there is a traffic detail bill without changed transaction data, the traffic detail bill is abnormal and an alarm is issued for the traffic detail bill.

[0058] Preferably, the method of executing an abnormal monitoring and alarming strategy for a data set stored in a StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list includes:

[0059] Exception capture is performed for each traffic detail in the data set, specifically:

[0060] The current moment Each characteristic value in the traffic list is counted as , ;

[0061] Setting the abnormality judgment interval ;

[0062] Get the current time Before Each feature value at time ;

[0063] Calculate each eigenvalue in Interval Trend ;

[0064] Calculate the abnormal score of the traffic details ,in, is the static weight of each feature, is the dynamic trend characteristic of each feature, g is the number of features in the traffic details list;

[0065] Set abnormal warning threshold ;

[0066] The abnormal score and abnormal warning thresholds Make comparisons;

[0067] like , then the traffic details are normal;

[0068] like , then the traffic details are abnormal and an early warning is issued for the traffic details.

[0069] By monitoring the data sets stored in StarRocks for anomalies, the system can quickly identify abnormal traffic details and issue alerts in a timely manner. The abnormal monitoring and alerting strategy allows the system to intervene in the early stages of the problem and prevent potential threats from evolving into actual damage. This real-time monitoring and rapid response mechanism can significantly improve the security of the system and reduce losses caused by abnormal activities. Timely alert information provides the security team with the opportunity to conduct in-depth analysis and take necessary measures, thereby improving the overall security of the trading environment.

[0070] The present invention has the following beneficial effects:

[0071] 1. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse can dynamically optimize the efficiency of monitoring and capture by adjusting the strategy based on high-risk data changes and transaction volume in the time window. First, the number of changes in high-risk data directly reflects the potential risks in the transaction system. By capturing high-risk data first, the pertinence of data analysis can be improved and resources can be avoided from being occupied by low-risk data. Secondly, the adjustment strategy based on transaction volume can ensure that the system dynamically expands the time window under high load and adapts to the scenario of frequent transaction fluctuations, thereby reducing the risk of losing key data. In addition, through the combined calculation of risk factors and active factors, the adjustment of the time window can dynamically adapt to different business scenarios, such as capturing data more frequently during peak trading hours, and extending the window during low-activity periods to reduce the consumption of system resources. In summary, the advantage of adjusting the time window based on high-risk data changes and transaction volume is that it can dynamically adapt to the system load while ensuring the priority capture of high-risk data, thereby improving monitoring efficiency and resource utilization.

[0072] 2. This transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse can achieve refined management of stored data by dynamically adjusting the priority threshold through normal distribution. The characteristic of normal distribution is that it reflects the central trend of data in a concentrated manner. By setting standards, the ratio of hot data to cold data can be flexibly controlled. For example, by adjusting the quantile value, the priority threshold can be dynamically adjusted to optimize the data storage strategy in the StarRocks database. Normal distribution can also reflect the distribution characteristics of data priority, helping the system to better identify and classify high-priority data, thereby ensuring that important data is preferentially stored in high-performance databases. This method is highly adaptable and robust, and can dynamically adjust the storage strategy by recalculating the mean and standard deviation even when the data distribution changes. In summary, the benefit of using normal distribution when dynamically adjusting the priority threshold is that it uses statistical characteristics to achieve dynamic optimization of data storage and effectively balances storage performance and cost.

[0073] 3. The advantage of introducing trend analysis when analyzing traffic details in this transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse is that it can identify the difference between current data and historical change patterns. Trend analysis can reveal the direction and speed of change of characteristic values ​​in traffic details, such as the growth rate of function number access frequency or the convergence trend of IP distribution entropy. This dynamic trend characteristic can be used as a key basis for anomaly identification and is more adaptable to real-time monitoring scenarios than static analysis. By combining trend features with static features, the system can more accurately locate potential risk behaviors, such as identifying sudden abnormal operations during peak trading periods. In addition, trend analysis is also predictive, that is, it predicts future abnormal events through trend changes in a short period of time, thereby providing more forward-looking early warning capabilities. In summary, the advantage of using trends when analyzing traffic details is that it can capture abnormal behaviors through dynamic changes, improving the monitoring sensitivity and early warning accuracy of the system.

[0074] 4. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse has the advantage of obtaining traffic data packets through bypass mirroring technology in that it can copy the traffic data packets of the port in the network device to the bypass port in real time without interrupting network communication. First, this technology realizes the non-intrusive collection of traffic data, that is, copying the traffic packet to the bypass port without changing the original configuration of the network device, thereby ensuring the high availability of the system. Secondly, bypass mirroring has high real-time performance, so that traffic data can be immediately transmitted to the subsequent decoding and analysis process, meeting the needs of real-time monitoring and anomaly detection. In addition, bypass mirroring supports customized traffic collection for specific ports, protocols or applications, and this flexibility makes monitoring more accurate. The centralized processing of traffic data packets through the bypass port can also reduce the performance impact of data collection on the main system, and improve the stability and scalability of the system. In summary, the advantage of obtaining traffic data packets through bypass mirroring technology is that it can collect traffic data in real time in a non-intrusive manner, ensuring the operation of the system while providing efficient and reliable data input for subsequent decoding and analysis.

[0075] 5. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse can significantly improve the efficiency and accuracy of data decoding by decoding traffic data packets through the real-time data analysis engine. First, this method can quickly process large-scale data, and the parallel computing power can ensure the real-time decoding of thousands or even millions of data packets per second, providing sufficient data support for downstream analysis. Secondly, the real-time data analysis engine can generate traffic details in a unified format, and structure the complex fields in the original data packets into standardized entries, which is convenient for subsequent query and processing. In addition, preprocessing logic such as data cleaning, field verification and redundant field removal can be embedded in the decoding process to improve the quality and consistency of the data. The decoded traffic details are directly pushed to efficient message queues such as kafka message queues, providing input for subsequent real-time computing engines, further optimizing the efficiency of data transmission. In summary, the benefit of decoding traffic data packets through the real-time data analysis engine is that it can efficiently and accurately convert raw traffic data into structured details, while optimizing the transmission and processing links of data streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Embodiment 1, refer to Figure 1 , a transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse, including:

[0079] S1, based on the bypass mirroring technology, copying the traffic data packets of the port in the network device to the bypass port, wherein the bypass port is used to store the traffic data packets obtained using the bypass mirroring technology;

[0080] The benefit of obtaining traffic data packets through bypass mirroring technology is that it can realize non-intrusive collection of traffic data, thereby ensuring the stability of the network system. First, bypass mirroring technology copies the traffic data packets of the port in the network device to the bypass port, which will not cause any interference to the transmission of the original data stream, effectively ensuring business continuity. Secondly, this technology can capture network traffic data in real time, providing an accurate input data source for subsequent decoding and analysis. At the same time, the flexibility of bypass mirroring enables it to set the monitoring range according to needs, such as targeted collection of data streams for specific ports, protocols or applications, which greatly improves the pertinence and efficiency of monitoring. In addition, the data processing characteristics of the bypass port can also significantly reduce the occupation of the original network device performance, thereby improving the operating efficiency and reliability of the entire system. In summary, bypass mirroring technology can not only collect traffic data in a low-cost and efficient manner, but also provide a strong guarantee for the stability and performance of the system.

[0081] S2. Decode the traffic data packet based on the parallel processing mechanism of the real-time data analysis engine, obtain the traffic details corresponding to the traffic data packet, and push the traffic details to the Kafka message queue;

[0082] In this embodiment, the traffic data is obtained by bypassing the traffic mode, which is actually a data packet, which contains all the data accessed by each network request. The data packet is decoded by the parallel processing mechanism of the real-time data analysis engine to generate a traffic detail list. The content of the traffic detail list includes function number, channel number, fund account number, stock code, request body, return body, session request time, response time, client IP and port, server IP and port, etc. The data decoding engine adopts a distributed architecture, which can process a large number of data packets at the same time, improving the efficiency of data decoding. The decoded traffic detail list is pushed through the kafka message queue to ensure the real-time nature of the data.

[0083] The benefit of decoding traffic packets through the real-time data analysis engine is that it significantly improves the efficiency and accuracy of data processing. The engine uses a parallel processing mechanism to quickly decode a large number of traffic packets to meet the decoding needs in a high-concurrency environment. This decoding method converts complex traffic data into standardized traffic details, providing a unified data structure for downstream data analysis, storage, and query. In addition, the data cleaning and verification functions in the decoding process can effectively filter abnormal data and ensure that the generated traffic details have a high degree of accuracy and consistency. At the same time, the decoded traffic details are directly pushed to message queues such as the kafka message queue, which greatly reduces the delay in data transmission and ensures the real-time nature of the data. In summary, the benefit of decoding traffic packets through the real-time data analysis engine is that it can not only efficiently decode large-scale data, but also provide high-quality data support for real-time monitoring and anomaly detection.

[0084] S3, obtaining transaction data in the transaction system database, executing the risk classification strategy, and classifying the transaction data into three categories, namely, low-risk data, medium-risk data, and high-risk data;

[0085] Execute the dynamic window adjustment strategy, adjust the time window of the CDC technology, capture the changed transaction data in the transaction system database, and record it as the changed transaction data;

[0086] In this embodiment, CDC technology is used to obtain data changes in the transaction system database, such as customer entrustment data, and the captured transaction system data is sent to the Kafka message queue to send the changed transaction data to the Kafka message queue;

[0087] S4. Use the Flink real-time computing engine to synchronize the changed transaction data and traffic details in the Kafka message queue, and form a data set with the synchronized data.

[0088] Save the data set to the StarRocks online query platform database;

[0089] For any data in the data set, a dynamic priority calculation strategy is executed to calculate the data priority in real time and dynamically adjust the data storage location according to the priority;

[0090] S5. Execute the abnormal monitoring and alarm strategy for the data set stored in the StarRocks online query platform database, obtain the abnormal traffic details, and issue an alarm for the abnormal traffic details;

[0091] In this embodiment, Kafka is an open source distributed stream processing platform, which is mainly used for high-throughput, low-latency real-time data stream processing. It provides efficient message queue functions by publishing messages to topics and subscribing by consumers. Kafka has a distributed architecture, supports horizontal expansion, high availability and fault tolerance, and ensures data reliability through message persistence and partition replication. Kafka is suitable for big data environments, can process massive messages, and supports low-latency transmission and real-time data stream processing. It is widely used in data integration, log aggregation, real-time analysis and other fields.

[0092] The transaction data in the transaction system database is obtained, and the risk classification strategy is executed to classify the transaction data into three categories, including:

[0093] For each transaction data in the transaction system database:

[0094] Extract the risk feature value in the transaction data and record it as , , where n is the number of risk characteristic values;

[0095] In this embodiment, common transaction data risk characteristic values ​​include: transaction amount, transaction frequency, transaction location, transaction time, transaction type, account change, device information, IP address, geographic location change, and number of transaction failures.

[0096] Calculating risk scores for transaction data ,in, is the weight of the risk characteristic value;

[0097] Setting a high risk threshold and medium risk threshold ;

[0098] like , then the transaction data is high-risk data;

[0099] like , then the transaction data is medium-risk data;

[0100] like , then the transaction data is low-risk data.

[0101] The execution of the dynamic window adjustment strategy, adjusting the time window of the CDC technology, and capturing data changes in the transaction system database include:

[0102] Get the current time window ;

[0103] Get the total number of transaction data changes in the current time window ;

[0104] Get the total number of times The total number of medium and high risk data changes ;

[0105] Get the transaction volume in the current time window ;

[0106] Get the time window as The historical maximum transaction volume ;

[0107] Calculate the adjusted time window ,in, is the risk adjustment factor, is the transaction volume adjustment coefficient.

[0108] By adjusting the strategy based on high-risk data changes and trading volume in the time window, the accuracy of monitoring and resource allocation can be dynamically optimized. First, the frequency of changes in high-risk data directly reflects the potential risk level of the system. By dynamically adjusting the time window to capture more high-risk data, the timely processing of critical data can be prioritized. In addition, adjusting the time window in combination with trading volume can adapt to different trading scenarios. For example, in high-volume periods, shortening the time window can increase the capture frequency and avoid missing critical data; while in low-volume periods, extending the time window can reduce system resource consumption and improve efficiency. The combination of the two can not only ensure the coverage of high-risk data, but also effectively allocate system resources. In summary, the benefit of adjusting the time window based on high-risk data changes and trading volume is that it can optimize resource utilization while ensuring monitoring accuracy, providing flexibility and efficiency for system operation.

[0109] The method of executing a dynamic priority calculation strategy for any data in the data set, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority includes:

[0110] When data is stored in the StarRocks online query platform database:

[0111] Get the time when the data is saved to the StarRocks online query platform database ;

[0112] Get the current time ;

[0113] Get the total number of times p that the data is queried in the time interval from the save time to the current time;

[0114] Get the time when the data was last queried ;

[0115] Set the priority of the data at the current moment ,in, is the adjustment amplitude of the control priority;

[0116] Setting Priority Thresholds ;

[0117] Compare the priority of the data to the priority threshold:

[0118] like , the data will be saved in the StarRocks online query platform database;

[0119] like , then store the data in distributed data.

[0120] The method of executing a dynamic priority calculation strategy for any data in the data set, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority includes:

[0121] Dynamically adjust priority thresholds , specifically:

[0122] Get the current time The priority of all data in the data set, calculate the mean of all priorities ;

[0123] Calculate the current time The standard deviation of the priorities of all data in the data set ;

[0124] Calculate the current time Priority threshold ,in, is the standard quantile value under normal distribution, which is determined by the proportion of data stored in the StarRocks online query platform database.

[0125] The benefit of dynamically adjusting the priority threshold through normal distribution is that it can accurately optimize the storage strategy according to the distribution characteristics of real-time data. Normal distribution reflects the central tendency and dispersion of data through mean and standard deviation. By setting the priority threshold using the standard quantile value zpz_pzp, the ratio of high-priority data to low-priority data can be dynamically controlled. For example, when business needs change, you only need to adjust the quantile value to adapt to the new storage strategy without manually reclassifying the data. This method can still maintain high flexibility and robustness when the data distribution changes. Through this statistical method, high-priority data can be preferentially stored in a database with higher performance, while low-priority data is migrated to a storage system with more cost advantages, thereby achieving a balance between performance and cost. In summary, the benefit of using normal distribution when dynamically adjusting the priority threshold is that it realizes intelligent management of storage based on statistical characteristics, which not only improves efficiency but also reduces costs.

[0126] In this embodiment, see the table below, which shows the proportion of database data stored on the StarRocks online query platform and its corresponding quantile value. The table contains two columns of data: "The proportion of data in the database" represents the percentage of data in the database. For example, 50% represents the data in the middle of the database (i.e., the median), and the "quantile value" represents the standard normal distribution (Z score) corresponding to the data proportion or the statistical quantile under a certain distribution. It can be used for: Database performance analysis: Analyze the distribution of data of different percentages to optimize queries. Statistical modeling: If these values ​​come from a standard normal distribution, they can be used to calculate confidence intervals or probability predictions. Data index optimization: Optimize the index strategy of StarRocks queries according to different data distributions to improve query efficiency.

[0127]

[0128] The method includes executing an abnormal monitoring and alarming strategy for a data set stored in the StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list, including:

[0129] Get any changed transaction data in the data set, traverse all traffic details in the data set, and match the traffic details of all changed transaction data;

[0130] Determine whether there is a flow detail bill without change transaction data;

[0131] If all traffic details have change transaction data, the traffic details are normal;

[0132] If there is a traffic detail bill without changed transaction data, the traffic detail bill is abnormal and an alarm is issued for the traffic detail bill.

[0133] The method includes executing an abnormal monitoring and alarming strategy for a data set stored in the StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list, including:

[0134] Exception capture is performed for each traffic detail in the data set, specifically:

[0135] The current moment Each characteristic value in the traffic list is counted as , ;

[0136] Setting the abnormality judgment interval ;

[0137] Get the current time Before Each feature value at time ;

[0138] Calculate each eigenvalue in Interval Trend ;

[0139] Calculate the abnormal score of the traffic details ,in, is the static weight of each feature, is the dynamic trend characteristic of each feature, g is the number of features in the traffic details list;

[0140] Set abnormal warning threshold ;

[0141] The abnormal score and abnormal warning thresholds Make comparisons;

[0142] like , then the traffic details are normal;

[0143] like , then the traffic details are abnormal and an early warning is issued for the traffic details.

[0144] In this embodiment, StarRocks is queried in real time on the big data BI self-service analysis platform to display the comparison results of the customer entrustment flow detail data and the transaction system entrustment data table. If a certain entrustment flow detail data does not exist in the transaction system entrustment data table, it means that the customer entrustment data is lost, and the operation and maintenance personnel are promptly notified to conduct an investigation. Through the preset query conditions and warning thresholds, real-time monitoring and warning are provided to each operation and maintenance personnel. For example, if the access TPS of a certain function number exceeds a certain threshold, or the request return status exceeds a certain proportion or number, the operation and maintenance personnel can also be notified to check the service status in a timely manner.

[0145] In this embodiment, the big data BI self-service analysis platform is a tool that integrates data access, analysis and visualization, and is designed to help users perform autonomous data query and analysis without technical support. Through a simple graphical interface, users can integrate data from different sources, perform query, filter, aggregate and group operations, and visualize complex data through charts and dashboards. The platform also supports customized reports, real-time analysis and collaborative sharing to improve decision-making efficiency. Some platforms introduce machine learning algorithms to provide intelligent analysis and prediction, helping users discover potential patterns in data and further optimize the decision-making process.

[0146] In this embodiment, refer to the following tips:

[0147]

[0148] When the Kafka traffic details data contains counter entrustment data that does not exist, an autonomous analysis alarm is issued, and the operation and maintenance personnel are prompted to log in and check in time; when the Kafka traffic details data contains counter entrustment data that does not exist, the alarm is lifted and the platform recovers autonomously;

[0149] The benefit of introducing trend analysis when analyzing traffic details is that it can dynamically capture the changing patterns of data, which is more forward-looking and adaptable than static analysis. Trend analysis can reveal the direction of change of characteristic values ​​in traffic details, such as the increasing trend of function number access frequency or the fluctuation amplitude of IP distribution entropy, which are often precursors to abnormal behavior. Through trend analysis, the system can discover short-term change patterns in real time and issue timely warnings for potential risks. In addition, trend analysis can also be combined with static features to form a more comprehensive anomaly detection mechanism, thereby significantly improving the sensitivity and accuracy of anomaly detection. In summary, the benefit of using trend analysis when analyzing traffic details is that it captures abnormal behavior through dynamic change characteristics, improving the sensitivity of system monitoring and the timeliness of warnings.

[0150] By monitoring the data sets stored in StarRocks for anomalies, the system can quickly identify abnormal traffic details and issue alerts in a timely manner. The abnormal monitoring and alerting strategy allows the system to intervene in the early stages of the problem and prevent potential threats from evolving into actual damage. This real-time monitoring and rapid response mechanism can significantly improve the security of the system and reduce losses caused by abnormal activities. Timely alert information provides the security team with the opportunity to conduct in-depth analysis and take necessary measures, thereby improving the overall security of the trading environment.

[0151] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0152] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse, characterized in that: include: S1, based on the bypass mirroring technology, copying the traffic data packets of the port in the network device to the bypass port, wherein the bypass port is used to store the traffic data packets obtained using the bypass mirroring technology; S2. Decode the traffic data packet based on the parallel processing mechanism of the real-time data analysis engine, obtain the traffic details corresponding to the traffic data packet, and push the traffic details to the Kafka message queue; S3, obtaining transaction data in the transaction system database, executing the risk classification strategy, and classifying the transaction data into three categories, namely, low-risk data, medium-risk data, and high-risk data; Execute the dynamic window adjustment strategy, adjust the time window of the CDC technology, capture the changed transaction data in the transaction system database, and record it as the changed transaction data; Send the change transaction data to the kafka message queue; S4. Use the Flink real-time computing engine to synchronize the changed transaction data and traffic details in the Kafka message queue, and form a data set with the synchronized data. Save the data set to the StarRocks online query platform database; For any data in the data set, a dynamic priority calculation strategy is executed to calculate the data priority in real time and dynamically adjust the data storage location according to the priority; S5. Execute the abnormal monitoring alarm strategy for the data set stored in the StarRocks online query platform database, obtain the abnormal traffic details, and issue an alarm for the abnormal traffic details.

2. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse according to claim 1 is characterized in that: The transaction data in the transaction system database is obtained, and the risk classification strategy is executed to classify the transaction data into three categories, including: For each transaction data in the transaction system database: Extract the risk feature value in the transaction data and record it as , , where n is the number of risk characteristic values; Calculating risk scores for transaction data ,in, is the weight of the risk characteristic value; Setting a high risk threshold and medium risk threshold ; like , then the transaction data is high-risk data; like , then the transaction data is medium-risk data; like , then the transaction data is low-risk data.

3. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse according to claim 2 is characterized in that: The execution of the dynamic window adjustment strategy, adjusting the time window of the CDC technology, and capturing the changed transaction data in the transaction system database includes: Get the current time window ; Get the total number of transaction data changes in the current time window ; Get the total number of times The total number of medium and high risk data changes ; Get the transaction volume in the current time window ; Get the time window as The historical maximum transaction volume ; Calculate the adjusted time window ,in, is the risk adjustment factor, is the transaction volume adjustment coefficient.

4. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse according to claim 1 is characterized in that: The method of executing a dynamic priority calculation strategy for any data in the data set, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority includes: When data is stored in the StarRocks online query platform database: Get the time when the data is saved to the StarRocks online query platform database ; Get the current time ; Get the total number of times p that the data is queried in the time interval from the save time to the current time; Get the time when the data was last queried ; Set the priority of the data at the current moment ,in, is the adjustment amplitude of the control priority; Setting Priority Thresholds ; Compare the priority of the data to the priority threshold: like , the data will be saved in the StarRocks online query platform database; like , then store the data in distributed data.

5. The transaction data monitoring and early warning method based on bypass flow and real-time data warehouse according to claim 4 is characterized in that: The method of executing a dynamic priority calculation strategy for any data in the data set, calculating the priority of the data in real time, and dynamically adjusting the storage location of the data according to the priority includes: Dynamically adjust priority thresholds , specifically: Get the current time The priority of all data in the data set, calculate the mean of all priorities ; Calculate the current time The standard deviation of the priorities of all data in the data set ; Calculate the current time Priority threshold ,in, is the standard quantile value under normal distribution, which is determined by the proportion of data stored in the StarRocks online query platform database.

6. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse according to claim 1 is characterized in that: The method includes executing an abnormal monitoring and alarming strategy for a data set stored in the StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list, including: Get any changed transaction data in the data set, traverse all traffic details in the data set, and match the traffic details of all changed transaction data; Determine whether there is a flow detail bill without change transaction data; If all traffic details have change transaction data, the traffic details are normal; If there is a traffic detail bill without changed transaction data, the traffic detail bill is abnormal and an alarm is issued for the traffic detail bill.

7. The transaction data monitoring and early warning method based on bypass traffic and real-time data warehouse according to claim 6 is characterized in that: The method includes executing an abnormal monitoring and alarming strategy for a data set stored in the StarRocks online query platform database, obtaining an abnormal traffic list, and issuing an alarm for the abnormal traffic list, including: Exception capture is performed for each traffic detail in the data set, specifically: The current moment Each characteristic value in the traffic list is counted as , ; Setting the abnormality judgment interval ; Get the current time Before Each feature value at time ; Calculate each eigenvalue in Interval Trend ; Calculate the abnormal score of the traffic details ,in, is the static weight of each feature, is the dynamic trend characteristic of each feature, g is the number of features in the traffic details list; Set abnormal warning threshold ; The abnormal score and abnormal warning thresholds Make comparisons; like , then the traffic details are normal; like , then the traffic details are abnormal and an early warning is issued for the traffic details.

Images