Stepped dynamic early warning method based on mixed data source

Abstract

The invention discloses a stepped dynamic early warning method based on a mixed data source, and relates to the technical field of big data processing and risk early warning. According to the method, firstly, private data, common data and plug-in snapshot data are collected through independent channels, and a standardized feature data set is generated through three-layer data lake preprocessing and federal learning feature alignment; and then, performing outlier analysis by adopting a hybrid model of an isolated forest and an LSTM, quickly identifying static abnormal points by the isolated forest, capturing time sequence abnormal points by the LSTM, performing weighted fusion on abnormal scores of the static abnormal points and the time sequence abnormal points, generating an outlier risk probability value, and completing quantitative grading of low, medium and high risks. And finally, triggering stepped early warning response according to the risk level, and configuring differentiated disposal strategies from low to high. Through multi-source data fusion, hybrid model analysis and a dynamic optimization mechanism, the problems that a traditional early warning method is single in data source, rough in judgment and rigid in strategy are solved, and accurate and self-adaptive active risk defense is achieved.

Description

Technical Field

[0001] This invention relates to the field of big data processing and risk warning technology, specifically to a tiered dynamic early warning method based on hybrid data sources. Background Technology

[0002] In the process of enterprise digital operations, risk warning is a key link in ensuring stable business operations. As enterprise digital operations evolve towards deep integration, risk warning has upgraded from a "passive response" to a core infrastructure for "proactive defense," directly linked to business continuity and compliance cost control. However, current mainstream warning systems still suffer from multi-dimensional technical bottlenecks, making it difficult to adapt to the needs of explosive data growth and increasingly complex business scenarios.

[0003] Traditional risk warning methods have several limitations: They rely on a single data source, often depending on internal enterprise private data, neglecting the value of industry-wide shared data and front-end interactive data, resulting in incomplete risk identification dimensions and a tendency to miss cross-source related risks; outlier detection methods are crude, often using fixed thresholds or simple statistical methods, failing to capture complex data patterns and time-series characteristics, leading to a high misjudgment rate of outlier risks; warning strategies are rigid, employing a "one-size-fits-all" approach without tiered responses based on risk severity, resulting in excessive warnings for low-risk events consuming resources, and insufficient response to high-risk events causing delays in handling; and models and strategies are outdated, unable to adapt to changes in data distribution and the evolution of business scenarios, leading to a decline in warning accuracy over long-term use. While existing multi-source data warning solutions attempt to integrate internal and external data, the lack of a systematic fusion architecture makes it difficult to guarantee data quality, and the modal heterogeneity, semantic conflicts, and redundant information of multi-source data remain unresolved.

[0004] Therefore, this invention provides a tiered dynamic early warning method based on a hybrid data source. It establishes a multimodal data acquisition channel for private data, collecting transaction data, inventory data, and financial data through internal enterprise API interfaces. Incremental synchronization ensures data real-time performance. The collected data undergoes data cleaning, standardization, and feature extraction to generate a standardized feature dataset with reduced dimensions. A hybrid outlier analysis model integrating Isolation Forest and Long Short-Term Memory (LSTM) networks is constructed. The Isolation Forest algorithm quickly identifies outliers in the data, while the LSTM model captures long-term dependencies in time-series data. The deviation between actual and predicted values ​​is calculated to aid in anomaly detection. Differentiated early warning response strategies are configured based on outlier risk levels. The data flow of the fused feature layer is monitored in real time, and a dynamic optimization mechanism is established. Every day at midnight, the model is incrementally trained using the previous day's new data and risk event samples to update model parameters. Early warning performance indicators are calculated weekly. When the false positive rate is high, the weight ratio of Isolation Forest and LSTM is adjusted; when the false negative rate is too high, historical samples are relabeled and the training set is expanded. Monthly, the feature extraction logic and early warning strategy parameters in the feature engineering stage are updated according to changes in business scenarios to ensure that the early warning mechanism always adapts to business needs. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is to provide a tiered dynamic early warning method based on a hybrid data source. This method solves the problems of single data source and rigid early warning strategies in existing risk warning systems, and enables tiered early warning for abnormal transactions such as fraudulent transactions, malicious orders, and fake transactions, thereby reducing transaction losses.

[0006] The technical solution adopted in this invention provides a tiered dynamic early warning method based on hybrid data sources. This includes real-time collection of transaction order data, user information data, industry fraud cases, fraudulent transaction feature database data, and real-time capture of user page operation snapshots during order placement via platform API interfaces. This integrates three types of data sources: private data, shared backflow data, and plugin snapshot data. By fusing the Isolation Forest algorithm and the LSTM model, and optimizing through weighted voting and cross-validation, a dual-dimensional outlier judgment of both static anomalies and dynamic trends is achieved, improving the accuracy of risk identification. A three-tiered risk ladder corresponding to differentiated response early warning method is proposed, addressing the problems of excessive resource consumption in low-risk situations and insufficient response in high-risk situations through cascading adaptation of risks and strategies.

[0007] In a preferred embodiment, the present invention provides a multi-source data acquisition and preprocessing method. Independent acquisition channels are established for three types of data sources: private data, shared backflow data, and plugin snapshot data. Real-time incremental processing is performed on private data via API (Application Programming Interface), while shared data undergoes batch timed processing and plugin data streaming. Different synchronization strategies are employed for different types of data. Simultaneously, a three-layer data lake is constructed: a raw layer, a cleaning layer, and a feature layer. The raw layer uses HDFS to preserve the original data structure and prevent information loss. The cleaning layer uses tools to process the data and resolve data corruption issues. The feature layer employs a federated learning lateral feature alignment algorithm to generate a unified feature set, which is stored in a time-series database for subsequent event sequence analysis.

[0008] In a preferred embodiment, the present invention also provides a hybrid model of Isolation Forest and LSTM to improve the accuracy and robustness of risk identification. The Isolation Forest algorithm utilizes its unsupervised and high-efficiency characteristics to quickly scan global features. By constructing isolated trees and randomly dividing the data space, it quickly identifies static outliers and initially identifies isolated points that are significantly different from most data patterns. The LSTM model, by setting hidden layers, predicts feature values ​​for future time based on input features from past time periods. It obtains outliers by the deviation between actual and predicted values. The outputs of Isolation Forest and LSTM are weighted and fused to obtain the outlier risk probability. Based on the analysis of a large number of historical risk events, the continuous probability is divided into three intervals: low, medium, and high, to achieve quantitative classification of risk.

[0009] In a preferred embodiment, the present invention provides a tiered early warning strategy, which combines the three-level output of outlier risk with a three-level tiered early warning strategy action. Low-risk alerts are marked and notified in the background without interrupting business processes. Medium-risk alerts are displayed in a pop-up window on the front end, suggesting that operations be paused and notifying higher-level management to intervene and verify for manual intervention. High-risk alerts are intercepted, the system automatically forces a process freeze, and notifies responsible personnel at all levels through multiple channels, initiating an emergency approval process. The strategy configuration platform includes a visual interface that allows business personnel to instantly adjust the thresholds, notification channels, and approval nodes for each risk level. All strategies are parameterized, supporting rapid adjustments to adapt to different business lines or scenarios. Attached Figure Description

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a structural diagram of a tiered dynamic early warning method based on a hybrid data source according to the present invention; Figure 2 This is a flowchart of the multi-source data acquisition and preprocessing process of this invention; Figure 3 This is a flowchart of the isolated forest and LSTM hybrid model of the present invention. Detailed Implementation

[0011] To better understand the purpose, system architecture, and functional implementation of this embodiment, the embodiments and features described herein can be combined with each other without conflict. The exemplary embodiments disclosed herein will be described below with reference to the accompanying drawings, including specific technical details disclosed to aid understanding; however, these details should be considered exemplary rather than restrictive. Therefore, those skilled in the art should understand that various improvements and adjustments can be made to the embodiments described herein without departing from the scope and core ideas of the invention. Similarly, for clarity, detailed descriptions of well-known technologies, functions, and structures are omitted in the following description.

[0012] Example 1 Figure 1 This is a structural diagram of a tiered dynamic early warning method based on a hybrid data source, according to the present invention.

[0013] like Figure 1 As shown, a tiered dynamic early warning method 100 based on a hybrid data source consists of steps S110 to S140: In step S110, multi-source data is acquired, three-layer data lake preprocessing is performed, and a unified feature set is generated; In step S120, data features are extracted, a hybrid model is used for analysis, risk probabilities are output, and risk classification is obtained. In step S130, a classification strategy is specified based on the risk classification, and the platform auxiliary operation is configured with parameters. In step S140, a closed-loop optimization mechanism is implemented, where the system collects early warning effect data and feeds back to dynamically optimize the hybrid model.

[0014] According to an embodiment of the present invention, in step S110, transaction order data, user information data, industry fraud cases, fake transaction feature database data, and real-time snapshots of user page operations when placing orders are collected in real time through the platform API interface. This integrates three types of data sources: private data, shared backflow data, and plugin snapshot data. Independent collection channels are established for the three types of data sources. Private data is collected incrementally in real time through the API interface; shared backflow data is collected in batches at regular intervals through the industry data platform API; and plugin data is streamed by capturing page interaction data through browser plugins. Different synchronization strategies are used to ensure data integrity and timeliness from the source. The process involves three layers: the raw layer, the cleaning layer, and the feature layer. The raw layer uses HDFS (Hadoop Distributed File System) to preserve the original data and prevent information loss. The cleaning layer uses ETL (Extract, Transform, Load) tools to process the data and solve the problem of data corruption. The feature layer uses a federated learning horizontal feature alignment algorithm to generate a unified feature set, which is stored in a time-series database for subsequent event sequence analysis.

[0015] According to an embodiment of the present invention, in step S120, the Isolation Forest algorithm utilizes its unsupervised and high-efficiency characteristics to quickly scan global features. By constructing an isolated tree to randomly divide the data space, it quickly identifies static outliers such as deviations from the historical mean of single large transactions, and initially identifies isolated points that are significantly different from most data patterns. The LSTM model, by setting hidden layers, predicts the feature values ​​of future times from the input feature sequences of past times. Anomalies are obtained by the deviation between the actual values ​​and the predicted values. The outputs of the Isolation Forest and LSTM are dynamically weighted and fused to obtain the outlier risk probability. Based on the analysis of a large number of historical risk events, the continuous probability is divided into three intervals: low, medium, and high, to achieve quantitative classification of risk.

[0016] According to an embodiment of the present invention, in step S130, the three-level output of outlier risk corresponds to a three-level tiered early warning strategy action. Low-risk warnings are marked and notified in the background without interrupting the business process. Medium-risk warnings are displayed in a pop-up window on the front end, suggesting that the operation be suspended and notifying the superior management to intervene and verify for manual intervention. High-risk warnings are intercepted, the system automatically forces the process to freeze, and upgrades the notification to responsible personnel at all levels through multiple channels, initiating an emergency approval process. A strategy configuration platform is provided, including a visual interface that allows business personnel to adjust the thresholds, notification channels, and approval nodes of each risk level in real time. All strategies are parameterized, supporting rapid adjustments to adapt to different business lines or scenarios.

[0017] According to an embodiment of the present invention, in step S140, data on the effectiveness of early warning execution, such as early warning trigger logs, manual handling records, and feedback from business systems, are collected. Key indicators such as false positive rate and false negative rate are calculated to determine whether the indicators exceed the threshold. When the false positive rate or false negative rate exceeds the threshold, optimization is triggered, including retraining the model, adjusting the weight parameters of the isolated forest and LSTM, adjusting the risk thresholds of different boundaries, and updating the early warning strategy. The optimized model and strategy are then redeployed to the system.

[0018] Figure 2 This is a flowchart of the multi-source data acquisition and preprocessing process of this invention.

[0019] like Figure 2 As shown, acquisition synchronization functions and periodic parameters are defined for the input private data, public return data, and plugin snapshot data, respectively. The incremental data volume for private data acquisition is specified. As shown in equation (1) below.

[0020] (1) in, for Total amount of private data at any given time. The incremental synchronization interval is set, and real-time incremental synchronization is performed through a private data API to ensure data timeliness.

[0021] Incremental data volume in shared return data As shown in equation (2) below.

[0022] (2) in, For the first Heaven from the first Data sets obtained from industry data sources To ensure a shared number of data sources, daily full batch updates are implemented. The implementation process includes a data quality verification step, where the collected data undergoes format validation and integrity checks; only qualified data proceeds to subsequent processing.

[0023] Plugin snapshot data is collected in real-time via browser plugins and WebSocket technology. A customized plugin is deployed on the user's browser to monitor user page actions, including click events, form filling, and page dwell times. The collected data is transmitted in real-time to the server via WebSocket connections, where it is parsed and processed in real-time by the Flink stream processing engine. A session management mechanism is employed during implementation to ensure the integrity and continuity of user behavior sequences.

[0024] The raw data layer is built using the HDFS distributed file system, and is stored hierarchically according to data source and date. A unified data directory structure is established during implementation, and all raw data is stored in JSON format, preserving the original appearance of the data and supporting data traceability.

[0025] The data cleaning layer is built on the Spark distributed computing framework to construct an ETL processing pipeline. During implementation, specific cleaning rules are formulated for different types of data: private data focuses on removing duplicate values, filtering outliers, and filling missing values; public data mainly performs format standardization and enumeration value mapping; and plug-in data focuses on session integrity verification and abnormal behavior filtering.

[0026] The feature data layer uses federated learning technology to realize feature alignment and fusion of multi-source data. First, the features of each data source are analyzed for correlation and distribution detection. Then, the feature space is aligned by using a horizontal federated learning algorithm to protect data privacy. The feature fusion adopts a weighted strategy to finally generate a unified feature vector, as shown in equation (3) below.

[0027] (3) in, , and These are the feature weights, and the sum of the features is 1. For private data feature sets, For a shared reflow dataset, It is a feature set for plugin snapshot data. All feature data is stored in the InfluxDB time series database, supporting efficient time series querying and analysis.

[0028] Figure 3 This is a flowchart of the isolated forest and LSTM hybrid model of the present invention.

[0029] like Figure 3 As shown, the multi-source heterogeneous data in the feature dataset undergoes standardized preprocessing, including numerical normalization, missing value imputation, and feature dimension alignment, to generate a unified standardized feature dataset, providing a consistent foundation for subsequent model input. In the left branch, an isolated forest is used to identify static outliers. First, multiple isolated trees are initialized as the basic structure of the isolated forest. For each isolated tree, a feature dimension and threshold are randomly selected, and the data space is recursively partitioned until each leaf node contains only one data point or the maximum depth of the tree is reached. For each data point, the number of path edges from the root node to the leaf node in the isolated tree is calculated, reflecting the degree of isolation of the data point.

[0030] Based on the statistical distribution of path length, static outliers that deviate from the normal pattern are screened out, and precision anomaly scores are calculated based on the average path length. As shown in equation (4) below.

[0031] (4) in, Sample size The average path length constant at that time Average path length in isolated forests The calculation formula is shown in equation (5) below.

[0032] (5) in, The number of isolated trees, For the first The output of each isolated tree shows the length of the abnormal path; the shorter the path, the higher the abnormal score. The calculation formula is shown in equation (6) below.

[0033] (6) in, Let Euler's constant be 1. The closer it is to 1, the higher the degree of static anomaly.

[0034] The right branch selects continuous historical feature data for LSTM to identify temporal anomalies, and forms an input sequence in chronological order. The two-layer LSTM hidden layer captures the long-term dependencies of the time series based on contextual information, and uses input gate, forget gate and output gate to control the memory and discard of information. Based on the historical feature sequence data, the LSTM outputs the prediction result of the feature value of the next time point, and the deviation value is obtained by subtracting the predicted value from the actual observed feature value, as shown in the following equation (7).

[0035] (7) in, For the first The first eigenvector of the day Each dimension's actual value, For each predicted value, a larger deviation indicates a higher degree of anomaly in the time series. Based on the statistical distribution of the deviation, outliers in the time series are identified. Anomaly scores are then calculated based on the deviation. As shown in equation (8) below.

[0036] (8) in, , These are the minimum and maximum values ​​of historical prediction deviation, respectively. The closer the time series anomaly score is to 1, the higher the degree of anomaly in the time series.

[0037] Static anomaly score for isolated forest neighbors And LSTM timing anomaly score By performing weighted fusion, the outlier risk probability is finally obtained. As shown in equation (9) below.

[0038] (9) in, and The sum is 1, and dynamic adjustments are made based on the varying proportions of static and temporal anomalies to improve the accuracy of outlier risk probabilities. Through a collaborative mechanism of static anomaly identification, temporal anomaly identification, and weighted fusion, precise and hierarchical risk identification is achieved, capturing both global anomaly patterns and observing dynamic changes over time. Finally, based on historical risk event sample annotations, risk thresholds are set, classifying risk probabilities into low, medium, and high risk levels, thus completing the quantitative classification of risks and achieving precise risk mapping. This provides a decision-making basis for subsequent tiered early warning systems.

[0039] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0040] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A tiered dynamic early warning method based on hybrid data sources, characterized in that, Includes the following steps: S1. Multi-source data acquisition, three-layer data lake preprocessing, and generation of a unified feature set; S2. Extract data features, perform hybrid model analysis, output risk probabilities, and obtain risk classifications; S3. Based on risk classification, specify graded early warning strategies and parameterize the platform to assist in operation; S4. Closed-loop optimization mechanism: The system collects early warning effect data and feeds back to dynamically optimize the hybrid model.

2. The tiered dynamic early warning method based on a hybrid data source as described in claim 1, characterized in that, In step S1, the multi-source data includes private data, shared backflow data, and plugin snapshot data; The data acquisition includes real-time incremental acquisition of private data through API interfaces, batch and timed acquisition of shared backflow data through industry data platform APIs, and streaming processing of plugin snapshot data by capturing page interaction data through browser plugins.

3. The tiered dynamic early warning method based on a hybrid data source as described in claim 1, characterized in that, In step S1, the three-layer data lake preprocessing includes: The raw layer uses a distributed file system to store the raw data, preserving the original data structure. The cleaning layer uses ETL tools to process the data and resolve data corruption issues; The feature layer employs a federated learning lateral feature alignment algorithm to generate a unified feature set, which is then stored in a time-series database.

4. The tiered dynamic early warning method based on a hybrid data source as described in claim 1, characterized in that, In step S2, the hybrid model analysis includes the Isolation Forest algorithm and the LSTM model; The isolated forest algorithm quickly identifies static outliers by constructing isolated trees to randomly divide the data space. The LSTM model predicts past time feature sequences by setting hidden layers and obtains anomalies by the deviation between the actual and predicted values.

5. The tiered dynamic early warning method based on a hybrid data source according to claim 4, characterized in that, The isolated forest algorithm identifies static outliers by constructing multiple isolated trees to randomly partition the data space and calculating the path length of data points. As shown in equation (1): (1)； in, Average path length in isolated forests Sample size The average path length constant at that time.

6. The tiered dynamic early warning method based on a hybrid data source according to claim 4, characterized in that, The LSTM model predicts future feature values ​​and time series anomaly scores by inputting feature sequences from the past. As shown in equation (2): (2)； in, , These are the minimum and maximum values ​​of the historical prediction deviation, respectively.

7. The tiered dynamic early warning method based on a hybrid data source according to claim 1, characterized in that, In step S2, the output risk probability is obtained by weighted fusion of the outputs of the Isolation Forest algorithm and the LSTM model. As shown in equation (3): (3)； in, and The sum is 1, and it is dynamically adjusted according to different proportions of static anomalies and time-series anomalies.

8. The tiered dynamic early warning method based on a hybrid data source according to claim 1, characterized in that, In step S3, the tiered early warning strategy includes: Low-risk notifications are sent to the background without interrupting business processes. A pop-up alert will be displayed for medium-risk cases. It is recommended to suspend operations and notify higher management for intervention and verification. For high-risk cases, intercept and freeze the process, notify responsible personnel at all levels through multiple channels, and initiate emergency approval procedures.

9. The tiered dynamic early warning method based on a hybrid data source according to claim 8, characterized in that, The parameterized configuration platform provides a visual interface that allows business personnel to adjust the thresholds, notification channels, and approval nodes for each risk level.

10. The tiered dynamic early warning method based on a hybrid data source according to claim 1, characterized in that, In step S4, the closed-loop optimization mechanism includes: The model is incrementally trained daily using the new data from the previous day and samples of risk events. Calculate the false positive rate and false negative rate weekly. If the false positive rate is too high, adjust the weight ratio of Isolation Forest and LSTM. If the false negative rate is too high, relabel historical samples and expand the training set. The feature engineering logic and early warning strategy parameters are updated monthly based on changes in business scenarios.

Images