Electronic standing book intelligent early warning method and system based on big data analysis

Abstract

The invention discloses an electronic machine account intelligent early warning system based on big data analysis. A distributed data medium table (based on a Hive number bin of a Hadoop 3. X framework and a Spark SQL component, adopting an ORC column type storage format, having daily processing capacity greater than or equal to 5 million pieces, realizing dynamic capacity expansion through YARN resource scheduling, supporting lateral expansion to ten-million-level throughput, and adopting date and business dimension dual hash for a data partitioning strategy); the Flink streaming computation engine is used for ensuring data consistency by adopting Exactly-Once semantics, realizing Checkpoint barrier synchronization based on a Kafka message queue, controlling delay within 200ms, and supporting dynamic back pressure regulation, and the peak value processing capability reaches 100,000 pieces per second; provided is a visual early warning billboard. The system response delay is controlled within 500ms (the actually measured industrial scene average delay is 378ms) through the multi-source data parallel access and stream processing channel, the efficiency is improved by 6 times compared with the traditional scheme, and the real-time performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a pre-warning method and system, in particular to an electronic account intelligent pre-warning method and system based on big data analysis. BACKGROUND

[0002] The electronic account management technology has developed from traditional database storage to an intelligent system integrating big data analysis. The current mainstream solution uses a relational database (such as Oracle, MySQL) to realize structured storage, combined with ETL tools for periodic data cleaning, but has defects such as poor real-time performance and single analysis dimension. In the fields of finance, supply chain, etc., the electronic account needs to process more than one million unstructured data (such as scanned bills, IoT device logs) per day, and the traditional technical architecture faces great challenges.

[0003] Defects of prior art (1) Limitations of static rule pre-warning: 90% of the current systems rely on preset threshold rules (such as amount exceeding limit, process exceeding time), resulting in a false positive rate of up to 35%-40%. For example, a bank's credit account system cannot identify associated transaction patterns, resulting in 127,000 invalid alerts per year.

[0004] (2) Obstacles to multi-source data fusion: Enterprise financial, logistics, contract, and other account data are scattered in heterogeneous systems, and existing technologies lack a unified data bloodline tracking mechanism. Tests show that cross-system data verification has an average delay of 4.3 hours, seriously affecting the timeliness of risk response.

[0005] (3) Insufficient intelligence: Although some enterprises have introduced machine learning (such as random forest algorithm), the model update cycle is as long as 7-15 days, making it difficult to adapt to dynamic business changes. Industry research in 2024 shows that 83% of account systems still cannot automatically generate correction suggestions. SUMMARY

[0006] The present application aims to provide an electronic account intelligent pre-warning method and system based on big data analysis to solve the problems existing in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solution: an electronic account intelligent pre-warning system, comprising: Distributed data center (Hive data warehouse based on Hadoop 3.x architecture + Spark SQL component, using ORC columnar storage format, daily processing capacity > 5 million, dynamic expansion through YARN resource scheduling, supporting horizontal expansion to tens of millions of throughput, data partition strategy using date + business dimension double hashing); Flink stream computing engine (Exactly-Once semantics to ensure data consistency, Checkpoint barrier synchronization based on Kafka message queue, delay control within 200ms, peak processing capacity up to 100,000 per second, support for dynamic back pressure adjustment); Visual warning board (integrated Echarts 5.0 chart library, provided 12 kinds of visual components such as time series heat map and three-dimensional scatter plot, supported multi-channel push such as SMS (Aliyun SMS service) / email (SMTP protocol) / enterprise WeChat (Webhook callback), alarm response rate ≥ 99.9%, average delivery delay < 2 seconds).

[0008] Specifically, the hot data storage layer (uses Redis 6.0 cluster cache to store real-time data, configures CRC16 consistency hash shards, read-write performance reaches 50,000 QPS, TTL strategy automatically cleans up expired data, supports Lua script extension data operation); Cold data storage layer (uses HDFS 3.x to store historical data, configures 3 replica redundancy strategy and ErasureCoding erasure code, supports PB-level data archiving and Snappy compression storage, cold and hot data separation threshold is set to 30 days).

[0009] Specifically, the program is executed by the processor to realize the steps of the method of any one of claims 1-3, the medium includes but is not limited to SSD solid state disk (Samsung 980 Pro NVMe protocol, read speed 7GB / s), NVMe storage device (Intel Optane persistent memory, delay < 10us) or distributed file system (Ceph RBD block storage, supports EC2 instance mounting), The program needs to be compiled to configure Java 11+ (ZGC garbage collector) or Python 3.8+ (Asyncio asynchronous framework) running environment, and the dependent library is automatically parsed through Maven central repository and PyPI image.

[0010] Storage device or distributed file system, the program needs to be compiled to configure Java 11+ or Python 3.8+ running environment.

[0011] Comprising the following steps: S1. Obtain the account data of SQL database, NoSQL database and API interface through multi-source data acquisition interface, wherein the SQL database supports relational database such as MySQL and Oracle, provides standard JDBC connection protocol and transaction isolation level configuration; the NoSQL database covers document storage of MongoDB, wide column storage of Cassandra and other unstructured data storage schemes, and supports sharding cluster deployment mode; the API interface is compatible with multiple communication protocols such as RESTful JSON / XML data exchange format and WebService SOAP protocol, and has OAuth2.0 authentication mechanism.

[0012] S2. A dynamic weight analysis model is constructed by using an LSTM neural network, the importance of data features is automatically learned through time series analysis, the model input layer contains 128 neurons and uses a tanh activation function, the feature weight parameters are updated every 6 hours through the RMSProp optimizer to perform a back propagation algorithm, and a sliding window verification mechanism is combined to ensure that the model adapts to changes in data distribution, and the model accuracy is maintained at more than 92%.

[0013] S3. A three-level early warning mechanism based on threshold triggering (preset static threshold, supporting double verification of quantile method and standard deviation method), trend fluctuation (dynamic threshold based on ARIMA (p, d, q) model, automatically calibrating difference order and moving average window) and association rule (cross-field rule mined by Apriori algorithm, minimum support set to 0.1 and confidence threshold 0.7), outputs high / medium / low three-level early warning signals through a weighted fusion strategy determined by an entropy weight method, and the comprehensive false alarm rate is less than 3%.

[0014] Compared with the prior art, the beneficial effects of the present application are: 1. Multi-source data parallel access and stream processing channel make the system response delay controlled within 500ms (average delay of 378ms in actual industrial scene), which is 6 times more efficient than the traditional scheme, and the real-time performance is improved.

[0015] 2. The dynamic weight algorithm reduces the consumption of computing resources by 43%, and realizes the saving of hardware cost through intelligent load balancing. The distributed storage architecture supports PB-level data throughput, and the performance is improved by 12 times compared with the single database.

[0016] 3. Modular design supports rapid access to new data sources (average connection period <3 person-days), and the business system is connected to the standardized interface to reduce the implementation cost, and the automatic processing interface realizes unmanned intervention in more than 90% of the conventional scenes. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The system architecture diagram of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0019] As shown in Figure 1 The present application has the following specific embodiments.

[0020] Embodiment 1 An electronic account intelligent early warning system for big data analysis comprises: A distributed data center (Hive data warehouse based on Hadoop 3.x architecture + Spark SQL component, using ORC columnar storage format, daily processing capacity ≥ 5 million, dynamic expansion is realized through YARN resource scheduling, supporting horizontal expansion to tens of millions of throughput, and data partition strategy adopts date + business dimension double hash); Flink streaming computing engine (using Exactly-Once semantic to ensure data consistency, realizing Checkpoint barrier synchronization based on Kafka message queue, delay control within 200ms, peak processing capacity up to 100,000 pieces / s, supporting dynamic back pressure adjustment); Visual early warning board (integrating Echarts 5.0 chart library, providing 12 kinds of visual components such as time sequence heat map and three-dimensional scatter plot, supporting multi-channel push such as SMS (Aliyun SMS service) / email (SMTP protocol) / Enterprise WeChat (Webhook callback), alarm response rate ≥ 99.9%, average delivery delay < 2 seconds).

[0021] Embodiment 2 The distributed data center includes: a hot data storage layer (using Redis 6.0 cluster cache real-time data, configuring CRC16 consistency hash sharding, read-write performance reaching 50,000 QPS, TTL strategy automatically cleaning expired data, supporting Lua script extending data operation); a cold data storage layer (using HDFS 3.x to store historical data, configuring 3-copy redundancy strategy and Erasure Coding error correction code, supporting PB-level data archiving and Snappy compression storage, cold and hot data separation threshold set to 30 days), the program is executed by the processor to realize the steps of the method of any one of claims 1-3, the medium includes but is not limited to SSD solid state disk (Samsung 980 Pro NVMe protocol, read speed 7GB / s), NVMe storage device (Intel Optane persistent memory, delay <10us) or distributed file system (Ceph RBD block storage, supporting EC2 instance mounting), the program needs to be configured with Java 11+ (ZGC garbage collector) or Python 3.8+ (Asyncio asynchronous framework) running environment when compiled, and the dependent library is automatically parsed through Maven central warehouse and PyPI image. Storage device or distributed file system, the program needs to be configured with Java 11+ or Python 3.8+ running environment when compiled.

[0022] The electronic account book intelligent early warning method comprises the following steps: S1. Obtain the account data of SQL database, NoSQL database and API interface through multi-source data acquisition interface, wherein the SQL database supports MySQL, Oracle and other relational databases, provides standard JDBC connection protocol and transaction isolation level configuration; the NoSQL database covers MongoDB document storage, Cassandra wide column storage and other unstructured data storage schemes, and supports sharding cluster deployment mode; the API interface is compatible with RESTful JSON / XML data exchange format, WebService SOAP protocol and other communication protocols, and has OAuth2.0 authentication mechanism.

[0023] S2. Adopting LSTM neural network to construct dynamic weight analysis model, automatically learning the importance of data features through time series analysis, the model input layer contains 128 neurons and adopts tanh activation function, and the feature weight parameters are updated every 6 hours through the reverse propagation algorithm of RMSProp optimizer, combined with sliding window verification mechanism to ensure that the model adapts to the change of data distribution, and the model accuracy is maintained above 92%.

[0024] S3. Three-level early warning mechanism based on threshold triggering (preset static threshold, supporting dual verification of quantile method and standard deviation method), trend fluctuation (dynamic threshold based on ARIMA(p, d, q) model, automatically calibrating difference order and moving average window) and association rule (cross-field rules mined by Apriori algorithm, minimum support set to 0.1 and confidence threshold 0.7), output high / medium / low three-level early warning signals through weighted fusion strategy determined by entropy weight method, comprehensive false positive rate below 3%; The missing values are filled by sliding window mean, and the window size is dynamically adjusted according to the data sampling frequency (30-minute window for high-frequency data, using exponential weighted moving average algorithm; 24-hour window for low-frequency data, using seasonal decomposition STL algorithm), and after filling, Kolmogorov-Smirnov test is performed to ensure data distribution consistency; The improved isolation forest algorithm is used to detect abnormal values, the local outlier factor (LOF) is introduced to optimize the detection accuracy of high-dimensional data, the subsampling number is set to 256 and the tree depth is set to 8, the Mahalanobis distance is used for multi-dimensional joint analysis, and the final false positive rate is controlled below 5% and an abnormality detection report is generated; Integrity evaluation unit (detect field missing rate through regular expression, support PCRE2 syntax rules, count null value proportion and generate data quality score of 0-100 points, trigger data completion process when score is below 60); Time effectiveness evaluation unit (calculate data delay time based on NTP time synchronization protocol, use PTP precise clock synchronization technology, distinguish network delay (measure through ICMP delay) and processing delay (record through buried point timer)); Logical evaluation unit (verify data compliance through 200+ pre-defined business rule templates, including data type verification, value range verification, business code mapping relationship verification, support custom rule extension through Groovy script engine).

[0025] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An electronic ledger intelligent early warning method, characterized in that The following steps are involved: S1. Ledger data from SQL databases, NoSQL databases, and API interfaces is acquired through multi-source data collection interfaces. SQL databases support relational databases such as MySQL and Oracle, providing standard JDBC connection protocols and transaction isolation level configuration. NoSQL databases include unstructured data storage solutions such as MongoDB's document storage and Cassandra's wide column storage, supporting sharded cluster deployment models. The API interface is compatible with multiple communication protocols, including the RESTful JSON / XML data exchange format and the WebService SOAP protocol, and features an OAuth2.0 authentication mechanism. Missing values ​​were filled using a sliding window mean, with the window size dynamically adjusted based on the data sampling frequency (a 30-minute window was used for high-frequency data, using an exponentially weighted moving average algorithm; a 24-hour window was used for low-frequency data, using the seasonally decomposed STL algorithm). After filling, the Kolmogorov-Smirnov test was performed to ensure data distribution consistency. An improved isolation forest algorithm is used to detect outliers. The local outlier factor (LOF) is introduced to optimize the detection accuracy in high-dimensional data. The subsampling number is set to 256 and the tree depth is set to 8. The Mahalanobis distance metric is combined to achieve multi-dimensional joint analysis. The final false alarm rate is controlled below 5% and an anomaly detection report is generated. S2. A dynamic weight analysis model was constructed using an LSTM neural network. This model automatically learns the importance of data features through time series analysis. The model input layer contains 128 neurons and uses the tanh activation function. The feature weight parameters are updated every six hours using the RMSProp optimizer and a backpropagation algorithm. A sliding window validation mechanism ensures that the model adapts to changes in data distribution, maintaining a model accuracy of over 92%. Completeness assessment unit (detects field missing rate through regular expressions, supports PCRE2 syntax rules, calculates the proportion of null values ​​and generates a data quality score of 0-100. When the score is lower than 60, the data completion process is triggered); Timeliness evaluation unit (calculates data delay based on the NTP time synchronization protocol, employs PTP precision clock synchronization technology, and distinguishes between network delay (measured via ICMP delay) and processing delay (recorded via embedded timer)); Logical evaluation unit (verifies data compliance through more than 200 predefined business rule templates, including data type verification, value range verification, and business code mapping relationship verification, and supports custom rule extension through the Groovy scripting engine); S3. A three-level warning mechanism based on threshold triggering (preset static threshold, supporting dual verification using quantile and standard deviation methods), trend fluctuation (dynamic threshold based on the ARIMA(p,d,q) model, with automatic calibration of the difference order and moving average window), and association rules (cross-field rules mined by the Apriori algorithm, with a minimum support value set to 0.1 and a confidence threshold of 0.7). A weighted fusion strategy determined by the entropy weight method outputs high / medium / low warning signals, with an overall false alarm rate of less than 3%.

2. An electronic ledger intelligent early warning system for implementing the method described in claim 1, characterized in that include: Distributed data center (based on the Hive data warehouse + Spark SQL components of the Hadoop 3.x architecture, using the ORC columnar storage format, with a daily processing capacity of ≥ 5 million records, dynamic expansion through YARN resource scheduling, and support for horizontal expansion to tens of millions of throughput. The data partitioning strategy uses double hashing of date and business dimensions); Flink streaming computing engine (using Exactly-Once semantics to ensure data consistency, implementing checkpoint barrier synchronization based on Kafka message queues, with latency controlled within 200ms, peak processing capacity of 100,000 records per second, and support for dynamic backpressure regulation); Visual early warning dashboard (integrated with the Echarts 5.0 chart library, providing 12 visualization components such as time series heat maps and 3D scatter plots, supporting multi-channel push notifications such as SMS (Alibaba Cloud SMS Service) / email (SMTP protocol) / Enterprise WeChat (Webhook callback), with an alarm response rate ≥ 99.9% and an average delivery delay of < 2 seconds).

3. The system according to claim 2, characterized in that: The distributed data center includes: Hot data storage layer (using Redis 6.0 cluster to cache real-time data, configured with CRC16 consistent hash sharding, with read and write performance reaching 50,000 QPS, TTL policy to automatically clean up expired data, and support for Lua scripts to extend data operations); Cold data storage layer (using HDFS 3.x to store historical data, configured with a three-copy redundancy strategy and Erasure Coding, supporting PB-level data archiving and Snappy compression storage, and setting the cold and hot data separation threshold to 30 days).

4. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by the processor, the steps of claim 1 are implemented. The media include but are not limited to SSD solid-state drives (Samsung 980 Pro NVMe protocol, read speed 7GB / s), NVMe storage devices (Intel Optane persistent memory, latency <10μs) or distributed file systems (CephRBD block storage, supporting EC2 instance mounting). When compiling the program, a Java 11+ (ZGC garbage collector) or Python 3.8+ (Asyncio asynchronous framework) runtime environment must be configured, and dependent libraries are automatically resolved through the Maven central repository and PyPI mirror; storage devices or distributed file systems, and a Java 11+ or Python 3.8+ runtime environment must be configured when compiling the program.

Images