Intelligent analysis method supporting real-time association and streaming processing of multiple data sources

Abstract

The invention provides an intelligent analysis method supporting multi-data-source real-time association and streaming processing, which comprises the following steps: acquiring multi-source data through a multivariate data aggregation mechanism, the multivariate data comprising business data and system data generated by a business system; performing cross-system data association on the multi-source data; constructing a streaming data processing framework, wherein the streaming data processing framework comprises operators after test optimization and arrangement; and performing complex event processing and risk assessment on the associated multi-source data through a streaming data processing framework to obtain a business risk assessment result in the multi-source data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to an intelligent analysis method that supports real-time correlation and streaming processing of multiple data sources. Background Technology

[0002] With the deepening of enterprise digital transformation, data sources such as various business systems, IoT devices, and user behavior logs are experiencing explosive growth. This data is often characterized by multiple sources, heterogeneity, real-time processing, and high concurrency. Traditional data processing methods mostly employ batch processing and offline analysis, which are insufficient to meet the needs of scenarios requiring low latency, high throughput, and strong correlation, such as real-time risk control, anomaly detection, and business monitoring. Existing technologies often face the following problems when handling real-time correlation of multi-source data: low correlation efficiency: cross-system data correlation relies on offline ETL, making real-time correlation and state synchronization impossible; limited processing capacity: traditional stream processing frameworks lack intelligent optimization in areas such as complex event processing, state management, and resource scheduling. Therefore, there is an urgent need for an analytical method that can support real-time access, intelligent correlation, and streaming processing of multiple data sources to improve the timeliness, accuracy, and resilience of data processing. Summary of the Invention

[0003] In view of this, the present invention proposes an intelligent analysis method that supports real-time correlation and streaming processing of multiple data sources to solve the problems existing in the prior art.

[0004] To achieve the above objectives, this invention proposes an intelligent analysis method that supports real-time correlation and streaming processing of multiple data sources, comprising: Multi-source data is acquired through a multi-source data aggregation mechanism, wherein the multi-source data includes business data and system data generated by the business system. Perform cross-system data association on the multi-source data; Construct a streaming data processing framework, wherein the streaming data processing framework includes tested, optimized, and orchestrated operators; By using a streaming data processing framework to perform complex event processing and risk assessment on the correlated multi-source data, business risk assessment results can be obtained from the multi-source data.

[0005] Optionally, the process of acquiring multi-source data includes: A channel connection is established with the business system to obtain the transmission data of the business system. The transmission data is automatically identified and parsed according to the corresponding protocol. The parsed data is then standardized to obtain multi-source data.

[0006] Optionally, the process of performing cross-system data association on the multi-source data includes: The multi-source data is stored with associated indexes, wherein the associated indexes include memory indexes, distributed caches, and persistent storage.

[0007] Optionally, cross-system data association of the multi-source data further includes: Obtain a query request, generate a related query execution plan based on the query request, extract the stored data according to the related query execution plan, perform matching calculations on the extracted data, sort the extracted data according to the matching calculation results, and provide the data according to the sorting results to obtain the final related multi-source data.

[0008] Optionally, the construction process of a streaming data processing framework includes: A test environment and test dataset are built. In the test environment, the operators in the streaming data processing framework are tested and their performance is analyzed using the test dataset. The operators are optimized based on the performance analysis results. The optimized operators are then intelligently orchestrated to obtain the orchestrated operators, thus obtaining the streaming data processing framework.

[0009] Optionally, the process of performing complex event processing on the correlated multi-source data includes: The event information in the multi-source data after association is extracted, vectorized, managed and filtered by operators in the streaming data processing framework. Statistical analysis and pattern mining are performed on the filtered event information to obtain normal event patterns. The system provides sample data from historical event streams through operators in the streaming data processing framework, trains machine learning operators with the sample data, and predicts future event types and times based on the trained machine learning operators. The feature vector representation of an event is obtained by operators in the streaming data processing framework, and the similarity of the time feature representation is calculated. Based on the similarity calculation results, the corresponding abnormal event is identified.

[0010] Optionally, the process of conducting a risk assessment on the correlated multi-source data includes: By using operators in the streaming data processing framework, risk features are obtained from the correlated multi-source data based on rules, statistics, and deep learning inference. The obtained risk features are then aggregated and analyzed to obtain business risk assessment results.

[0011] On the other hand, the present invention provides an intelligent analysis system that supports real-time correlation and streaming processing of multiple data sources for performing the above-described method.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Enable real-time unified access to multi-source heterogeneous data: Through intelligent protocol identification, efficient data parsing and standardized mapping mechanisms, it supports real-time access and integration of data from various IoT protocols and business systems, improving the flexibility and efficiency of data access.

[0013] 2. Supports low-latency real-time cross-system data association: Employs a multi-level index architecture and parallel query optimization to achieve millisecond-level multi-dimensional data association queries, meeting the needs of real-time risk control and business monitoring.

[0014] 3. Build a high-performance, scalable streaming processing framework: Based on operator-level performance profiling and optimization strategies, achieve in-depth optimization of the data processing pipeline; combined with an intelligent orchestration system, dynamically adjust resource allocation and task scheduling to improve the overall throughput and stability of the system.

[0015] 4. Achieve intelligent processing of complex events and real-time risk control: Improve the accuracy and real-time performance of anomaly detection and risk identification through probabilistic event pattern learning, adaptive parameter adjustment, and a three-layer risk control defense system.

[0016] 5. Supports system self-adaptation and continuous optimization: Through operator performance monitoring, dynamic parameter adjustment and model iteration mechanisms, the system has self-learning and self-adaptation capabilities, reducing manual maintenance costs and improving long-term operational reliability. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] This embodiment proposes an intelligent analysis method that supports real-time correlation and streaming processing of multiple data sources, such as... Figure 1As shown, the system supports real-time data association: it possesses real-time big data aggregation and processing capabilities, supporting data association, transformation, and complex streaming processing. The system supports real-time access and fusion of multi-source heterogeneous data, employing a complex event processing and streaming computing framework to achieve real-time data association, transformation, and aggregation across business systems. It also supports complex event detection and rule matching, providing real-time risk control and anomaly detection capabilities for business operations.

[0020] First, multi-source data is aggregated. This multi-source data refers to business data and system data generated by the enterprise's business systems during operations, including customer activity and behavior data, system operation and performance data, risk control and security data, and IoT device data from related business systems. This data comprehensively records the enterprise's customer activities, business operations, system status, risk conditions, and device status, forming the foundation for supporting real-time decision-making, risk control, operational optimization, and business innovation.

[0021] The data receiving gateway in the multi-source data aggregation mechanism adopts an asynchronous event-driven architecture based on a modern network programming framework. The entire processing flow is divided into four logically distinct stages. The first stage is connection management, which creates a dedicated event loop group to handle network connection requests and data read / write operations. Each newly established connection is assigned a unique connection identifier and a secure channel based on standard security protocols is established. During channel establishment, the encryption algorithms and data compression methods supported by both parties are negotiated. The connection pool maintains the detailed status of all active connections and promptly detects abnormal connections through a periodic heartbeat detection mechanism, attempting automatic recovery to ensure the continuous availability of the data transmission channel.

[0022] The second stage is automatic protocol identification. The gateway implements an intelligent protocol discovery mechanism, identifying the communication protocol type by analyzing the characteristic bytes at the beginning of data packets. The system maintains a protocol library containing fingerprints of various protocols, including characteristic patterns of common IoT and network protocols. For protocols that cannot be directly identified through feature matching, the gateway initiates a deep packet inspection process, inferring the protocol type by analyzing statistical feature patterns in the data stream. This dual identification mechanism significantly improves the accuracy and coverage of protocol identification.

[0023] The third stage is efficient data parsing. Each supported protocol corresponds to a parsing pipeline consisting of multiple processors. The processors in the parsing pipeline execute in a specific order, starting with raw byte decoding and progressively completing message type identification, payload extraction, quality of service verification, topic parsing, payload decoding, and finally message assembly. The parser implementation employs zero-copy optimization techniques, avoiding unnecessary data copying operations through memory view sharing. For payloads containing structured data, a streaming application programming interface (API) is used for field-by-field parsing, avoiding the construction of a complete object tree in memory, thus significantly reducing memory overhead.

[0024] The fourth stage is data standardization, establishing a three-layer data mapping model. The first layer is syntax mapping, mapping protocol-specific field names to a unified, general field naming system. The second layer is semantic mapping, using a domain-specific dictionary to convert raw values ​​into standardized business semantic expressions. The third layer is context enhancement, adding rich metadata information such as a receipt timestamp, gateway node identifier, and data quality score to each data event. The quality verification stage implements a powerful rule engine supporting various verification methods, including regular expression matching, numerical range checking, enumeration value verification, and data consistency checks. All data failing verification is routed to a dedicated exception handling queue, simultaneously triggering data quality alerts to notify relevant teams for troubleshooting.

[0025] After aggregating the data from multiple sources as described above, cross-system data association is then performed: This process is implemented through a three-layer relational index architecture, ensuring efficient mapping and real-time relational queries of data in memory, cache, and persistent storage.

[0026] The construction of the related index adopts a three-layer architecture design, including in-memory index, distributed cache, and persistent storage. Starting with the in-memory index construction, a high-performance caching library is used to implement an eviction policy based on the least recently used frequency (LRU), considering not only the most recent access time of the data but also statistically analyzing access frequency patterns. The index data structure uses an efficient bitmap to store the document identifier set, while for text fields, a finite state converter (FST) is used to store the inverted index for fast data association and matching. The bitmap index compresses the document identifier set into a bit array, supporting efficient AND, OR, and NOT bitwise operations at the CPU register level, thereby enabling real-time logical association and cross-filtering based on multi-dimensional conditions. The finite state converter (FST) used for text fields performs prefix compression storage of terms in memory, achieving fast lookup and fuzzy matching with O(word length) complexity, and directly outputting the corresponding bitmap reference. The combination of these two approaches allows the system to complete complex queries involving text matching and multi-condition association within milliseconds. The in-memory index periodically generates snapshots and asynchronously persists them to the local disk, ensuring rapid recovery of the index state after a system restart.

[0027] The distributed caching layer employs a proven distributed key-value storage cluster, using a proxy middleware to manage data shards. Each shard is configured with multiple replicas to ensure high availability. The cached data structure is meticulously designed, incorporating multiple index types, including primary key indexes, secondary indexes, and full-text indexes, to meet the needs of different query scenarios. The primary key index is the primary key index for the corresponding data, where the primary key represents the device, user, or business type. Secondary indexes are specific representations or IDs for these types, used to locate a specific device, user, or business. The full-text index is used for searching full-text content. These indexes are configured hierarchically. The cache update strategy adopts a write-through mode to ensure data consistency between the cache and the underlying database, avoiding dirty data issues.

[0028] The persistent storage layer utilizes a mature search and analytics engine cluster, managing data shards according to time dimensions. New indexes are created automatically daily, and when duplicate indexes are found, they are merged, placing the sub-indexes under the same index. The index mapping template predefines the data types and text analyzer configurations for all fields, supporting advanced text processing functions such as Chinese word segmentation, pinyin conversion, and synonym expansion. Index settings parameters have been thoroughly optimized, including key parameters such as the number of shards, the number of replicas, the refresh interval, and transaction log durability, finding the optimal balance between index performance and reliability.

[0029] For the data storage structure described above, the system executes a join query process, which consists of four steps: The first step is query parsing and association optimization. The system parses the association conditions submitted by the user (such as cross-system user behavior association), identifies the filtering conditions that can be pushed down to the storage layer, and generates the optimal association query execution plan.

[0030] The second step is to execute the parallel join query, which distributes the join query task to the relevant data shards. Each shard independently performs the join matching operation using its local index and extracts all the content under the current index. Then, the cross-system join results of all shards are aggregated through a distributed collection mode.

[0031] The third step is result refinement and relevance scoring. The preliminary matching results are refined by using a mature text relevance algorithm to calculate the degree of matching. Based on the degree of matching, the extracted content is sorted, and several results from the top of the sorted list are provided.

[0032] The fourth step is result encapsulation and return. The associated result set is returned to the caller in a paginated manner, and also includes rich metadata such as the total number of matches, query time, and shard statistics to help users understand the quality and completeness of the associated results.

[0033] Next, we built a streaming data processing framework: Operators, as the fundamental computational units in data processing systems, undertake a series of core functions, from data cleaning and transformation to aggregation analysis. In stream processing frameworks such as Apache Flink, operators can be specific operation units such as Map (data mapping), Filter (data filtering), KeyBy (key grouping), Window (time window), or Aggregate (aggregation calculation). Each operator is like a specialized workstation on a factory production line—the Map operator is responsible for format conversion (such as parsing a JSON string into a structured object), the Filter operator is responsible for quality screening (such as filtering out invalid transaction records), and the Aggregate operator is responsible for statistical summarization (such as calculating the total transaction amount per minute). These operators are connected through data streams to form a complete data processing pipeline, and the first step in optimizing operator performance is to establish a scientific test benchmark environment for these fundamental computational units.

[0034] The test environment is constructed using containerization technology to create a completely isolated, standardized test space, ensuring that each performance test of an operator is conducted in the exact same underlying environment. This environmental isolation is crucial for operator performance evaluation—it eliminates the interference of external factors such as hardware differences, operating system versions, and software dependencies on test results, making the comparison of the effects of different optimization strategies truly comparable. The test environment comprises three core layers: data generation tools are specifically designed for the processing characteristics of different types of operators, simulating data flow patterns under real business scenarios. For example, to test the transformation efficiency of the Map operator, the tool generates nested JSON data that requires complex parsing; to test the filtering capabilities of the Filter operator, the tool generates mixed data containing various boundary conditions; and to test the time management of the Window operator, the tool generates time-series data streams with out-of-order timestamps.

[0035] The test dataset was designed following a progressive approach, with three tiers of test case sets tailored to operators of varying complexity. The initial test set contains small datasets of hundreds of thousands of data points, primarily used to verify the basic functional integrity of the operators. For example, it tests whether a Filter operator can correctly identify and filter out all transaction records with negative amounts, and whether a Map operator can accurately convert temperature units. The intermediate test set expands to millions of data points, simulating typical business load scenarios and focusing on the stable performance of operators under moderate pressure. This stage tests the memory usage changes of the Aggregate operator under continuous data inflow and the load balancing capability of the KeyBy operator when data distribution is uneven. The advanced test set reaches tens or even hundreds of millions of data points, corresponding to extreme scenarios of peak system load, used to evaluate the operators' extreme processing capabilities and anomaly recovery mechanisms. In this stage, data skew is intentionally created to test the backpressure handling capability of the Window operator, and network fluctuations are simulated to test the fault tolerance and recovery mechanism of the Sink operator.

[0036] After the standardized testing environment is ready, the system enters the in-depth operator performance profiling phase. The goal of this phase is to make the operator computation process completely transparent, accurately quantifying the resource consumption and time overhead of each processing step. Professional performance analysis tools approach the operator execution process from multiple dimensions: CPU hotspot analysis uses sampling techniques to capture the most frequently executed code paths by the processor, identifying the functions and methods in the operator that consume the most computational resources. For example, the analysis might reveal that the JSON parsing function of a certain Map operator consumes 70% of the CPU time, or that the hash calculation of a certain Aggregate operator has become the bottleneck.

[0037] Runtime logging tools delve into the Java Virtual Machine to track the creation, survival, and destruction of objects during operator execution. Analysis revealed that a certain Filter operator created a new regular expression object each time data was filtered, even though these objects could be reused; or a certain Window operator generated a large number of short-lived small objects while maintaining state, putting significant pressure on the garbage collection system. The concurrency analysis module monitored lock contention between threads within operators, discovering performance degradation due to inadequate synchronization design. For example, a KeyBy operator used a coarse-grained lock when updating a global counter, severely limiting parallel processing capabilities.

[0038] Key performance indicators (KPIs) were collected to establish a refined measurement system for each operator. For the Filter operator, not only was the average filtering latency measured, but more importantly, P95 and P99 latency data were collected, because even if only 1% of the data was processed too slowly, it could cause the entire pipeline to be blocked. For the Map operator, the throughput metric (events / second) needed to be analyzed in conjunction with CPU utilization to determine if there were any issues with underutilization of computing resources. For stateful operators such as Window and Aggregate, detailed tracking of memory usage patterns was required: the frequency of garbage collection in the young and old generations, the pause time of each garbage collection, and the trend of state size changes over time. This detailed performance data constructed a complete performance profile for each operator, clearly revealing optimization directions and priorities.

[0039] Based on the precise diagnostic data obtained in the performance profiling phase, operator optimization work enters the targeted strategy implementation phase. The core idea of ​​this phase is to select the most suitable optimization technique according to the specific problem of each operator, avoiding blind optimization in a "one-size-fits-all" manner.

[0040] For operator memory issues identified during diagnosis, a tiered optimization strategy was implemented. For frequently created regular expression objects in the Filter operator, a lightweight object pool mechanism was introduced to improve object reuse. For the numerous temporary string objects in the Map operator, the internal character processing logic was optimized to reduce the generation of intermediate objects. For the state data maintained by the Window operator, intelligent serialization optimization was implemented, compressing the state size by 30% without affecting processing correctness. Specifically for large state operators, incremental checkpointing and state sharding management were implemented, reducing the overhead of state persistence by 50%.

[0041] In terms of computational optimization, the hardware potential of modern processors is fully explored. For the intensive numerical computations in the Aggregate operator, vectorization optimization is implemented, increasing floating-point throughput by 3 times through the SIMD instruction set. For the complex business logic in the Map operator, loop unrolling and branch prediction optimizations are performed, reducing computation latency on the critical path by 25%. For a few core operators that are indeed subject to Java Virtual Machine performance limitations, such as regular expression matching of specific patterns or complex geometric calculations, native libraries are called via JNI to achieve near-native code execution efficiency.

[0042] Input / output optimizations are tailored to the data transmission characteristics of the operators. For batch writes to the database by the Sink operator, batch size and refresh strategy are optimized to find the optimal balance between data consistency and write performance. For data reads by the Source operator, zero-copy technology is implemented to avoid unnecessary data copying between user space and kernel space. For data exchange between operators, serialization protocols and compression algorithms are optimized to reduce network transmission overhead by 40% while maintaining low latency.

[0043] Rigorous verification of optimization effects and integration with intelligent orchestration: Optimized operators must undergo rigorous regression testing for verification. Performance regression testing reruns the complete benchmark suite, requiring optimized operators to achieve preset quantitative targets: a 30% reduction in critical path processing latency, a 50% increase in throughput, and a 40% improvement in memory usage efficiency. Functional regression testing ensures that the optimization does not compromise the correctness of the operators; the Filter operator can still accurately filter, the Map operator can still correctly transform, and the Aggregate operator can still accurately perform statistics.

[0044] Operator-level optimization provides high-quality input data for job-level intelligent orchestration. As each operator undergoes deep optimization, its performance characteristics become more stable and predictable. Based on this optimized operator performance data, the intelligent orchestration system trains a more accurate multi-objective prediction model. The system can accurately predict how many CPU cores, memory capacity, and network bandwidth a job containing optimized Map, Filter, and Aggregate operators will need to process millions of data points.

[0045] During job execution, the intelligent orchestration system performs fine-grained monitoring at the operator level. When the actual performance of an operator deviates from the predicted value, the system can make precise adjustments. For example, if an increase in latency is detected for a Window operator, the system will automatically increase the number of parallel instances of that operator; if abnormal memory usage is detected for an Aggregate operator, the system will dynamically adjust heap memory allocation. This intelligent adjustment based on operator-level monitoring is faster, has less impact, and is more effective than coarse-grained job-level adjustments.

[0046] Operator optimization and intelligent orchestration have formed a virtuous cycle of synergistic evolution. Operator-level optimization improves the efficiency of basic computing units, providing better optimization space for intelligent orchestration; feedback collected by intelligent orchestration in actual operation guides the focus of the next round of operator optimization. Through this continuous cycle of micro and macro optimization, the data processing system achieves multi-level performance improvements from operators to jobs to the entire platform, ultimately providing users with efficient, stable, and intelligent data processing services.

[0047] Implementation of a complex event processing engine and real-time risk control system within a streaming data processing framework: Building upon the operator system of the streaming data processing framework, the complex event processing engine has achieved a fundamental evolution towards intelligence. This evolution is first reflected in the expansion of the probabilistic event pattern definition language, which can now fully leverage operator processing capabilities to express complex temporal logic. The system uses the Map operator to extract features and vectorize events, the Window operator to manage event sequence patterns with temporal fluctuations, and the Filter operator to perform conditional filtering of partially matching confidence events. More importantly, the system introduces automatic pattern learning capabilities. After conditional filtering, a statistical analysis and pattern mining pipeline built based on the Aggregate operator is implemented: the system automatically discovers frequently occurring pattern sequences from historical event streams, transforms the learned patterns into normal or high-quality recommended content, and recommends pattern templates that users may be interested in, providing corresponding information for controlling related events and displaying content of interest.

[0048] The matching engine deeply integrates machine learning techniques and operator processing capabilities. During the training phase, the system uses the Source operator to extract positive and negative sample data from the historical event stream, the Filter operator to filter valid samples, and the Map operator to perform feature extraction and transformation. The features extracted from each event include temporal features (analyzed using a time window operator), content features (extracted using a text processing operator), and contextual features (obtained using an association operator). The training process uses specialized machine learning operators to train the neural network model. These operators are deeply optimized to efficiently handle large-scale training data. The event sequence prediction model trained by the system can accurately predict the types and timing of future events.

[0049] During the inference phase, when a new event arrives at the system through the Source operator in the data access layer, its feature vector representation is calculated in real time. This process is completed by a feature extraction pipeline composed of multiple Map operators, each responsible for specific feature calculations. The system maintains the current event sequence state, manages the sequence's time window through the Window operator, and maintains the sequence's statistical features through the Aggregate operator. After encoding the current event sequence into a fixed-dimensional vector representation, the system uses a specialized matching operator to calculate the similarity between this sequence vector and a predefined pattern vector. When the similarity exceeds a set threshold, the matching operator triggers a pattern matching event, selecting the abnormal event corresponding to the similarity exceeding the set threshold.

[0050] The operator-optimized risk control system achieves a three-layer defense architecture: The risk control system's three-tiered defense architecture is built upon deeply optimized operators, forming a complete protection chain from rapid response to in-depth analysis. The first layer, a rule-based fast filtering engine, fully leverages the high-performance filtering capabilities of filter operators. The rule compilation process transforms user-defined risk patterns into efficient operator combinations, eliminating redundant calculations through operator optimization. Performance metrics achieve single-rule evaluation at the microsecond level, thanks to object pool optimization and improved vectorized computation of filter operators. The system supports the parallel execution of thousands of rules, each corresponding to an independent filter operator instance. These instances are optimized and scheduled through an intelligent orchestration system to ensure efficient resource utilization. The rule definition adopts a declarative syntax. For example, the high-frequency login detection rule "the same user fails to log in more than 3 times within 5 minutes" is compiled into a combination of the Window operator (time window management) and the Aggregate operator (counting statistics); the abnormal geolocation access rule "the user's login location changes by more than 1,000 kilometers within 2 hours" is compiled into a combination of the Map operator (geographic location calculation) and the Filter operator (condition judgment). Through the above series of Filter operator combinations and rule compilation evaluation, the current risk assessment index is output through the combination of operators with computation capabilities.

[0051] The second-layer statistical model analysis engine is built on a stateful function framework, fully leveraging the time window management capabilities of the Window operator and the incremental computation capabilities of the Aggregate operator. The system maintains independent sliding window statistics for each user, with this maintenance handled by dedicated state operators. Statistical characteristics of various behavioral indicators are calculated in real time and output as data, including login frequency, transaction amount distribution, and device usage patterns. These calculations are performed in parallel by multiple Aggregate operators. The system uses the 3σ principle to identify outliers, but improves upon the traditional static thresholding method by dynamically tracking baseline changes in indicators through an exponentially weighted moving average algorithm—a function implemented by the prediction operator. The statistical window slides forward at a fixed frequency, and the statistical calculations employ an incremental update algorithm, avoiding repeated scanning of historical data. This optimization improves the computational efficiency of the Aggregate operators. In particular, the system implements intelligent state serialization optimization. For the time window states maintained by the Window operator, a compression serialization algorithm is used to reduce the state size, significantly lowering the overhead of state management.

[0052] The third-layer deep learning inference engine represents the system's highest-level analytical capability. This layer is implemented using specially optimized neural network operators. The model employs a multi-head attention neural network architecture, capable of capturing long-range dependencies between events to generate predicted possible risk event types and probabilities. The model inputs are carefully designed: event sequence information is transformed into dense vectors through embedding operators; time interval features are processed by normalization operators; and attribute vector representations integrate multi-dimensional information such as user profiles, device fingerprints, and environmental features. This feature fusion is accomplished by feature concatenation operators. Training data comes from labeled datasets of historical risk events, and the collection and labeling process is tightly integrated with the data quality management system. The online inference service is deployed through a dedicated model service framework, supporting batch inference optimization—the system merges inference requests within a given time period using batch processing operators, leveraging the parallel computing power of GPUs to improve throughput. The neural network operators undergo specialized performance optimizations, including kernel fusion, memory access optimization, and computation graph optimization, reducing inference latency. The model update mechanism employs a two-tiered strategy: daily incremental training fine-tunes model parameters based on the day's risk event data; weekly full retraining re-examines the entire training set to ensure the model doesn't forget long-term patterns due to over-adaptation to recent data. These training tasks are executed by a dedicated cluster of training operators, with computing resources dynamically allocated through an intelligent orchestration system.

[0053] Risk scoring fusion mechanism: The risk scoring fusion mechanism integrates the outputs of the three-layer defense system into a unified risk assessment through a fusion operator. The fusion operator contains a deep learning model to output the final risk assessment result.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An intelligent analysis method supporting real-time correlation and streaming processing of multiple data sources, characterized in that, include: Multi-source data is acquired through a multi-source data aggregation mechanism, wherein the multi-source data includes business data and system data generated by the business system. Perform cross-system data association on the multi-source data; Construct a streaming data processing framework, wherein the streaming data processing framework includes tested, optimized, and orchestrated operators; By using a streaming data processing framework to perform complex event processing and risk assessment on the correlated multi-source data, business risk assessment results can be obtained from the multi-source data.

2. The method according to claim 1, characterized in that, The process of acquiring multi-source data includes: A channel connection is established with the business system to obtain the transmission data of the business system. The transmission data is automatically identified and parsed according to the corresponding protocol. The parsed data is then standardized to obtain multi-source data.

3. The method according to claim 1, characterized in that, The process of performing cross-system data association on the multi-source data includes: The multi-source data is stored with associated indexes, wherein the associated indexes include memory indexes, distributed caches, and persistent storage.

4. The method according to claim 3, characterized in that, Cross-system data association of the aforementioned multi-source data also includes: Obtain a query request, generate a related query execution plan based on the query request, extract the stored data according to the related query execution plan, perform matching calculations on the extracted data, sort the extracted data according to the matching calculation results, and provide the data according to the sorting results to obtain the final related multi-source data.

5. The method according to claim 1, characterized in that, The construction process of a streaming data processing framework includes: A test environment and test dataset are built. In the test environment, the operators in the streaming data processing framework are tested and their performance is analyzed using the test dataset. The operators are optimized based on the performance analysis results. The optimized operators are then intelligently orchestrated to obtain the orchestrated operators, thus obtaining the streaming data processing framework.

6. The method according to claim 1, characterized in that, The process of performing complex event processing on correlated multi-source data includes: The event information in the multi-source data after association is extracted, vectorized, managed and filtered by operators in the streaming data processing framework. Statistical analysis and pattern mining are performed on the filtered event information to obtain normal event patterns. The system provides sample data from historical event streams through operators in the streaming data processing framework, trains machine learning operators with the sample data, and predicts future event types and times based on the trained machine learning operators. The feature vector representation of an event is obtained by operators in the streaming data processing framework, and the similarity of the time feature representation is calculated. Based on the similarity calculation results, the corresponding abnormal event is identified.

7. The method according to claim 1, characterized in that, The process of conducting risk assessment on correlated multi-source data includes: By using operators in the streaming data processing framework, risk features are obtained from the correlated multi-source data based on rules, statistics, and deep learning inference. The obtained risk features are then aggregated and analyzed to obtain business risk assessment results.

8. An intelligent analysis system supporting real-time correlation and streaming processing of multiple data sources, characterized in that, Used to perform the method described in any one of claims 1-7.

Images