Financial data anomaly detection method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a financial data anomaly detection method, device and equipment and a medium, which can acquire multi-dimensional financial data from a plurality of data sources, and improve the anomaly detection accuracy in combination with the multi-dimensional data. A data interface of the Apache Flink framework is called to carry out real-time preprocessing on the multi-dimensional financial data, so that the data processing efficiency and the data quality are improved; the Apache Flink framework is used for performing stream processing and window calculation on the data stream to be processed, and multi-dimensional features are constructed, so that the detection accuracy is improved; and the multi-dimensional features are input into a fusion model pre-trained based on a machine learning algorithm, and an anomaly detection result and a response strategy are generated in combination with feedback data of a specified platform, so that the accuracy of the financial data anomaly detection result can be further ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for detecting anomalies in financial data. Background Art

[0002] In the financial field, real-time monitoring and anomaly detection are crucial for preventing fraud, credit risk, and credit default. However, existing anomaly detection technologies usually rely on static models and cannot adapt to the dynamically changing financial data environment, resulting in poor detection results.

[0003] Moreover, existing technologies often suffer from missed detections and false positives when detecting anomalies in real-time data streams, which affects the accuracy and timeliness of decision-making.

[0004] In summary, the existing technologies have the following main problems: poor model adaptability, high missed detection rate, and high false alarm rate, which cannot meet the strict requirements of the financial field for real-time monitoring. Summary of the Invention

[0005] In view of the above, it is necessary to provide a financial data anomaly detection method, device, equipment and medium, aiming to solve the problem that existing models cannot meet the real-time detection needs of financial data due to poor adaptability, high missed detection rate and high false alarm rate.

[0006] A method for detecting anomalies in financial data, comprising:

[0007] Acquire multi-dimensional financial data from multiple data sources;

[0008] Calling the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed;

[0009] Using the Apache Flink framework to perform stream processing and window calculation on the data stream to be processed to obtain a stream processing result;

[0010] Constructing multidimensional features according to the stream processing results;

[0011] Inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result;

[0012] generating an anomaly response strategy based on the initial anomaly detection result;

[0013] Sending the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation;

[0014] Generate target anomaly detection results and target anomaly response strategies based on the feedback data of the designated platform.

[0015] According to a preferred embodiment of the present invention, using the Apache Flink framework to perform stream processing and window calculation on the data stream to be processed to obtain a stream processing result includes:

[0016] Extracting a timestamp from the data stream to be processed; wherein the timestamp includes a business timestamp and a system processing time;

[0017] Generate a Watermark based on the timestamp; wherein the Watermark is used to represent the timestamp up to the current Watermark, and all data earlier than the timestamp has arrived;

[0018] Grouping the data stream to be processed according to the configured grouping strategy using a KeyBy operator to obtain at least one data stream group;

[0019] Allocate the data in each data stream group to each window according to the configured window size, sliding step size and the timestamp;

[0020] Triggering the computation of each window according to the Watermark to obtain the stream processing result, and persisting the stream processing result to the state backend using the KeyedState keyed state of the Apache Flink framework;

[0021] The calculation operation includes calculating the number of applications in each window, calculating the statistical value of the application amount in each window, and detecting the application time pattern to perform pattern-based event sequence detection.

[0022] According to a preferred embodiment of the present invention, constructing multidimensional features according to the stream processing results includes:

[0023] Extracting application frequency, time interval distribution, and amount change trend from the stream processing results as time series features;

[0024] Extracting device information features, geographic location features, and operation behavior sequences from the stream processing results to construct a user behavior profile as a behavior feature;

[0025] Analyze the social network analysis data, associated account patterns, and abnormal behavior identification results in the stream processing results as associated features based on a graph algorithm;

[0026] Extracting statistics and distribution characteristics of each dimension in the stream processing result as statistical features;

[0027] Obtaining derived features by performing feature crossover and feature transformation on the stream processing results;

[0028] Determining the importance of each of the time series features, the behavior features, the association features, the statistical features, and the derived features through information gain, and selecting a first feature from the time series features, the behavior features, the association features, the statistical features, and the derived features according to the importance of each feature;

[0029] Performing feature crossover and nonlinear transformation on the first feature to obtain a second feature;

[0030] Performing dimensionality reduction processing on the second feature to obtain a third feature;

[0031] Encoding the third feature according to the configuration format, and concatenating the encoded features into a vector to obtain the multidimensional feature;

[0032] The multidimensional features are persisted to a storage system, and an index of the multidimensional features is constructed.

[0033] According to a preferred embodiment of the present invention, inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result includes:

[0034] Dynamically loading model parameters of each sub-model in the fusion model from the model warehouse; wherein the sub-models include a random forest model, an Xgboost model, and a long short-term memory network;

[0035] Using each sub-model to process the multi-dimensional features respectively to obtain a processing result of each sub-model;

[0036] Get the weight of each sub-model;

[0037] Perform weighted calculation based on the processing results of each sub-model and the weight of each sub-model to obtain the outlier value;

[0038] When the abnormal value is greater than a high-risk threshold, marking the multi-dimensional financial data as high-risk and generating a high-priority alert; or

[0039] When the abnormal value is greater than the medium risk threshold and less than or equal to the high risk threshold, marking the multi-dimensional financial data as medium risk and generating a medium priority alert; or

[0040] When the abnormal value is greater than a low-risk threshold and less than or equal to a medium-risk threshold, marking the multi-dimensional financial data as low-risk and generating a low-priority alert; or

[0041] When the abnormal value is less than or equal to the low risk threshold, marking the multi-dimensional financial data as normal;

[0042] The marked risk levels and generated alerts are integrated to obtain the initial anomaly detection results.

[0043] According to a preferred embodiment of the present invention, generating an abnormality response strategy according to the initial abnormality detection result includes:

[0044] Performing similarity merging on the initial anomaly detection results according to an aggregation strategy to obtain aggregated data;

[0045] detecting whether the aggregated data requires a real-time response;

[0046] When the aggregated data requires a real-time response, the aggregated data is pushed to the alarm system via the Kafka message queue;

[0047] When the aggregated data corresponds to the high-priority alarm, determining that the abnormal response strategy is to control the alarm system to suspend the business process corresponding to the multi-dimensional financial data and send a review application to the expert platform; or

[0048] When the aggregated data corresponds to the medium priority alarm, determining that the abnormal response strategy is to control the alarm system to trigger a secondary check of the aggregated data and add the aggregated data to a manual review queue; or

[0049] When the aggregated data corresponds to the low-priority alarm, the abnormal response strategy is determined to control the alarm system to mark the aggregated data as data to be observed, and add the aggregated data to a batch review queue.

[0050] According to a preferred embodiment of the present invention, generating a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform includes:

[0051] When the feedback data shows that the target anomaly detection result is a true anomaly, the initial anomaly detection result is determined as the target anomaly detection result, and the anomaly response strategy is determined as the target anomaly response strategy; wherein the multi-dimensional financial data, the target anomaly detection result, and the target anomaly response strategy are added as cases to a knowledge base, and the business rules corresponding to the multi-dimensional financial data are adjusted; or

[0052] When the feedback data shows that the target anomaly detection result is a false anomaly, the initial anomaly detection result is marked as a false positive.

[0053] According to a preferred embodiment of the present invention, the method further includes:

[0054] At every preset period, or when the performance indicator of the fusion model is lower than the performance threshold, or when the cumulative new sample volume reaches the configured sample volume, historical samples are obtained, and new cases are loaded from the knowledge base to construct a full sample volume, and the full sample volume is determined as a training sample; or

[0055] When the incremental data captured in real time based on the Flink ProcessFunction meets a preset condition, the incremental data is determined as the training sample;

[0056] Configuring initial weights for each feature in the training sample based on feature importance analysis;

[0057] Using the SMOTE oversampling algorithm to generate synthetic samples based on the minority class in the training samples, and adding the synthetic samples to the training samples;

[0058] Utilizing the supplemented training samples to optimize and train the fusion model to obtain a new model;

[0059] Performing model evaluation and parameter optimization on the new model;

[0060] The optimized new model is deployed, and the model parameters of the optimized new model are recorded in the model warehouse.

[0061] A financial data anomaly detection device, comprising:

[0062] An acquisition unit, used to acquire multi-dimensional financial data from multiple data sources;

[0063] A preprocessing unit, configured to call a data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed;

[0064] A stream processing unit, configured to perform stream processing and window calculation on the data stream to be processed using the Apache Flink framework to obtain a stream processing result;

[0065] A construction unit, configured to construct a multidimensional feature according to the stream processing result;

[0066] An input unit, configured to input the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result;

[0067] a generating unit, configured to generate an anomaly response strategy according to the initial anomaly detection result;

[0068] A sending unit, configured to send the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation;

[0069] The generating unit is further configured to generate a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform.

[0070] A computer device, comprising:

[0071] a memory storing at least one instruction; and

[0072] A processor executes instructions stored in the memory to implement the financial data anomaly detection method.

[0073] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the financial data anomaly detection method.

[0074] As can be seen from the above technical solutions, the present invention can obtain multi-dimensional financial data from multiple data sources, and improve the accuracy of anomaly detection by combining multi-dimensional data; call the data interface of the Apache Flink framework to perform real-time preprocessing of multi-dimensional financial data, thereby improving data processing efficiency and data quality; use the Apache Flink framework to perform stream processing and window calculations on the processed data stream, and construct multi-dimensional features, thereby improving detection accuracy; input the multi-dimensional features into a fusion model pre-trained based on a machine learning algorithm, and combine it with feedback data from a specified platform to generate anomaly detection results and response strategies, which can further ensure the accuracy of financial data anomaly detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flow chart of a preferred embodiment of the financial data anomaly detection method of the present invention.

[0076] Figure 2 It is a functional module diagram of a preferred embodiment of the financial data anomaly detection device of the present invention.

[0077] Figure 3 It is a structural diagram of a computer device according to a preferred embodiment of the present invention for implementing the method for detecting anomalies in financial data. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the method for detecting anomalies in financial data according to the present invention. The order of the steps in the flow chart can be changed and some steps can be omitted according to different requirements.

[0080] The financial data anomaly detection method is applied to one or more computer devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Their hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0081] The computer device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0082] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0083] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0084] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0085] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0086] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0087] S10, obtaining multi-dimensional financial data from multiple data sources.

[0088] In this embodiment, the multi-dimensional financial data may include data obtained from the financial system, and may also include user behavior logs, historical transaction data, etc.

[0089] The integration of multi-dimensional data can provide a comprehensive data foundation for subsequent detection, thereby improving the accuracy of anomaly detection.

[0090] S11, calling the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed.

[0091] In this embodiment, calling the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data includes:

[0092] Calling the data interface to perform real-time cleaning on the multi-dimensional financial data to process missing values, outliers, and duplicate data; and / or

[0093] Standardizing the numerical features in the multi-dimensional financial data using the Z-score or Min-Max method; and / or

[0094] The sensitive information in the multi-dimensional financial data is encrypted or masked.

[0095] Among them, data cleaning can reduce data redundancy and improve data quality.

[0096] Among them, the stability of data distribution can be guaranteed through standardization processing.

[0097] Among them, encryption or masking can ensure data security and compliance.

[0098] S12: Use the Apache Flink framework to perform stream processing and window calculation on the data stream to be processed to obtain a stream processing result.

[0099] In this embodiment, the stream processing and window calculation are performed on the data stream to be processed by using the Apache Flink framework to obtain the stream processing result, which includes:

[0100] Extracting a timestamp from the data stream to be processed; wherein the timestamp includes a business timestamp and a system processing time;

[0101] Generate a Watermark based on the timestamp; wherein the Watermark is used to represent the timestamp up to the current Watermark, and all data earlier than the timestamp has arrived;

[0102] Grouping the data stream to be processed according to the configured grouping strategy using a KeyBy operator to obtain at least one data stream group;

[0103] Allocate the data in each data stream group to each window according to the configured window size, sliding step size and the timestamp;

[0104] Triggering the computation of each window according to the Watermark to obtain the stream processing result, and persisting the stream processing result to the state backend using the KeyedState keyed state of the Apache Flink framework;

[0105] The calculation operation includes calculating the number of applications in each window, calculating the statistical value of the application amount in each window, and detecting the application time pattern to perform pattern-based event sequence detection.

[0106] The business timestamp may be a time node related to the business, such as the time when the log is generated. The business timestamp may be preferably used to better meet the actual business scenario.

[0107] The watermark can be generated periodically (e.g., every 500ms) or based on event triggers. Watermarks allow the stream processing framework to correctly identify the temporal relationship of data. Even if data is out of order (e.g., network delays cause later data to arrive first), it can be processed in an orderly manner according to business time. This avoids window calculation errors caused by out-of-order data (e.g., late data is discarded), ensuring the accuracy of the results.

[0108] The KeyBy operator can be used to group data streams by key based on business identifiers such as the applicant ID, thereby forming at least one data stream group. After grouping, all data from the same applicant will be processed in the same parallelism (e.g., the same thread, the same task), ensuring state consistency. Furthermore, logical data partitioning allows subsequent "state management and window calculations" to be processed independently based on specific dimensions (such as the applicant dimension) (e.g., counting the number of applications for each applicant), effectively preventing interference between data from different applicants.

[0109] The window size and sliding step can be configured based on actual task requirements. For example, the window size can be set to 1 hour, meaning the window covers 1 hour of business time data; the sliding step can be set to 5 minutes, meaning a new window is generated every 5 minutes, and windows can overlap. Window configuration enables continuous, fine-grained monitoring, and window overlap ensures that no data is missed (e.g., data from 10:02 will be included in the 09:05-10:05 window).

[0110] Among them, through pattern-based event sequence detection, time patterns can be matched within the window, such as an event in which the interval between three consecutive applications is less than 10 minutes. This can identify intensive applications in a short period of time that are disguised as normal behaviors, thereby improving risk prevention and control capabilities.

[0111] Among them, the KeyedState keyed state of the Apache Flink framework is used to persist the stream processing results to the state backend (such as a distributed database, etc.), which can support fault recovery (for example, after the task is restarted, the historical calculation results can be restored from the state backend).

[0112] This embodiment uses Apache Flink's efficient stream processing capabilities to efficiently process real-time, large-scale financial data streams, meeting the strict requirements of the financial field for real-time monitoring.

[0113] S13: Construct multidimensional features according to the stream processing results.

[0114] In this embodiment, constructing multidimensional features according to the stream processing results includes:

[0115] Extracting application frequency, time interval distribution, and amount change trend from the stream processing results as time series features;

[0116] Extracting device information features, geographic location features, and operation behavior sequences from the stream processing results to construct a user behavior profile as a behavior feature;

[0117] Analyze the social network analysis data, associated account patterns, and abnormal behavior identification results in the stream processing results as associated features based on a graph algorithm;

[0118] Extracting statistics and distribution characteristics of each dimension in the stream processing result as statistical features;

[0119] Obtaining derived features by performing feature crossover and feature transformation on the stream processing results;

[0120] Determining the importance of each of the time series features, the behavior features, the association features, the statistical features, and the derived features through information gain, and selecting a first feature from the time series features, the behavior features, the association features, the statistical features, and the derived features according to the importance of each feature;

[0121] Performing feature crossover and nonlinear transformation on the first feature to obtain a second feature;

[0122] Performing dimensionality reduction processing on the second feature to obtain a third feature;

[0123] Encoding the third feature according to the configuration format, and concatenating the encoded features into a vector to obtain the multidimensional feature;

[0124] The multidimensional features are persisted to a storage system, and an index of the multidimensional features is constructed.

[0125] Among them, the application frequency and the time interval distribution can be used to identify abnormal behavior patterns with characteristics such as high frequency of application in a short period of time and fluctuating intervals.

[0126] Among them, the abnormal behavior pattern of abnormal fluctuation of amount can be identified through the amount change trend.

[0127] Among them, device anomalies can be identified through the device information characteristics, such as overseas IP application and high-frequency operation of new devices.

[0128] Among them, the geographic location features can be used to capture location anomalies such as remote login applications and short-term cross-regional movements.

[0129] Among them, abnormal operation paths such as skipping key steps and repeated submission can be captured through the operation behavior sequence.

[0130] The statistical features may include, but are not limited to, mean, variance, maximum, minimum, skewness, kurtosis, quantiles, etc. These statistical features can be used to capture data distribution anomalies such as excessive variance and abnormal distribution morphology, thereby assisting in identifying outliers.

[0131] Among them, different dimensional features can be combined through feature cross-talk, such as combining device type with geographic location.

[0132] Among them, feature transformation can be used to perform mathematical transformation on the original features, such as multiplying finance by a preset ratio.

[0133] Through feature crossover and feature transformation, high-order nonlinear features can be generated, thereby improving the model's ability to express complex patterns.

[0134] Among them, by determining the importance of each feature, feature redundancy can be reduced (for example, only one highly correlated feature is retained), the model training cost can be reduced, and the model accuracy can be improved (by removing noise features).

[0135] Among them, through feature intersection and nonlinear transformation, the nonlinear relationship of the first feature (such as the combined risk of "device type + region") can be explored, thereby improving the feature expression.

[0136] Among them, by performing dimensionality reduction processing on the second feature, the dimensionality curse (such as high-dimensional features leading to model overfitting and slow calculation) can be solved, thereby improving the model training efficiency.

[0137] Among them, by encoding the third feature according to the configuration format and splicing the encoded features into a vector, a standardized input can be generated to adapt to the input requirements of the machine learning model.

[0138] Among them, persisting the multidimensional features to a storage system and constructing an index of the multidimensional features can assist in rapid retrieval of historical features and support feature reuse.

[0139] S14, inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result.

[0140] In this embodiment, inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result includes:

[0141] Dynamically loading model parameters of each sub-model in the fusion model from the model repository; wherein the sub-models include a random forest model, an Xgboost (eXtreme Gradient Boosting) model, and a long short-term memory (LSTM) network;

[0142] Using each sub-model to process the multi-dimensional features respectively to obtain a processing result of each sub-model;

[0143] Get the weight of each sub-model;

[0144] Perform weighted calculation based on the processing results of each sub-model and the weight of each sub-model to obtain the outlier value;

[0145] When the abnormal value is greater than a high-risk threshold, marking the multi-dimensional financial data as high-risk and generating a high-priority alert; or

[0146] When the abnormal value is greater than the medium risk threshold and less than or equal to the high risk threshold, marking the multi-dimensional financial data as medium risk and generating a medium priority alert; or

[0147] When the abnormal value is greater than a low-risk threshold and less than or equal to a medium-risk threshold, marking the multi-dimensional financial data as low-risk and generating a low-priority alert; or

[0148] When the abnormal value is less than or equal to the low risk threshold, marking the multi-dimensional financial data as normal;

[0149] The marked risk levels and generated alerts are integrated to obtain the initial anomaly detection results.

[0150] Among them, the weight of each sub-model can be adjusted according to historical detection accuracy, business scenario adaptability, etc.

[0151] The high risk threshold, the medium risk threshold, and the low risk threshold can be configured based on experiments.

[0152] Through the above embodiments, different levels of warnings can be performed in combination with the abnormal values ​​obtained by the machine learning model, so that differentiated treatments can be taken for different risk levels.

[0153] Through multi-dimensional feature extraction and multi-model fusion, this embodiment can improve the accuracy of anomaly detection by more than 20% and reduce the false alarm rate by 30%.

[0154] S15: Generate an abnormality response strategy according to the initial abnormality detection result.

[0155] In this embodiment, generating an abnormality response strategy according to the initial abnormality detection result includes:

[0156] Performing similarity merging on the initial anomaly detection results according to an aggregation strategy to obtain aggregated data;

[0157] detecting whether the aggregated data requires a real-time response;

[0158] When the aggregated data requires real-time response, the aggregated data is pushed to the alarm system through the Kafka message queue;

[0159] When the aggregated data corresponds to the high-priority alarm, determining that the abnormal response strategy is to control the alarm system to suspend the business process corresponding to the multi-dimensional financial data and send a review application to the expert platform; or

[0160] When the aggregated data corresponds to the medium priority alarm, determining that the abnormal response strategy is to control the alarm system to trigger a secondary check of the aggregated data and add the aggregated data to a manual review queue; or

[0161] When the aggregated data corresponds to the low-priority alarm, the abnormal response strategy is determined to control the alarm system to mark the aggregated data as data to be observed, and add the aggregated data to a batch review queue.

[0162] Among them, the initial anomaly detection results can be similarly merged using aggregation algorithms such as sliding time window aggregation and similarity clustering based on the applicant ID, anomaly type, etc., thereby reducing invalid alarms, reducing the pressure of manual handling, and focusing on truly critical anomalies.

[0163] Among them, whether a real-time response is required can be determined based on the type of anomaly, risk level, etc., so as to distinguish the response priority and ensure that high-risk events are "intercepted in seconds" and low-risk events are "handled in order."

[0164] Among them, pushing exception information through the Kafka message queue can support millisecond-level response.

[0165] Among them, for the high-priority alarm, by suspending the business process corresponding to the multi-dimensional financial data, abnormal behavior can be quickly blocked, thereby reducing losses. The review of the expert platform can also reduce the misjudgment rate.

[0166] Among them, for the medium-priority alarms, combining secondary verification with lightweight verification and manual review can balance user experience and risk prevention and control.

[0167] Among them, for the low-priority alarms, by marking them as data to be observed and adding them to the batch review queue, excessive intervention in low-risk events can be avoided, operating costs can be reduced, and continuous monitoring can prevent the evolution of risks.

[0168] In this embodiment, a visual review interface can also be provided to support manual intervention and annotation, thereby forming a closed-loop feedback.

[0169] The multi-level early warning mechanism combined with the manual review interface has increased the efficiency of exception handling by 50% and significantly reduced the cost of manual review.

[0170] S16: Send the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation.

[0171] In this embodiment, the designated platform may be a platform for reviewing anomaly detection results and anomaly response strategies to improve the accuracy of the results.

[0172] S17: Generate a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform.

[0173] In this embodiment, generating a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform includes:

[0174] When the feedback data shows that the target anomaly detection result is a true anomaly, the initial anomaly detection result is determined as the target anomaly detection result, and the anomaly response strategy is determined as the target anomaly response strategy; wherein the multi-dimensional financial data, the target anomaly detection result, and the target anomaly response strategy are added as cases to a knowledge base, and the business rules corresponding to the multi-dimensional financial data are adjusted; or

[0175] When the feedback data shows that the target anomaly detection result is a false anomaly, the initial anomaly detection result is marked as a false positive.

[0176] In the above embodiment, when it is a true anomaly, it means that the detection result is accurate. At this time, the detection result and related data are added to the knowledge base as a case, which can be easily called and reused; when it is a false anomaly, it can be directly marked as a false alarm for timely correction.

[0177] In this embodiment, the method further includes:

[0178] At every preset period, or when the performance indicator of the fusion model is lower than the performance threshold, or when the cumulative new sample volume reaches the configured sample volume, historical samples are obtained, and new cases are loaded from the knowledge base to construct a full sample volume, and the full sample volume is determined as a training sample; or

[0179] When the incremental data captured in real time based on the Flink ProcessFunction meets a preset condition, the incremental data is determined as the training sample;

[0180] Configuring initial weights for each feature in the training sample based on feature importance analysis;

[0181] Using the SMOTE oversampling algorithm to generate synthetic samples based on the minority class in the training samples, and adding the synthetic samples to the training samples;

[0182] Utilizing the supplemented training samples to optimize and train the fusion model to obtain a new model;

[0183] Performing model evaluation and parameter optimization on the new model;

[0184] The optimized new model is deployed, and the model parameters of the optimized new model are recorded in the model warehouse.

[0185] The preset period may be 0:00 every day or every Monday. Starting the update at a fixed period (such as 0:00 every day or every Monday) is suitable for scenarios where data changes slowly.

[0186] The performance threshold may include an accuracy threshold, a precision threshold, etc.

[0187] The sample size can be configured according to actual needs.

[0188] The preset condition can include the requirement that the incremental data captured in real time by Flink ProcessFunction must be 100 new entries within an hour. Flink ProcessFunction supports incremental learning and real-time updates of model parameters. Flink's incremental learning solution reduces model update time from hours to minutes, significantly improving the model's real-time performance.

[0189] Among them, configuring initial weights for each feature in the training sample based on feature importance analysis can allow the model to focus on key features, thereby reducing redundant feature interference and improving training efficiency and accuracy.

[0190] Among them, using the SMOTE oversampling algorithm to generate synthetic samples based on the minority class in the training samples, and supplementing the synthetic samples to the training samples can solve the class imbalance problem and improve the detection ability of the minority class.

[0191] Among them, model evaluation and parameter optimization can be achieved through cross-validation and real-time performance indicator comparison, thereby ensuring the high performance and stability of the online model.

[0192] To deploy the optimized new model, a grayscale release combined with online learning hot updates can be used. Grayscale release involves first deploying the new model to a small portion of traffic (e.g., 1% of requests) and verifying that there are no anomalies before expanding the scope. Online learning hot updates involve loading incrementally learned model parameters in real time through Flink ProcessFunction to avoid service interruptions. This deployment approach avoids the risks of a full release, and online learning hot updates ensure "real-time effectiveness of model parameters," ensuring service continuity.

[0193] Particularly, by recording the model parameters of the optimized new model into the model warehouse, model traceability can be supported.

[0194] As can be seen from the above technical solutions, the present invention can obtain multi-dimensional financial data from multiple data sources, and improve the accuracy of anomaly detection by combining multi-dimensional data; call the data interface of the Apache Flink framework to perform real-time preprocessing of multi-dimensional financial data, thereby improving data processing efficiency and data quality; use the Apache Flink framework to perform stream processing and window calculations on the processed data stream, and construct multi-dimensional features, thereby improving detection accuracy; input the multi-dimensional features into a fusion model pre-trained based on a machine learning algorithm, and combine it with feedback data from a specified platform to generate anomaly detection results and response strategies, which can further ensure the accuracy of financial data anomaly detection results.

[0195] like Figure 2 Figure 1 shows a functional block diagram of a preferred embodiment of a financial data anomaly detection device according to the present invention. The financial data anomaly detection device 11 comprises an acquisition unit 110, a preprocessing unit 111, a stream processing unit 112, a construction unit 113, an input unit 114, a generation unit 115, and a sending unit 116. As used herein, a module or unit refers to a series of computer program segments that can be executed by a processor and perform fixed functions, and are stored in a memory. The functions of each module or unit in this embodiment will be described in detail in subsequent embodiments.

[0196] The acquisition unit 110 is configured to acquire multi-dimensional financial data from multiple data sources;

[0197] The preprocessing unit 111 is configured to call the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed;

[0198] The stream processing unit 112 is configured to perform stream processing and window calculation on the data stream to be processed using the Apache Flink framework to obtain a stream processing result;

[0199] The construction unit 113 is configured to construct a multi-dimensional feature according to the stream processing result;

[0200] The input unit 114 is used to input the multi-dimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result;

[0201] The generating unit 115 is configured to generate an abnormality response strategy according to the initial abnormality detection result;

[0202] The sending unit 116 is configured to send the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation;

[0203] The generating unit 115 is further configured to generate a target anomaly detection result and a target anomaly response strategy according to the feedback data from the designated platform.

[0204] As can be seen from the above technical solutions, the present invention can obtain multi-dimensional financial data from multiple data sources, and improve the accuracy of anomaly detection by combining multi-dimensional data; call the data interface of the Apache Flink framework to perform real-time preprocessing of multi-dimensional financial data, thereby improving data processing efficiency and data quality; use the Apache Flink framework to perform stream processing and window calculations on the processed data stream, and construct multi-dimensional features, thereby improving detection accuracy; input the multi-dimensional features into a fusion model pre-trained based on a machine learning algorithm, and combine it with feedback data from a specified platform to generate anomaly detection results and response strategies, which can further ensure the accuracy of financial data anomaly detection results.

[0205] like Figure 3 FIG. 1 is a schematic diagram of the structure of a computer device according to a preferred embodiment of the present invention for implementing the method for detecting anomalies in financial data.

[0206] The computer device 1 may include a memory 12, a processor 13 and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a financial data anomaly detection program.

[0207] Those skilled in the art will understand that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may have either a bus structure or a star structure. The computer device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0208] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the scope of protection of the present invention and included here by reference.

[0209] The memory 12 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the computer device 1. Furthermore, the memory 12 may include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can be used not only to store application software and various types of data installed in the computer device 1, such as the code of a financial data anomaly detection program, but also to temporarily store data that has been output or is about to be output.

[0210] In some embodiments, the processor 13 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a combination of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, connecting the various components of the entire computer device 1 using various interfaces and circuits. It executes programs or modules stored in the memory 12 (e.g., a financial data anomaly detection program) and accesses data stored in the memory 12 to perform various functions and process data.

[0211] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned various financial data anomaly detection method embodiments, such as Figure 1 Steps shown.

[0212] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a preprocessing unit 111, a stream processing unit 112, a construction unit 113, an input unit 114, a generation unit 115, and a sending unit 116.

[0213] The integrated unit implemented as a software functional module can be stored in a computer-readable storage medium. The software functional module, stored in a storage medium, includes instructions for causing a computer device (which can be a personal computer, computer equipment, or network equipment, etc.) or a processor to execute portions of the financial data anomaly detection method described in various embodiments of the present invention.

[0214] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments.

[0215] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory, etc.

[0216] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0217] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0218] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The figure shows that only one straight line is used, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13.

[0219] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power supply may also include one or more DC or AC power supplies, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be detailed here.

[0220] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0221] Optionally, the computer device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the computer device 1 and to display a visual user interface.

[0222] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0223] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0224] Combine Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement a method for detecting anomalies in financial data, and the processor 13 can execute the plurality of instructions to implement:

[0225] Acquire multi-dimensional financial data from multiple data sources;

[0226] Calling the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed;

[0227] Using the Apache Flink framework to perform stream processing and window calculation on the data stream to be processed to obtain a stream processing result;

[0228] Constructing multidimensional features according to the stream processing results;

[0229] Inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result;

[0230] generating an anomaly response strategy based on the initial anomaly detection result;

[0231] Sending the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation;

[0232] Generate target anomaly detection results and target anomaly response strategies based on the feedback data of the designated platform.

[0233] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0234] It should be noted that the data involved in this case were all obtained legally. The software tools or components not produced by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0235] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0236] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0237] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0238] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0239] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0240] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0241] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the present invention may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting anomalies in financial data, characterized in that: The financial data anomaly detection method includes: Acquire multi-dimensional financial data from multiple data sources; Calling the data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed; Using the Apache Flink framework to perform stream processing and window calculation on the data stream to be processed to obtain a stream processing result; Constructing multidimensional features according to the stream processing results; Inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result; generating an anomaly response strategy based on the initial anomaly detection result; Sending the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation; Generate target anomaly detection results and target anomaly response strategies based on the feedback data of the designated platform.

2. The financial data anomaly detection method according to claim 1, wherein: The stream processing and window calculation are performed on the data stream to be processed by using the Apache Flink framework to obtain the stream processing result, which includes: Extracting a timestamp from the data stream to be processed; wherein the timestamp includes a business timestamp and a system processing time; Generate a Watermark based on the timestamp; wherein the Watermark is used to represent the timestamp up to the current Watermark, and all data earlier than the timestamp has arrived; Grouping the data stream to be processed according to the configured grouping strategy using a KeyBy operator to obtain at least one data stream group; Allocate the data in each data stream group to each window according to the configured window size, sliding step size and the timestamp; Triggering the computation of each window according to the Watermark to obtain the stream processing result, and persisting the stream processing result to the state backend using the KeyedState keyed state of the Apache Flink framework; The calculation operation includes calculating the number of applications in each window, calculating the statistical value of the application amount in each window, and detecting the application time pattern to perform pattern-based event sequence detection.

3. The financial data anomaly detection method according to claim 1, wherein: The constructing of multidimensional features according to the stream processing results includes: Extracting application frequency, time interval distribution, and amount change trend from the stream processing results as time series features; Extracting device information features, geographic location features, and operation behavior sequences from the stream processing results to construct a user behavior profile as a behavior feature; Analyze the social network analysis data, associated account patterns, and abnormal behavior identification results in the stream processing results as associated features based on a graph algorithm; Extracting statistics and distribution characteristics of each dimension in the stream processing result as statistical features; Obtaining derived features by performing feature crossover and feature transformation on the stream processing results; Determining the importance of each of the time series features, the behavior features, the association features, the statistical features, and the derived features through information gain, and selecting a first feature from the time series features, the behavior features, the association features, the statistical features, and the derived features according to the importance of each feature; Performing feature crossover and nonlinear transformation on the first feature to obtain a second feature; Performing dimensionality reduction processing on the second feature to obtain a third feature; Encoding the third feature according to the configuration format, and concatenating the encoded features into a vector to obtain the multidimensional feature; The multidimensional features are persisted to a storage system, and an index of the multidimensional features is constructed.

4. The financial data anomaly detection method according to claim 1, wherein: Inputting the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result includes: Dynamically loading model parameters of each sub-model in the fusion model from the model warehouse; wherein the sub-models include a random forest model, an Xgboost model, and a long short-term memory network; Using each sub-model to process the multi-dimensional features respectively to obtain a processing result of each sub-model; Get the weight of each sub-model; Perform weighted calculation based on the processing results of each sub-model and the weight of each sub-model to obtain the outlier value; When the abnormal value is greater than a high-risk threshold, marking the multi-dimensional financial data as high-risk and generating a high-priority alert; or When the abnormal value is greater than the medium risk threshold and less than or equal to the high risk threshold, marking the multi-dimensional financial data as medium risk and generating a medium priority alert; or When the abnormal value is greater than a low-risk threshold and less than or equal to a medium-risk threshold, marking the multi-dimensional financial data as low-risk and generating a low-priority alert; or When the abnormal value is less than or equal to the low risk threshold, marking the multi-dimensional financial data as normal; The marked risk levels and generated alerts are integrated to obtain the initial anomaly detection results.

5. The financial data anomaly detection method according to claim 4, characterized in that: Generating an abnormality response strategy according to the initial abnormality detection result includes: Performing similarity merging on the initial anomaly detection results according to an aggregation strategy to obtain aggregated data; detecting whether the aggregated data requires a real-time response; When the aggregated data requires a real-time response, the aggregated data is pushed to the alarm system via the Kafka message queue; When the aggregated data corresponds to the high-priority alarm, determining that the abnormal response strategy is to control the alarm system to suspend the business process corresponding to the multi-dimensional financial data and send a review application to the expert platform; or When the aggregated data corresponds to the medium priority alarm, determining that the abnormal response strategy is to control the alarm system to trigger a secondary check of the aggregated data and add the aggregated data to a manual review queue; or When the aggregated data corresponds to the low-priority alarm, the abnormal response strategy is determined to control the alarm system to mark the aggregated data as data to be observed, and add the aggregated data to a batch review queue.

6. The financial data anomaly detection method according to claim 4, wherein: Generating a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform includes: When the feedback data shows that the target anomaly detection result is a true anomaly, the initial anomaly detection result is determined as the target anomaly detection result, and the anomaly response strategy is determined as the target anomaly response strategy; wherein the multi-dimensional financial data, the target anomaly detection result, and the target anomaly response strategy are added as cases to a knowledge base, and the business rules corresponding to the multi-dimensional financial data are adjusted; or When the feedback data shows that the target anomaly detection result is a false anomaly, the initial anomaly detection result is marked as a false positive.

7. The financial data anomaly detection method according to claim 6, wherein: The method further comprises: At every preset period, or when the performance indicator of the fusion model is lower than the performance threshold, or when the cumulative new sample volume reaches the configured sample volume, historical samples are obtained, and new cases are loaded from the knowledge base to construct a full sample volume, and the full sample volume is determined as a training sample; or When the incremental data captured in real time based on the Flink ProcessFunction meets a preset condition, the incremental data is determined as the training sample; Configuring initial weights for each feature in the training sample based on feature importance analysis; Using the SMOTE oversampling algorithm to generate synthetic samples based on the minority class in the training samples, and adding the synthetic samples to the training samples; Utilizing the supplemented training samples to optimize and train the fusion model to obtain a new model; Performing model evaluation and parameter optimization on the new model; The optimized new model is deployed, and the model parameters of the optimized new model are recorded in the model warehouse.

8. A financial data anomaly detection device, characterized in that: The financial data anomaly detection device includes: An acquisition unit, used to acquire multi-dimensional financial data from multiple data sources; A preprocessing unit, configured to call a data interface of the Apache Flink framework to perform real-time preprocessing on the multi-dimensional financial data to obtain a data stream to be processed; A stream processing unit, configured to perform stream processing and window calculation on the data stream to be processed using the Apache Flink framework to obtain a stream processing result; A construction unit, configured to construct a multidimensional feature according to the stream processing result; An input unit, configured to input the multidimensional features into a fusion model pre-trained based on a machine learning algorithm to obtain an initial anomaly detection result; a generating unit, configured to generate an anomaly response strategy according to the initial anomaly detection result; A sending unit, configured to send the initial anomaly detection result and the anomaly response strategy to a designated platform for confirmation; The generating unit is further configured to generate a target anomaly detection result and a target anomaly response strategy based on the feedback data of the designated platform.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the financial data anomaly detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the financial data anomaly detection method according to any one of claims 1 to 7.

Images