Method and device for detecting discrete points in bank data, medium and bank system

Abstract

The invention provides a method and a device for detecting discrete points in bank data, a medium and a bank system. The method comprises the following steps: acquiring formatted data, wherein the formatted data is data obtained by formatting user behavior log data in bank data; sequentially carrying out classification processing and assignment processing on the formatted data by adopting a ClickHouse column storage technology to obtain feature information; and at least vectorizing the feature information to obtain a user behavior feature vector, processing the user behavior feature vector by adopting a multi-dimensional aggregation function to obtain a final feature score, and determining whether the user behavior log data corresponding to the feature score is a discrete point or not according to the size of the final feature score. The problem that the efficiency of detecting the abnormal behavior of the user behavior log in the bank data is low due to the fact that an existing scheme is difficult to deal with the scene of mass data such as the bank data is solved.

Description

Technical Field

[0001] The present application relates to the technical field of bank data processing, and in particular to a method for detecting discrete points in bank data, a device for detecting discrete points in bank data, a computer-readable storage medium, and a bank system. Background Art

[0002] Currently, abnormal behavior detection in user behavior logs of bank data is mostly asynchronous offline detection with small data volumes. Traditional outlier detection focuses on describing methods without considering implementation scenarios. That is, general methods use statistical methods to fit data using standard distribution models in statistics to detect outliers. The biggest drawback of this method is that it cannot perform large-scale parallel computing when facing massive data. Existing solutions are difficult to cope with scenarios with massive data such as bank data, resulting in low efficiency in detecting abnormal behavior in user behavior logs of bank data. Summary of the Invention

[0003] The main purpose of this application is to provide a method for detecting discrete points in bank data, a device for detecting discrete points in bank data, a computer-readable storage medium and a bank system, so as to at least solve the problem that existing solutions are difficult to cope with scenarios such as massive data such as bank data, resulting in low efficiency in detecting abnormal behaviors in user behavior logs in bank data.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for detecting discrete points in bank data is provided, the method comprising: obtaining formatted data, wherein the formatted data is data obtained by formatting user behavior log data in the bank data; using ClickHouse's column storage technology to perform classification processing and assignment processing on the formatted data in sequence to obtain feature information; at least vectorizing the feature information to obtain a user behavior feature vector, and using a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and according to the size of the final feature score, determining whether the user behavior log data corresponding to the feature score is a discrete point.

[0005] Furthermore, the column storage technology of ClickHouse is used to classify and assign values ​​to the formatted data in sequence to obtain feature information, including: importing the formatted data into a basic log table, and using the column storage technology of ClickHouse to classify all the formatted data in the basic log table according to preset classification rules and then perform column storage, wherein the common features of the column-stored data include the number of user sessions, the average number of pages visited by users, the success rate of the user interface and the average stay time of users; according to the column storage results of the formatted data, the features of the formatted data are assigned values ​​to obtain the feature information.

[0006] Furthermore, a multi-dimensional aggregation function is used to process the user behavior feature vector to obtain a final feature score, including: applying the SQL query in the ClickHouse to obtain the mean and standard deviation of the user behavior feature vector; using the Z-Score formula to calculate the feature score of the user behavior feature vector; and performing weighted averaging on all the feature scores to obtain the final feature score.

[0007] Furthermore, the Z-Score formula is used to calculate the feature score of the user behavior feature vector, including: determining the feature score according to Z = (xi-mu) / sigma; wherein Z is the feature score, mu is the mean of the user behavior feature vector, sigma is the standard deviation, and xi is the actual observation value of the i-th user behavior feature vector.

[0008] Furthermore, after determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score, the method also includes: updating the aggregation result based on the new data using the materialized view of ClickHouse's column storage technology; recalculating the weight of each feature based on the updated materialized view data; analyzing the difference between the new data and the historical data on each feature to adjust the weight according to the degree of difference; and applying the updated feature weight to the multidimensional aggregation function to adjust the core parameters in the multidimensional aggregation function.

[0009] Furthermore, obtaining formatted data includes: monitoring and collecting the user behavior log data in the bank data in real time through Filebeat, and converting the user behavior log data into the user behavior log data in streaming data format; sending the user behavior log data in the streaming data format to the Kafka message queue; using a message processing engine to read the user behavior log data in the streaming data format from the Kafka message queue, and performing data formatting and cutting processing to obtain the formatted data to separate the fields related to user behavior, and the data formatting processing includes data parsing and data type conversion.

[0010] Furthermore, data formatting and cutting processing are performed to obtain the formatted data, including: decoding the user behavior log data in each of the stream data formats, parsing the decoded user behavior log data, extracting key fields to remove irrelevant or erroneous data entries; converting the parsed data into a predefined structured format, and cutting the data in the structured format to obtain the formatted data.

[0011] According to another aspect of the present application, a device for detecting discrete points in bank data is provided, comprising: an acquisition unit for acquiring formatted data, wherein the formatted data is data obtained by formatting user behavior log data in the bank data; a first processing unit for using ClickHouse's column storage technology to perform classification processing and assignment processing on the formatted data in sequence to obtain feature information; a second processing unit for at least vectorizing the feature information to obtain a user behavior feature vector, and using a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0013] According to another aspect of the present application, a banking system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0014] By applying the technical solution of the present application, massive data is first classified and identified through ClickHouse's column storage technology, and then assigned values, so as to cope with massive data scenarios, and then discrete points are identified based on multi-dimensional aggregation functions, thereby improving the efficiency of detecting abnormal behaviors in user behavior logs in bank data, thereby solving the problem that existing solutions are difficult to cope with massive data scenarios such as bank data, resulting in low efficiency of existing solutions in detecting abnormal behaviors in user behavior logs in bank data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 A schematic flow chart of a method for detecting discrete points in bank data provided according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing the principle of a method for detecting discrete points in bank data provided according to an embodiment of the present application is shown;

[0018] Figure 3 A structural block diagram of a device for detecting discrete points in bank data provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] As introduced in the background technology, currently, the abnormal behavior detection for user behavior logs in bank data is mostly asynchronous offline detection with small data volumes. Traditional outlier detection mainly focuses on describing the method without considering the implementation scenario. That is to say, the general method is to detect outliers by fitting the data with the standard distribution model in statistics based on statistical methods. The biggest disadvantage of this method is that when facing massive data, it is impossible to perform large-scale parallel computing. The existing solutions are difficult to cope with the scenario of massive data such as bank data, resulting in low efficiency of the existing solutions in detecting abnormal behavior in user behavior logs in bank data. In order to solve the problem that the existing solutions are difficult to cope with the scenario of massive data such as bank data, resulting in low efficiency of the existing solutions in detecting abnormal behavior in user behavior logs in bank data, the embodiments of the present application provide a method for detecting discrete points in bank data, a device for detecting discrete points in bank data, a computer-readable storage medium and a bank system.

[0023] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] In this embodiment, a method for detecting discrete points in bank data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] Figure 1 FIG. 1 is a flow chart of a method for detecting discrete points in bank data according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0026] Step S101, obtaining formatted data, wherein the formatted data is data obtained by formatting user behavior log data in bank data;

[0027] Step S102: Use ClickHouse's column storage technology to perform classification and assignment processing on the formatted data to obtain feature information.

[0028] Step S103, at least vectorize the above-mentioned feature information to obtain a user behavior feature vector, and use a multi-dimensional aggregation function to process the above-mentioned user behavior feature vector to obtain a final feature score, and determine whether the above-mentioned user behavior log data corresponding to the above-mentioned feature score is a discrete point based on the size of the above-mentioned final feature score.

[0029] At least in the process of vectorizing the feature information, when multiple parameters are needed, the multiple parameters are weightedly summed for feature information of the same type to obtain a user behavior feature vector.

[0030] In the above steps, the massive data is first classified and identified through ClickHouse's column storage technology, and then assigned values, so as to cope with massive data scenarios. Then, discrete points are identified based on multi-dimensional aggregation functions, which improves the efficiency of detecting abnormal behaviors in user behavior logs in bank data. This solves the problem that existing solutions are difficult to cope with massive data scenarios such as bank data, resulting in low efficiency in detecting abnormal behaviors in user behavior logs in bank data.

[0031] Using ClickHouse's columnar storage, it is possible to achieve statistical results in seconds on data sets of billions; through the incremental information processing module, it achieves real-time dynamic updates of user behavior data; through feature extraction and scoring mechanisms, it can accurately capture and identify outliers in user behavior; it supports operation in a distributed environment and is suitable for large-scale user behavior monitoring scenarios.

[0032] Among them, in step S102, the column storage technology of ClickHouse is used to classify and assign values ​​to the above-mentioned formatted data in sequence to obtain feature information, including: importing the above-mentioned formatted data into the basic log table, and using the column storage technology of ClickHouse to classify all the above-mentioned formatted data in the above-mentioned basic log table according to preset classification rules and then perform column storage, wherein the common characteristics of the column-stored data include the number of user sessions, the average number of pages visited by users, the success rate of the user interface and the average stay time of users; according to the column storage results of the above-mentioned formatted data, the characteristics of the above-mentioned formatted data are assigned values ​​to obtain the above-mentioned feature information.

[0033] Column-based storage technology allows ClickHouse to store data on a column-by-column basis, meaning that values ​​of the same type (such as the number of sessions for all users) are stored in the same physical location. This storage method significantly improves data compression and storage efficiency, while reducing disk I / O operations and speeding up data queries, enabling statistical analysis of even large amounts of data in a short period of time. Column-based storage makes aggregation operations (such as averages and counts) particularly efficient because it avoids unnecessary data reads. ClickHouse's built-in aggregation functions, such as AVG(), COUNT(), and SUM(), leverage the characteristics of columns directly at the storage level to quickly calculate results, which is extremely beneficial for real-time monitoring and outlier detection. ClickHouse supports distributed deployment and can scale horizontally to handle larger data volumes. This means that as daily active users and transaction volume for mobile banking grow, the system can easily adapt by adding nodes without changing the existing data storage or processing logic, ensuring system stability and performance in the face of data expansion. Because ClickHouse's column-based storage allows for efficient processing of all data in a column, the calculation of characteristic values ​​(such as average user dwell time) becomes extremely fast. This means that we can generate user feature information in real time, update the materialized view of user behavior in a timely manner, and provide the latest data for outlier detection; using the Z-Score algorithm to score feature information in real time can quickly identify abnormal behavior. Columnar storage technology makes data query and calculation almost instantaneous. Even in the face of hundreds of millions of transaction data, it can achieve real-time outlier detection, meeting the financial industry's high real-time requirements for security and risk control; by directly extracting and scoring features in ClickHouse, the need for data transmission between systems is reduced, the complexity and latency of the data processing pipeline are reduced, and the overall system efficiency is improved; because ClickHouse's columnar storage can more effectively utilize storage space and computing resources, compared with traditional row-based storage databases, it can significantly reduce hardware costs and operation and maintenance costs, while increasing the speed of data processing and analysis.

[0034] Among them, a multi-dimensional aggregation function is used to process the above-mentioned user behavior feature vector to obtain the final feature score, including: applying the SQL query in the above-mentioned ClickHouse to obtain the mean and standard deviation of the above-mentioned user behavior feature vector; using the Z-Score formula to calculate the feature score of the above-mentioned user behavior feature vector; and performing weighted average processing on all the above-mentioned feature scores to obtain the above-mentioned final feature score.

[0035] ClickHouse's high-speed query capability combined with the flexibility of SQL can quickly calculate statistical data on user behavior characteristics, and then calculate the Z-Score value to achieve real-time outlier detection; the Z-Score formula can standardize the scores of various features, making it possible to compare features of different scales, improving the accuracy of the scores, and ensuring the reliability of anomaly detection; weighted average processing allows the scores to be adjusted according to the importance of the features, ensuring that the system pays more attention to those features that have a greater impact on abnormal behavior detection, improving the pertinence and effectiveness of detection; ClickHouse's columnar storage and query optimization mechanism makes it possible to perform complex calculations on large-scale data sets without consuming a lot of computing resources, thereby improving the resource efficiency of the system; through the combination of SQL queries and Z-Score calculations, the system can quickly respond to new data streams, instantly update users' feature scores, and improve the response speed of anomaly detection.

[0036] In addition, the Z-Score formula is used to calculate the feature score of the above-mentioned user behavior feature vector, including: determining the above-mentioned feature score according to Z = (xi-mu) / sigma; wherein Z is the above-mentioned feature score, mu is the mean of the above-mentioned user behavior feature vector, sigma is the above-mentioned standard deviation, and xi is the actual observation value of the i-th above-mentioned user behavior feature vector.

[0037] In the scenario of large-scale real-time data monitoring and anomaly detection: Z-Score provides a general statistical method to measure the deviation between each feature value and the mean of the data set. By dividing this deviation by the standard deviation, Z-Score can convert the values ​​of different features to the same scale, so that anomaly detection does not depend on the original scale of the feature. Because Z-Score transforms the data to the scale of a standard normal distribution, a unified threshold (for example, an absolute value greater than 3) can be used to determine which feature value is abnormal. This simplifies the rules for anomaly detection and allows the system to maintain consistency and predictability when detecting anomalies on any feature. Calculating Z-Score in real time in ClickHouse can promptly capture changes in user behavior, especially the emergence of abnormal behavior. This real-time performance is critical for security systems that require immediate response, such as being able to take immediate action when potential fraud is detected.

[0038] In one embodiment of the present application, after determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score, the method further includes: updating the aggregation result based on the new data using the materialized view of ClickHouse's column storage technology; recalculating the weight of each feature based on the updated materialized view data; analyzing the difference between the new data and the historical data on each feature to adjust the weight according to the degree of difference; and applying the updated feature weight to the multidimensional aggregation function to adjust the core parameters in the multidimensional aggregation function.

[0039] In mobile banking scenarios, real-time abnormal transaction detection is a crucial security measure. Daily user transactions, including logins, transfers, payments, and inquiries, generate massive amounts of log data. This data contains user behavioral characteristics, such as transaction frequency, transaction time, transaction amount, and login location. Traditional outlier detection methods struggle to adapt to these real-time and data volume requirements. Specific use cases for adjusting the core parameters in the aforementioned multi-dimensional aggregation functions include: Using ClickHouse's columnar storage: Mobile banking log data is first imported into the ClickHouse database, where its columnar storage technology is used to store various dimensions of user behavior (such as transaction amount and frequency). The advantage of columnar storage lies in its ability to efficiently process large amounts of data, particularly for aggregation and statistical analysis, significantly reducing query time. Materialized views are created in ClickHouse, which automatically calculate and store key statistical metrics, such as the average transaction amount and number of transactions per user. When new data (such as new transaction records) enter the database, the materialized views are automatically updated, ensuring that the aggregation results reflect the latest data status. Regarding updating aggregated results based on new data: For example, suppose mobile banking transaction volume suddenly increases on a certain day, perhaps due to a promotion or holiday. The materialized view will automatically capture this increase in transaction frequency and update the corresponding statistical indicators.

[0040] The system then recalculates the weight of each feature based on updated statistics such as transaction frequency and transaction amount. For example, if the increase in transaction frequency is significantly greater than the change in transaction amount, the weight of transaction frequency may be increased accordingly in multi-dimensional outlier detection. By comparing the distribution of new data with historical data, the system can identify which features have significantly changed in their statistical properties and adjust the weights accordingly to ensure that the scoring algorithm can better adapt to current user behavior patterns. The system will adjust the core parameters of the multi-dimensional aggregation function based on the updated weights. For example, when calculating the Z-Score, the weights of different features will affect the score calculation, thereby more accurately identifying abnormal behavior.

[0041] Adjusting the core parameters of the aforementioned multi-dimensional aggregation functions offers the following benefits: Due to the real-time update nature of ClickHouse's materialized views, the system can instantly respond to changes in transaction behavior and implement real-time anomaly detection, which is crucial for promptly detecting and preventing potential fraudulent activity. The dynamic weight adjustment mechanism enables the scoring algorithm to automatically adapt to seasonal variations in user behavior, holiday effects, and other factors, maintaining accurate anomaly detection and maintaining stable performance even in environments with rapidly changing data distributions. ClickHouse's columnar storage and efficient aggregation functions significantly reduce the time it takes to calculate statistical metrics, enabling real-time analysis of large-scale data. By continuously updating feature weights, the system can more accurately identify behaviors that truly deviate from the norm, reducing false positives and avoiding unnecessary interference with legitimate user transactions. Dynamic weight adjustment reduces unnecessary computational resource waste by prioritizing features that are more critical for anomaly detection and assigning less weight to features that have little change or impact. Automated updates and weight adjustments reduce the need for manual intervention, reducing the complexity and cost of system operations and maintenance, allowing security teams to focus more on analyzing and addressing anomalies rather than maintaining the system.

[0042] In one embodiment of the present application, obtaining formatted data includes: monitoring and collecting the above-mentioned user behavior log data in the bank data in real time through Filebeat, and converting the above-mentioned user behavior log data into the above-mentioned user behavior log data in the streaming data format; sending the above-mentioned user behavior log data in the above-mentioned streaming data format to the Kafka message queue; using a message processing engine to read the above-mentioned user behavior log data in the above-mentioned streaming data format from the above-mentioned Kafka message queue, and performing data formatting and cutting processing to obtain the above-mentioned formatted data to separate the fields related to user behavior, and the above-mentioned data formatting processing includes data parsing and data type conversion.

[0043] Filebeat monitors and collects user behavior log data in real time, ensuring data freshness and continuity. This is crucial for anomaly detection, as any delay can hinder immediate response to anomalous behavior. Filebeat converts log data into a streaming data format, facilitating subsequent processing by real-time analytics systems. This streaming data format seamlessly integrates with stream processing platforms like Kafka, streamlining data transmission and processing. The Kafka message queue acts as a data buffer, ensuring stable data transmission in high-concurrency and high-volume scenarios. Kafka also supports persistent data storage, ensuring data is not lost even in the event of system failures. The message processing engine reads streaming data from Kafka and performs data formatting and segmentation. This process isolates fields related to user behavior, facilitating subsequent outlier detection analysis. Data parsing and data type conversion ensure that all data is processed in a unified and standardized format, significantly improving data processing efficiency and accuracy. The combination of Filebeat and Kafka significantly reduces the time delay between data generation and data availability, enabling real-time monitoring and anomaly detection systems to promptly detect and respond to anomalous behavior. The layered design of data collection, transmission, and preprocessing allows each component to be independently optimized and expanded, while also improving the flexibility and maintainability of the system. For example, Filebeat can focus on efficient data collection without having to worry about the details of data transmission or preprocessing; Kafka focuses on data buffering and transmission to ensure high availability and persistence of data; the message processing engine focuses on data formatting and preprocessing to provide clean, structured data for subsequent analysis. Through stream data processing, the system is able to process large amounts of data with limited resources. Real-time transmission and processing of data avoids the storage of large amounts of data in memory, thereby reducing resource consumption. The preprocessed data has been cut and formatted and can be used directly as input to the outlier detection algorithm, or stored in high-speed databases such as ClickHouse for fast query and analysis.

[0044] In one embodiment of the present application, data formatting and cutting processing are performed to obtain the above-mentioned formatted data, including: decoding the above-mentioned user behavior log data in each of the above-mentioned stream data formats, parsing the decoded above-mentioned user behavior log data, extracting key fields to remove irrelevant or erroneous data entries; converting the parsed data into a predefined structured format, and cutting the data in the structured format to obtain the above-mentioned formatted data.

[0045] The decoding and parsing process is designed to remove irrelevant, redundant, or erroneous data entries, ensuring data accuracy for subsequent analysis. Through in-depth decoding and parsing of the data, the system can identify and filter out anomalous or incomplete records, preventing these from negatively impacting analytical results. By parsing user behavior log data, the system accurately extracts key fields related to user behavior, such as user ID, transaction type, transaction amount, and timestamp. These fields are crucial for constructing user behavior feature vectors and provide effective information for outlier detection. Converting the parsed data into a predefined structured format helps ensure that all data is stored and processed in a consistent format. This reduces complexity in subsequent data processing and improves efficiency and accuracy. Slicing structured data—breaking it down into a format that's easier to analyze—removes unnecessary data early in the processing process, reducing the amount of data required in subsequent stages and improving overall processing efficiency. The use of formatted data simplifies outlier detection and other data analysis processes. The predefined structured format ensures data consistency, allowing analytical algorithms to focus more on the data's inherent patterns rather than differences in data formats. Decoding user behavior log data from various streaming formats, along with subsequent parsing and segmentation, enhances the system's robustness. Even in the face of diverse data formats or fluctuations in data quality, the system can still reliably extract valid information and perform analysis. Formatted data is the cornerstone of advanced analytics, such as data analysis, machine learning, and outlier detection. Only after data is correctly parsed and formatted can these analytical methods be effectively applied to uncover valuable information and insights. Parsing and removing irrelevant data entries reduces the exposure of sensitive data and enhances data security. The system retains and processes only necessary information, reducing the risk of data leakage.

[0046] Alternative Data Storage and Processing Options: Column-oriented databases like Apache Druid or Google BigQuery can also achieve high-speed data processing capabilities similar to ClickHouse. A combination of Hadoop HDFS and Apache Spark: Use HDFS to store large amounts of data and Spark for distributed data processing and real-time analytics. Time series databases (such as InfluxDB or TimescaleDB): These databases may be suitable alternatives for applications that require processing time series data. Reasons for Alternative Options: These alternative technologies can also handle large amounts of data, support fast queries and analysis, and offer similar efficiency and scalability.

[0047] Alternatives to feature extraction and materialized views: Using deep learning or machine learning models, such as LSTM-based and autoencoder-based anomaly detection models, can extract complex features and perform outlier detection without relying on materialized views. Using graph databases (such as Neo4j): In some application scenarios, graph databases can also replace materialized views for correlation analysis and feature extraction. Reasons for this alternative: Deep learning and graph databases can handle more complex relationships and time series information, making them suitable for scenarios with high feature extraction requirements.

[0048] Alternatives to Outlier Detection Algorithms: Machine Learning-Based Outlier Detection: Replace the z-score detection algorithm with methods such as support vector machine (SVM)-based outlier detection (One-Class SVM), isolation forest, and local outlier factor (LOF). In some scenarios, outlier detection may require predefined rules and conditions rather than statistical methods. Reasons for this alternative: Machine learning models can handle more complex patterns and outliers, while rule-based systems can provide more specific detection logic in specific scenarios.

[0049] like Figure 2 As shown in the figure, the above method essentially involves the following steps: User behavior flow information in monitoring logs is stored in ClickHouse via a messaging engine. Columnar storage is used to process data, enabling instant statistical results for billions of data points. Statistical calculations are performed on billions of data points. User behavior information is extracted through feature extraction and quantitative views, and the extracted features are scored. User behavior is then vectorized, ultimately achieving feature-based data. Furthermore, an incremental information indicator update module is implemented to dynamically update user indicator vectors for incremental user behavior.

[0050] The monitoring log includes a data access unit that accesses and processes user behavior log data through the messaging engine. Based on the business logs, Filebeat collects system monitoring logs as streaming data and sends them to the Kafka message queue. The message processing engine then accesses the data and processes it into formatted data to support subsequent steps.

[0051] ClickHouse's column-based storage includes data storage units. Based on ClickHouse's column-based storage, it stores and manages large-scale user behavior data. Materialized views are used to cut data into structured data. At the same time, the ClickHouse column-based storage structure is used to store data in columns, making it possible for subsequent data processing and calculation.

[0052] User feature information includes: Feature extraction unit: extracts key features from stored data and generates corresponding materialized views. User features are defined in multiple dimensions based on the nine major mobile banking service modules (public services, digital currency 521 services, life services, financial services, user services, account services, loan services, credit card services, and insurance services). They represent user behavior information at nine service levels. The characteristics of each service module are divided into two parts. The common part is: the number of user sessions, the average number of pages visited by users, the user interface success rate, and the average user stay time. The specific part is: the frequency of access to the core interface of each service module, the frequency of access to the non-core interface of each service module, and the vulnerability conversion rate of each service module in individual scenarios.

[0053] The feature extraction quantization view includes an outlier detection unit that scores and vectorizes feature information to generate a user behavior feature vector. Because features have both common and individual components, the individual components are weighted and summed to form a uniform feature vector of the same dimension, facilitating unified calculation and extraction.

[0054] Incremental information indicator update: The existing classification center indicator is calculated based on the existing data. Moreover, the classification center is not static as new data is added. Incremental update of the classification center vector: Process the newly added user behavior data and update the existing user feature vectors by dynamic weighted summation.

[0055] Aggregation function output: The extracted feature information is scored and user behavior is vectorized using an aggregation function (such as the z-score). By setting a threshold, when the feature score exceeds the threshold, it is considered an outlier or abnormal behavior. After defining the features in part (3), the multi-dimensional Z-Score algorithm with weighted average can improve the accuracy of outlier detection and reduce the system's computational overhead.

[0056] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for detecting discrete points in bank data of the present application will be described in detail below with reference to specific embodiments.

[0057] This embodiment relates to a specific method for detecting discrete points in bank data, including:

[0058] Obtaining formatted data, where the formatted data is obtained by formatting the user behavior log data in the bank data;

[0059] Specifically, Filebeat is used to monitor and collect user behavior log data in bank data in real time, and convert the user behavior log data into user behavior log data in streaming data format; the user behavior log data in streaming data format is sent to the Kafka message queue; the message processing engine is used to read the user behavior log data in streaming data format from the Kafka message queue, and decode the user behavior log data in each streaming data format, parse the decoded user behavior log data, extract key fields, and remove irrelevant or erroneous data entries; the parsed data is converted into a predefined structured format, and the structured format data is cut and processed to obtain formatted data;

[0060] ClickHouse's column storage technology is used to classify and assign values ​​to the formatted data to obtain feature information.

[0061] Specifically, the formatted data is imported into the basic log table, and ClickHouse's column storage technology is used to classify all the formatted data in the basic log table according to the preset classification rules and then store them in column format. The common characteristics of the column-stored data include the number of user sessions, the average number of pages visited by users, the success rate of the user interface and the average stay time of users; according to the column storage results of the formatted data, the characteristics of the formatted data are assigned to obtain feature information.

[0062] At least perform vectorization processing on the feature information to obtain a user behavior feature vector, and use a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determine whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score.

[0063] Specifically, we apply SQL queries in ClickHouse to obtain the mean and standard deviation of the user behavior feature vector; use the Z-Score formula to calculate the feature score of the user behavior feature vector; and perform a weighted average of all feature scores to obtain the final feature score. The Z-Score formula is used to calculate the feature score of the user behavior feature vector, including determining the feature score according to Z = (xi - mu) / sigma; where Z is the feature score, mu is the mean of the user behavior feature vector, sigma is the standard deviation, and xi is the actual observed value of the i-th user behavior feature vector.

[0064] After determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score, the materialized view using ClickHouse's column storage technology updates the aggregation result based on the new data; based on the data of the updated materialized view, the weight of each feature is recalculated; the difference between the new data and the historical data on each feature is analyzed to adjust the weight according to the degree of difference; the updated feature weight is applied to the multi-dimensional aggregation function to adjust the core parameters in the multi-dimensional aggregation function.

[0065] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] The embodiments of the present application also provide a device for detecting discrete points in bank data. It should be noted that the device for detecting discrete points in bank data in the embodiments of the present application can be used to execute the method for detecting discrete points in bank data provided in the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation by hardware, or a combination of software and hardware, is also possible and conceivable.

[0067] The following introduces the detection device of discrete points in bank data provided by the embodiment of the present application.

[0068] Figure 3 1 is a structural block diagram of a device for detecting discrete points in bank data according to an embodiment of the present application. Figure 3 As shown, the device includes:

[0069] An acquisition unit 31 is used to acquire formatted data, where the formatted data is obtained by formatting the user behavior log data in the bank data; a first processing unit 32 is used to use ClickHouse's column storage technology to perform classification processing and assignment processing on the formatted data in sequence to obtain feature information; a second processing unit 33 is used to at least vectorize the feature information to obtain a user behavior feature vector, and use a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determine whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score.

[0070] In the above-mentioned device, massive data is first classified and identified through ClickHouse's column storage technology, and then assigned values, so as to cope with massive data scenarios. Then, discrete points are identified based on multi-dimensional aggregation functions, which improves the efficiency of detecting abnormal behaviors in user behavior logs in bank data, thereby solving the problem that existing solutions are difficult to cope with massive data scenarios such as bank data, resulting in low efficiency of existing solutions in detecting abnormal behaviors in user behavior logs in bank data.

[0071] In one embodiment of the present application, the first processing unit includes a first processing module and a second processing module. The first processing module is used to import the above-mentioned formatted data into the basic log table, and use ClickHouse's column storage technology to classify all the above-mentioned formatted data in the above-mentioned basic log table according to preset classification rules and then store them in column format. The common characteristics of the column-stored data include the number of user sessions, the average number of pages visited by users, the success rate of the user interface and the average stay time of users; the second processing module is used to assign values ​​to the characteristics of the above-mentioned formatted data according to the column storage results of the above-mentioned formatted data to obtain the above-mentioned characteristic information.

[0072] Column-based storage technology allows ClickHouse to store data on a column-by-column basis, meaning that values ​​of the same type (such as the number of sessions for all users) are stored in the same physical location. This storage method significantly improves data compression and storage efficiency, while reducing disk I / O operations and speeding up data queries, enabling statistical analysis of even large amounts of data in a short period of time. Column-based storage makes aggregation operations (such as averages and counts) particularly efficient because it avoids unnecessary data reads. ClickHouse's built-in aggregation functions, such as AVG(), COUNT(), and SUM(), leverage the characteristics of columns directly at the storage level to quickly calculate results, which is extremely beneficial for real-time monitoring and outlier detection. ClickHouse supports distributed deployment and can scale horizontally to handle larger data volumes. This means that as daily active users and transaction volume for mobile banking grow, the system can easily adapt by adding nodes without changing the existing data storage or processing logic, ensuring system stability and performance in the face of data expansion. Because ClickHouse's column-based storage allows for efficient processing of all data in a column, the calculation of characteristic values ​​(such as average user dwell time) becomes extremely fast. This means that we can generate user feature information in real time, update the materialized view of user behavior in a timely manner, and provide the latest data for outlier detection; using the Z-Score algorithm to score feature information in real time can quickly identify abnormal behavior. Columnar storage technology makes data query and calculation almost instantaneous. Even in the face of hundreds of millions of transaction data, it can achieve real-time outlier detection, meeting the financial industry's high real-time requirements for security and risk control; by directly extracting and scoring features in ClickHouse, the need for data transmission between systems is reduced, the complexity and latency of the data processing pipeline are reduced, and the overall system efficiency is improved; because ClickHouse's columnar storage can more effectively utilize storage space and computing resources, compared with traditional row-based storage databases, it can significantly reduce hardware costs and operation and maintenance costs, while increasing the speed of data processing and analysis.

[0073] In one embodiment of the present application, the second processing unit includes a third processing module, a fourth processing module and a fifth processing module. The third processing module is used to apply the SQL query in the above-mentioned ClickHouse to obtain the mean and standard deviation of the above-mentioned user behavior feature vector; the fourth processing module is used to use the Z-Score formula to calculate the feature score of the above-mentioned user behavior feature vector; the fifth processing module is used to perform weighted averaging processing on all the above-mentioned feature scores to obtain the above-mentioned final feature score.

[0074] ClickHouse's high-speed query capability combined with the flexibility of SQL can quickly calculate statistical data on user behavior characteristics, and then calculate the Z-Score value to achieve real-time outlier detection; the Z-Score formula can standardize the scores of various features, making it possible to compare features of different scales, improving the accuracy of the scores, and ensuring the reliability of anomaly detection; weighted average processing allows the scores to be adjusted according to the importance of the features, ensuring that the system pays more attention to those features that have a greater impact on abnormal behavior detection, improving the pertinence and effectiveness of detection; ClickHouse's columnar storage and query optimization mechanism makes it possible to perform complex calculations on large-scale data sets without consuming a lot of computing resources, thereby improving the resource efficiency of the system; through the combination of SQL queries and Z-Score calculations, the system can quickly respond to new data streams, instantly update users' feature scores, and improve the response speed of anomaly detection.

[0075] In one embodiment of the present application, the fourth processing module includes a determination submodule for determining the above-mentioned feature score based on Z = (xi-mu) / sigma; wherein Z is the above-mentioned feature score, mu is the mean of the above-mentioned user behavior feature vector, sigma is the above-mentioned standard deviation, and xi is the actual observation value of the i-th above-mentioned user behavior feature vector.

[0076] In the scenario of large-scale real-time data monitoring and anomaly detection: Z-Score provides a general statistical method to measure the deviation between each feature value and the mean of the data set. By dividing this deviation by the standard deviation, Z-Score can convert the values ​​of different features to the same scale, so that anomaly detection does not depend on the original scale of the feature. Because Z-Score transforms the data to the scale of a standard normal distribution, a unified threshold (for example, an absolute value greater than 3) can be used to determine which feature value is abnormal. This simplifies the rules for anomaly detection and allows the system to maintain consistency and predictability when detecting anomalies on any feature. Calculating Z-Score in real time in ClickHouse can promptly capture changes in user behavior, especially the emergence of abnormal behavior. This real-time performance is critical for security systems that require immediate response, such as being able to take immediate action when potential fraud is detected.

[0077] In one embodiment of the present application, the above-mentioned device also includes a third processing unit, a fourth processing unit, a fifth processing unit and a sixth processing unit. The third processing unit is used to determine whether the above-mentioned user behavior log data corresponding to the above-mentioned feature score is a discrete point based on the size of the above-mentioned final feature score, and then use the materialized view of ClickHouse's column storage technology to update the aggregation result according to the new data; the fourth processing unit is used to recalculate the weight of each feature based on the data of the updated materialized view; the fifth processing unit is used to analyze the difference between the above-mentioned new data and historical data in each feature to adjust the weight according to the degree of difference; the sixth processing unit is used to apply the updated feature weight to the multidimensional aggregation function to adjust the core parameters in the above-mentioned multidimensional aggregation function.

[0078] Thanks to the real-time updating nature of ClickHouse's materialized views, the system can instantly respond to changes in transaction behavior and achieve real-time anomaly detection, which is crucial for promptly detecting and preventing potential fraud. The dynamic weight adjustment mechanism enables the scoring algorithm to automatically adapt to seasonal changes in user behavior, holiday effects, and other factors, maintaining the accuracy of anomaly detection and maintaining stable performance even in environments with rapidly changing data distribution. The efficiency of ClickHouse's columnar storage and aggregation functions significantly reduces the time it takes to calculate statistical indicators, making real-time analysis of large-scale data possible. By continuously updating feature weights, the system can more accurately identify which behaviors truly deviate from the normal range, thereby reducing false positives and avoiding unnecessary interference with normal user transactions. Dynamic weight adjustment reduces unnecessary waste of computing resources by placing greater emphasis on features that are more critical for anomaly detection and assigning less weight to features that have little change or impact. Automated updates and weight adjustments reduce the need for manual intervention, reducing the complexity and cost of system operation and maintenance, allowing security teams to focus more on analyzing and addressing abnormal behavior rather than maintaining the system.

[0079] In one embodiment of the present application, the acquisition unit includes a sixth processing module, a seventh processing module and an eighth processing module. The sixth processing module is used to monitor and collect the above-mentioned user behavior log data in the bank data in real time through Filebeat, and convert the above-mentioned user behavior log data into the above-mentioned user behavior log data in the streaming data format; the seventh processing module is used to send the above-mentioned user behavior log data in the above-mentioned streaming data format to the Kafka message queue; the eighth processing module is used to use the message processing engine to read the above-mentioned user behavior log data in the above-mentioned streaming data format from the above-mentioned Kafka message queue, and perform data formatting processing and cutting processing to obtain the above-mentioned formatted data to separate the fields related to user behavior. The above-mentioned data formatting processing includes data parsing and data type conversion.

[0080] Filebeat monitors and collects user behavior log data in real time, ensuring data freshness and continuity. This is crucial for anomaly detection, as any delay can hinder immediate response to anomalous behavior. Filebeat converts log data into a streaming data format, facilitating subsequent processing by real-time analytics systems. This streaming data format seamlessly integrates with stream processing platforms like Kafka, streamlining data transmission and processing. The Kafka message queue acts as a data buffer, ensuring stable data transmission in high-concurrency and high-volume scenarios. Kafka also supports persistent data storage, ensuring data is not lost even in the event of system failures. The message processing engine reads streaming data from Kafka and performs data formatting and segmentation. This process isolates fields related to user behavior, facilitating subsequent outlier detection analysis. Data parsing and data type conversion ensure that all data is processed in a unified and standardized format, significantly improving data processing efficiency and accuracy. The combination of Filebeat and Kafka significantly reduces the time delay between data generation and data availability, enabling real-time monitoring and anomaly detection systems to promptly detect and respond to anomalous behavior. The layered design of data collection, transmission, and preprocessing allows each component to be independently optimized and expanded, while also improving the flexibility and maintainability of the system. For example, Filebeat can focus on efficient data collection without having to worry about the details of data transmission or preprocessing; Kafka focuses on data buffering and transmission to ensure high availability and persistence of data; the message processing engine focuses on data formatting and preprocessing to provide clean, structured data for subsequent analysis. Through stream data processing, the system is able to process large amounts of data with limited resources. Real-time transmission and processing of data avoids the storage of large amounts of data in memory, thereby reducing resource consumption. The preprocessed data has been cut and formatted and can be used directly as input to the outlier detection algorithm, or stored in high-speed databases such as ClickHouse for fast query and analysis.

[0081] In one embodiment of the present application, the eighth processing module includes a first processing sub-module and a second processing sub-module, the first processing sub-module is used to decode the above-mentioned user behavior log data in each of the above-mentioned stream data formats, parse the decoded above-mentioned user behavior log data, extract key fields, and remove irrelevant or erroneous data entries; the eighth processing module is used to convert the parsed data into a predefined structured format, and cut the data in the structured format to obtain the above-mentioned formatted data.

[0082] The decoding and parsing process is designed to remove irrelevant, redundant, or erroneous data entries, ensuring data accuracy for subsequent analysis. Through in-depth decoding and parsing of the data, the system can identify and filter out anomalous or incomplete records, preventing these from negatively impacting analytical results. By parsing user behavior log data, the system accurately extracts key fields related to user behavior, such as user ID, transaction type, transaction amount, and timestamp. These fields are crucial for constructing user behavior feature vectors and provide effective information for outlier detection. Converting the parsed data into a predefined structured format helps ensure that all data is stored and processed in a consistent format. This reduces complexity in subsequent data processing and improves efficiency and accuracy. Slicing structured data—breaking it down into a format that's easier to analyze—removes unnecessary data early in the processing process, reducing the amount of data required in subsequent stages and improving overall processing efficiency. The use of formatted data simplifies outlier detection and other data analysis processes. The predefined structured format ensures data consistency, allowing analytical algorithms to focus more on the data's inherent patterns rather than differences in data formats. Decoding user behavior log data from various streaming formats, along with subsequent parsing and segmentation, enhances the system's robustness. Even in the face of diverse data formats or fluctuations in data quality, the system can still reliably extract valid information and perform analysis. Formatted data is the cornerstone of advanced analytics, such as data analysis, machine learning, and outlier detection. Only after data is correctly parsed and formatted can these analytical methods be effectively applied to uncover valuable information and insights. Parsing and removing irrelevant data entries reduces the exposure of sensitive data and enhances data security. The system retains and processes only necessary information, reducing the risk of data leakage.

[0083] The device for detecting discrete points in bank data includes a processor and a memory. The acquisition unit, first processing unit, and second processing unit are stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. All of the modules are located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0084] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, existing solutions are unable to cope with massive amounts of data, such as banking data, resulting in low efficiency in detecting abnormal behavior in user behavior logs within banking data.

[0085] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is run, the device where the computer-readable storage medium is located is controlled to execute the method for detecting discrete points in bank data.

[0087] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the method for detecting discrete points in the bank data when running.

[0088] An embodiment of the present invention provides a device comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented: obtaining formatted data, wherein the formatted data is data obtained by formatting user behavior log data in bank data; using ClickHouse's columnar storage technology to sequentially classify and assign values ​​to the formatted data to obtain feature information; at least vectorizing the feature information to obtain a user behavior feature vector, and using a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score. The device herein may be a server, a PC, a PAD, a mobile phone, etc.

[0089] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with at least the following method steps: obtaining formatted data, wherein the formatted data is data obtained by formatting user behavior log data in bank data; using ClickHouse's column storage technology to sequentially classify and assign values ​​to the formatted data to obtain feature information; at least vectorizing the feature information to obtain a user behavior feature vector, and using a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determining whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score.

[0090] The present application also provides a banking system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the above methods. First, the massive data is classified and identified by the column storage technology of ClickHouse, and assigned values, so as to cope with the massive data scenario, and then the discrete points are identified according to the multi-dimensional aggregation function, which improves the detection efficiency of abnormal behavior of user behavior logs in bank data, thereby solving the problem that the existing solutions are difficult to cope with the scenario of massive data such as bank data, resulting in the low detection efficiency of abnormal behavior of user behavior logs in bank data.

[0091] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0092] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0097] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0100] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0101] 1) The method for detecting discrete points in bank data of the present application first classifies and identifies massive data through ClickHouse's column storage technology, and assigns values, so as to cope with massive data scenarios, and then identifies discrete points based on multi-dimensional aggregation functions, thereby improving the detection efficiency of abnormal behaviors in user behavior logs in bank data, thereby solving the problem that existing solutions are difficult to cope with massive data scenarios such as bank data, resulting in low detection efficiency of abnormal behaviors in user behavior logs in bank data.

[0102] 2) The device for detecting discrete points in bank data of the present application first classifies and identifies massive data through ClickHouse's column storage technology, and assigns values, so as to cope with massive data scenarios, and then identifies discrete points based on multi-dimensional aggregation functions, thereby improving the detection efficiency of abnormal behaviors in user behavior logs in bank data, thereby solving the problem that existing solutions are difficult to cope with massive data scenarios such as bank data, resulting in low detection efficiency of abnormal behaviors in user behavior logs in bank data.

[0103] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for detecting discrete points in bank data, characterized in that: include: Obtaining formatted data, wherein the formatted data is data obtained by formatting user behavior log data in bank data; The formatted data is sequentially classified and assigned values ​​using ClickHouse's column storage technology to obtain feature information; At least the feature information is vectorized to obtain a user behavior feature vector, and the user behavior feature vector is processed using a multi-dimensional aggregation function to obtain a final feature score, and based on the size of the final feature score, it is determined whether the user behavior log data corresponding to the feature score is a discrete point.

2. The method according to claim 1, characterized in that The formatted data is sequentially classified and assigned values ​​using ClickHouse's column storage technology to obtain feature information, including: Import the formatted data into the basic log table, and use ClickHouse's column storage technology to classify all the formatted data in the basic log table according to preset classification rules and then store them in column format. The common characteristics of the column-stored data include the number of user sessions, the average number of pages visited by users, the success rate of the user interface, and the average stay time of users; According to the column storage result of the formatted data, the characteristics of the formatted data are assigned values ​​to obtain the characteristic information.

3. The method according to claim 1, characterized in that The user behavior feature vector is processed using a multi-dimensional aggregation function to obtain a final feature score, including: Apply the SQL query in ClickHouse to obtain the mean and standard deviation of the user behavior feature vector; Calculate the feature score of the user behavior feature vector using the Z-Score formula; A weighted average is performed on all the feature scores to obtain the final feature score.

4. The method according to claim 3, characterized in that The feature score of the user behavior feature vector is calculated using the Z-Score formula, including: Determine the feature score according to Z=(xi-mu) / sigma; Wherein, Z is the feature score, mu is the mean of the user behavior feature vector, sigma is the standard deviation, and xi is the actual observation value of the i-th user behavior feature vector.

5. The method according to claim 1, wherein After determining whether the user behavior log data corresponding to the feature score is a discrete point based on the magnitude of the final feature score, the method further includes: Materialized views using ClickHouse's columnar storage technology update aggregation results based on new data; Recalculate the weight of each feature based on the updated materialized view data; Analyzing the difference between the new data and the historical data on each feature to adjust the weight according to the degree of difference; The updated feature weights are applied to the multi-dimensional aggregation function to adjust the core parameters of the multi-dimensional aggregation function.

6. The method according to claim 1, characterized in that Get formatted data, including: Monitor and collect the user behavior log data in the bank data in real time through Filebeat, and convert the user behavior log data into the user behavior log data in stream data format; Send the user behavior log data in the stream data format to the Kafka message queue; A message processing engine is used to read the user behavior log data in the stream data format from the Kafka message queue, and perform data formatting and cutting processing to obtain the formatted data to separate the fields related to user behavior. The data formatting processing includes data parsing and data type conversion.

7. The method according to claim 6, characterized in that Performing data formatting and cutting processing to obtain the formatted data includes: Decoding the user behavior log data in each of the stream data formats, parsing the decoded user behavior log data, extracting key fields, and removing irrelevant or erroneous data entries; The parsed data is converted into a predefined structured format, and the data in the structured format is cut to obtain the formatted data.

8. A device for detecting discrete points in bank data, characterized in that: include: an acquiring unit, configured to acquire formatted data, wherein the formatted data is data obtained by formatting user behavior log data in the bank data; The first processing unit is used to use ClickHouse's column storage technology to sequentially perform classification processing and assignment processing on the formatted data to obtain feature information; The second processing unit is used to at least vectorize the feature information to obtain a user behavior feature vector, and use a multi-dimensional aggregation function to process the user behavior feature vector to obtain a final feature score, and determine whether the user behavior log data corresponding to the feature score is a discrete point based on the size of the final feature score.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A banking system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method of any one of claims 1 to 7.

Images