Massive risk control flow data processing method, device, equipment and medium

Abstract

The present application relates to the technical field of data processing, and discloses a large-scale risk control data processing method, device, equipment and medium, comprising: obtaining risk control data generated by a preset traffic observation and statistics application, writing the risk control data into a preset time series database in batches according to a preset time window, performing redundancy reduction processing on the risk control data by using the time series database to obtain reduced redundancy data, performing hotspot data aggregation on the reduced redundancy data according to pre-acquired historical query data by using the time series database to obtain hotspot data, receiving a query statement sent by a preset query application, performing visual information query on the hotspot data and the reduced redundancy data in the time series database according to the query statement to obtain visual information, performing visual display on the visual information based on a preset visual application, and performing real-time monitoring and abnormal warning on the visual information. The efficiency of processing the risk control data is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for large-scale risk control flow data processing. Background Technology

[0002] In modern risk control systems, core applications such as Soji and Mihawk need to process massive amounts of time-series risk control transaction data. For example, Soji's daily transaction data volume has reached 3.9 billion records, with a data size exceeding 70GB. This type of data is characterized by strong time series, high growth, and high query frequency, which places stringent requirements on the real-time performance, stability, and efficiency of the processing system.

[0003] In traditional technical solutions, the storage and analysis of risk control transaction data rely on a multi-heterogeneous database architecture, typically integrating multiple databases such as MySQL, Elasticsearch (ES), and ClickHouse, and using message queues (MQ) for data flow, ultimately relying on Grafana for monitoring and visualization. However, this architecture has exposed many key problems in practical applications.

[0004] In existing technologies, MySQL's master-slave synchronization mechanism has inherent latency, making it difficult to meet real-time write requirements. While ClickHouse boasts high throughput, its batch insert latency often reaches the minute level, failing to support high-frequency real-time data writes. Furthermore, the synchronization link composed of Flink and Canal suffers from high data transmission latency due to its complex architecture, further exacerbating the real-time performance issue. Elasticsearch responds slowly when processing complex aggregation queries, unable to quickly return high-dimensional statistical results. ClickHouse, due to resource contention between write and query operations, experiences drastic performance fluctuations under high load scenarios, making query stability difficult to guarantee. The hybrid architecture of MySQL+ES+ClickHouse results in data being repeatedly stored across multiple databases, causing severe resource redundancy and increasing the difficulty of maintaining data consistency. In addition, the operational costs of multi-component collaboration are extremely high, and frequent resource expansion is required as data volume grows, limiting system scalability. The end-to-end latency from data generation to completion of storage, processing, and query support can reach tens of minutes, far from meeting the business requirements of second-level decision-making in risk control scenarios, significantly restricting the timeliness of risk identification and interception. These issues collectively lead to the inefficiency of traditional risk control flow data processing solutions in large-scale data scenarios, making it difficult to meet the core requirements of risk control operations for real-time performance, stability, and scalability. Summary of the Invention

[0005] This invention provides a method, apparatus, computer equipment, and medium for large-scale risk control flow data processing, in order to solve the problem of low efficiency in various existing risk control flow data processing methods on the market.

[0006] Firstly, a method for processing large-scale risk control flow data is provided, including: Acquire risk control flow data generated by preset flow observation and statistics applications, and write the risk control flow data into a preset time series database in batches according to a preset time window; The risk control flow data is processed using the time-series database to reduce redundancy, resulting in redundancy-reduced data. The time-series database is used to aggregate hotspot data based on pre-acquired historical query data to obtain hotspot data; Receive a query statement sent by a preset query application, and perform a visualization information query on the hot data and the redundancy reduction data in the time series database according to the query statement to obtain visualization information; The visualization information is displayed using a preset visualization application, and the visualization information is monitored in real time and anomaly warnings are issued.

[0007] Secondly, a large-scale risk control flow data processing device is provided, including: The data writing module is used to acquire the risk control flow data generated by the preset flow observation and statistics application, and write the risk control flow data into the preset time series database in batches according to the preset time window; The redundancy reduction processing module is used to perform redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data. The hotspot aggregation module is used to perform redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data. The data query module is used to aggregate hot data from the redundancy reduction data using the time-series database based on pre-acquired historical query data, and obtain hot data. The real-time monitoring module is used to receive query statements sent by preset query applications, and perform visualization information queries on the hot data and the redundancy reduction data in the time series database according to the query statements to obtain visualization information.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described large-scale risk control flow data processing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned large-scale risk control flow data processing method.

[0010] The aforementioned solution for large-scale risk control transaction data processing, including methods, devices, computer equipment, and storage media, allows for the acquisition of risk control transaction data (including timestamps, amounts, etc.) via message queues or APIs. After mandatory field validation (checking the existence and validity of core fields to prevent data from being sent to erroneous terminals) and timestamp conversion, the data is batch-written into a time-series database (such as TDengine) according to rolling / session time windows, reducing frequent writes and improving performance. The time-series database reduces redundancy in the risk control transaction data by randomly sampling to identify highly repetitive fields, deduplicating them, and storing redundant fields in a dictionary table with an index, thus reducing storage costs. Analyzing the time-series database query logs, dividing time windows to determine hot time ranges, statistically analyzing frequently queried fields, pre-querying and aggregating based on these fields to obtain hot data, improves subsequent query efficiency. When receiving query statements, hot data is queried first; if no match is found, reduced-redundancy data is queried to obtain status data. Status data is visualized using tools such as Grafana, with real-time monitoring and warnings for data exceeding thresholds. This improves the efficiency of risk control transaction data processing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a large-scale risk control flow data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a large-scale risk control flow data processing method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a large-scale risk control flow data processing device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The large-scale risk control flow data processing method provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain risk control transaction data generated by a preset traffic observation and statistics application through the client. Based on a preset time window, the server batches the risk control transaction data into a preset time-series database. The time-series database is used to perform redundancy reduction processing on the risk control transaction data to obtain redundancy-reduced data. The time-series database is then used to aggregate hotspot data from the redundancy-reduced data based on pre-acquired historical query data to obtain hotspot data. The server receives query statements sent by a preset query application and performs visualization information queries on the hotspot data and redundancy-reduced data in the time-series database based on the query statements to obtain visualization information. Based on a preset visualization application, the visualization information is displayed visually, and real-time monitoring and anomaly warnings are provided, improving the efficiency of risk control transaction data management. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a large-scale risk control flow data processing method provided in an embodiment of the present invention includes the following steps: S1. Obtain the risk control flow data generated by the preset flow observation and statistics application, and write the risk control flow data into the preset time series database in batches according to the preset time window.

[0016] In this embodiment of the invention, the acquisition of risk control transaction data generated by the preset traffic observation and statistics application can be achieved by capturing the risk control transaction data generated by the traffic observation and statistics application in real time through message queues (such as Kafka, RabbitMQ) or API interfaces.

[0017] In detail, the data format of the risk control transaction data is usually JSON or CSV, and may include fields such as timestamp, user ID, transaction amount, and device identifier.

[0018] In this embodiment of the invention, the time window is a scrolling window (e.g., every 10 seconds, 20 seconds) or a session window (based on event intervals).

[0019] In this embodiment of the invention, the time-series database may refer to TDengine.

[0020] In this embodiment of the invention, the batch writing involves accumulating data within a window and writing it to the time-series database in batches, thereby reducing database connection overhead and improving write performance. In this embodiment of the invention, the step of batch writing the risk control flow data into a preset time-series database according to a preset time window includes: The risk control flow data is obtained through a preset message queue; The required fields of the risk control data are validated, and the validation results are obtained. Based on the verification results, determine whether the risk control data passes the verification of the required fields. If the operation fails, the risk control data will be sent to a preset error handling terminal. If successful, the risk control data will be converted to a timestamp format to obtain a converted timestamp. The risk control data is written into the time-series database based on the conversion timestamp.

[0021] In detail, the preset message queue is a data buffer component that enables asynchronous communication. It can effectively alleviate the speed difference between data production and processing. By obtaining the risk control flow data through the preset message queue, the processing system can be prevented from being overloaded due to sudden data influx. At the same time, the stability and order of data acquisition are guaranteed, providing a reliable data source for subsequent processing.

[0022] In detail, the mandatory field validation of the risk control transaction data requires a predefined list of mandatory fields (such as transaction time, transaction amount, account identifier, and other core fields) based on business rules or data specifications. Then, for each transaction data entry, the existence of each mandatory field in the list is checked sequentially (i.e., whether the field has a corresponding item in the data), and the validity of the field values ​​is verified (e.g., non-empty, conforming to a preset format or value range; for example, the date field must conform to the format "YYYY-MM-DD HH:MM:SS", and the amount field must be a non-negative number). If any mandatory field in a transaction data entry is missing (no corresponding item) or has an invalid field value (e.g., an empty string, incorrect format), the data entry is marked as "validation failed," and the specific missing or invalid field name and reason are recorded. If all mandatory fields exist and their values ​​are valid, the entry is marked as "validation passed."

[0023] In this embodiment of the invention, by acquiring risk control flow data generated by a preset flow observation and statistics application, and writing the risk control flow data into a preset time series database in batches according to a preset time window, the frequent writing of single data can be avoided, thereby improving query efficiency.

[0024] S2. Use the time-series database to perform redundancy reduction processing on the risk control flow data to obtain redundancy-reduced data.

[0025] In this embodiment of the invention, the redundancy reduction processing of the risk control flow data using the time-series database can be achieved by removing duplicate information (such as device IDs) stored repeatedly in the time-series database, storing redundant information only once, and using pointers to index the stored information (redundant information stored only once) of the remaining data.

[0026] In this embodiment of the invention, the step of using the time-series database to perform redundancy reduction processing on the risk control flow data to obtain redundancy-reduced data includes: Randomly sample the aforementioned risk control flow data to obtain sampled data; Count the number of times each field appears repeatedly in the sampled data; Redundant fields are determined based on the number of repetitions and a preset repetition threshold. After deduplicating the risk control data based on the redundant fields, the redundant fields are written into a dictionary table. An index is created for the deduplicated risk control data based on the dictionary table and the redundant fields to obtain the reduced redundancy data.

[0027] In detail, the risk control flow data is randomly sampled, and the sampled data is extracted from the original data according to a preset ratio (such as 10%) to ensure that the sample can reflect the field distribution characteristics of the overall data.

[0028] In detail, determining redundant fields based on the number of repetitions and a preset repetition threshold involves setting a repetition threshold (e.g., 70%) and comparing the statistically obtained repetition rates of each field with it. If the repetition rate of a field exceeds the threshold, that field is considered redundant.

[0029] In detail, after deduplicating the risk control flow data according to the redundant fields, the redundant fields are written into a dictionary table. This involves extracting the unique value of each redundant field and storing it in a specially designed dictionary table. The original data only retains the index (such as ID) of the value in the dictionary table.

[0030] In detail, the step of creating an index for the deduplicated risk control transaction data based on the dictionary table and the redundant fields to obtain reduced redundancy data utilizes the mapping relationships in the dictionary table to build an indexing system for the deduplicated risk control transaction data. When it is necessary to query the original value, the system can quickly locate the corresponding item in the dictionary table through the index.

[0031] In this embodiment of the invention, by using the time-series database to perform redundancy reduction processing on the risk control flow data, redundancy-reduced data is obtained, which can reduce storage costs and improve storage efficiency.

[0032] S3. Using the time-series database, perform hotspot data aggregation on the redundancy reduction data based on the pre-acquired historical query data to obtain hotspot data.

[0033] In this embodiment of the invention, the redundancy reduction data is aggregated using the time-series database based on pre-acquired historical query data to obtain hotspot data. This can be achieved by collecting logs from query applications, extracting historical query data, and analyzing high-frequency query patterns, including analyzing high-frequency query fields such as "total user transaction amount for the day" and "number of device anomalies for the day"; and analyzing the hotspot time range, for example, if the data query rate in the last 24 hours reaches 80%, the last 24 hours can be identified as the hotspot time range. Then, aggregation dimensions are defined based on the hotspot time range, such as grouping by time granularity (minutes, hours), region, user tags, etc. Pre-queries are performed based on high-frequency query fields to obtain query fields, and the query fields are aggregated according to the aggregation dimensions to obtain the hotspot data.

[0034] In this embodiment of the invention, the step of using the time-series database to aggregate hotspot data based on pre-acquired historical query data to obtain hotspot data includes: Obtain the query logs of the time-series database, and obtain historical query data for a preset time period based on the query logs; The historical query data is divided into time windows to obtain window query data; Calculate the total amount of query data in each window of the window query data; Analyze the hot time range of the window query data based on the total amount of query data; Calculate the query frequency of each query field in the historical query data; Based on the query frequency, identify the high-frequency query fields in the historical query data; The time-series database is used to perform a pre-query based on the high-frequency query field to obtain the pre-query results; The pre-query results are aggregated based on the hotspot time range to obtain hotspot data.

[0035] In detail, the step of obtaining the query logs of the time-series database and retrieving historical query data for a preset time period based on the query logs is achieved by periodically collecting query logs within a preset time period (e.g., the last 30 days) through the log interface of the time-series database (such as TDengine). The logs contain information such as SQL statements, execution time, and query fields.

[0036] In detail, the step of dividing the historical query data into time windows to obtain window query data is to use a sliding window or fixed window algorithm (such as one window per hour) to divide the historical query data.

[0037] In detail, the step of analyzing the hotspot time range of the window query data based on the total query data involves identifying the hotspot time range by applying threshold judgment or clustering algorithms based on the total window query data. For example, a threshold can be set at 120% of the daily average query volume, and windows exceeding this threshold are defined as hotspot windows.

[0038] In this embodiment of the invention, the step of analyzing the hotspot time range of the window query data based on the total amount of query data includes: The window query data is filtered for invalid queries to obtain the window filtered query data; Calculate the average amount of query data based on the total amount of query data in each window of the window-filtered query data; Identify data windows in the window filter query data that are larger than the average amount of the query data to obtain excess query window data; Calculate the gradient value of the total amount of query data in adjacent windows in the excess query window data to obtain a set of gradient values; Based on the set of gradient values, the hotspot time range in the excess query window data is determined.

[0039] In detail, the invalid query filtering of the window query data can be performed by identifying and filtering internal system operations (such as metadata queries like SHOW TABLES), failed SQL queries (records with unsuccessful status codes), and abnormal data generated during the testing or debugging phase through preset rules.

[0040] In detail, the gradient value of the total amount of query data in adjacent windows in the excess query window data is calculated, the excess windows are sorted in chronological order, and then the difference in query volume between adjacent windows (i.e., gradient value) is calculated to reflect the rate of change of query volume in the time dimension.

[0041] In detail, the process involves identifying hotspot time ranges in the excessive query window data based on the gradient value set. By analyzing the changing characteristics of the gradient values, the boundaries of sudden increases and decreases in query volume are identified, thereby determining continuous hotspot time periods. In implementation, a gradient threshold (such as based on quantiles or standard deviations of gradient values) is first set. When a gradient value exceeds a positive threshold, it is marked as the start point of a hotspot; when it falls below a negative threshold, it is marked as the end point of a hotspot. Then, adjacent start and end points are paired, overlapping or continuous intervals are merged, and invalid hotspots with excessively short durations are filtered out.

[0042] In this embodiment of the invention, confirming the hotspot time range in the excessive query window data based on the gradient value set includes: The increase or decrease relationship of the total amount of query data in adjacent windows in the excess query window data is determined based on the sign of the gradient values ​​in the gradient value set. Based on the increase / decrease relationship, identify the hotspot start boundary and hotspot end boundary in the excess query window data; The hotspot range window in the excess query window data is determined based on the hotspot start boundary and the hotspot end boundary. Determine whether there are adjacent hotspot range windows in the excess query window data; If they exist, merge adjacent hotspot range windows, extract the time range corresponding to the merged hotspot range window, and obtain the hotspot time range. If it does not exist, extract the time range corresponding to each hotspot range window to obtain the hotspot time range.

[0043] In this embodiment of the invention, by using the time-series database to aggregate hot data based on pre-acquired historical query data, the efficiency of subsequent queries can be improved.

[0044] S4. Receive a query statement sent by a preset query application, and perform a visualization information query on the hot data and the redundancy reduction data in the time series database according to the query statement to obtain visualization information.

[0045] In this embodiment of the invention, the step of performing a visualization information query on the hot data and the reduced redundancy data in the time-series database according to the query statement is achieved by first determining whether the query statement hits the hot data. When the query condition matches the pre-aggregation dimension (such as "query the total transaction amount of each user today"), the data is directly obtained from the aggregation result, skipping the original table scan; if the query statement does not hit the hot data, then the reduced redundancy data is queried.

[0046] In this embodiment of the invention, by receiving a query statement sent by a preset query application, and performing a visualization query on the hot data and the redundancy reduction data in the time-series database according to the query statement, visualization information can be obtained, thereby improving query efficiency.

[0047] In this embodiment of the invention, the step of performing a visualization information query on the hot data and the redundancy reduction data in the time-series database according to the query statement to obtain visualization information includes: The query fields of the query statement are extracted through data parsing; Determine whether the hot data contains the query field; If the hot data contains the query field, then a visualization information query is performed on the hot data according to the query statement to obtain the visualization information; If the hot data does not contain the query field, then a visualization information query is performed on the reduced redundancy data according to the query statement to obtain the visualization information.

[0048] S5. Based on a preset visualization application, the visualization information is displayed in a visual format, and the visualization information is monitored in real time and anomaly warnings are issued.

[0049] In this embodiment of the invention, the visualization of the information based on a preset visualization application can be achieved by connecting to visualization tools such as Grafana and Tableau to display the information.

[0050] In detail, the visualized information may include cluster status, node information, insertion and query requests, resource usage, and other information.

[0051] In this embodiment of the invention, the real-time monitoring and anomaly warning of the visualized information refers to real-time monitoring of whether the relevant data in the visualized information exceeds a preset threshold, and issuing a warning if the threshold is exceeded.

[0052] As can be seen, in the above solution, risk control transaction data (including timestamps, amounts, etc.) is obtained through message queues or APIs. After mandatory field validation (checking the existence and validity of core fields, preventing data from being sent to erroneous terminals) and timestamp conversion, it is written in batches to a time-series database (such as TDengine) according to rolling / session time windows, reducing frequent writes and improving performance. The time-series database reduces redundancy of risk control transaction data by randomly sampling to identify highly repetitive fields, deduplicating them, and storing redundant fields in a dictionary table with an index, reducing storage costs. The time-series database query logs are analyzed to divide time windows to determine hot time ranges, and frequently queried fields are statistically analyzed. Pre-queries are performed and aggregated to obtain hot data, improving the efficiency of subsequent queries. When receiving query statements, hot data is queried first; if no match is found, reduced redundant data is queried to obtain status data. Status data is visualized using tools such as Grafana, and real-time monitoring and alerts are given for abnormal data exceeding thresholds. This improves the efficiency of risk control transaction data processing.

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0054] In one embodiment, a large-scale risk control flow data processing device is provided, which corresponds one-to-one with the large-scale risk control flow data processing method described in the above embodiments. For example... Figure 3 As shown, this large-scale risk control flow data processing device includes a data writing module 101, a redundancy reduction processing module 102, a hotspot aggregation module 103, a data query module 104, and a real-time monitoring module 105. Detailed descriptions of each functional module are as follows: The data writing module 101 is used to acquire the risk control flow data generated by the preset flow observation and statistics application, and write the risk control flow data into the preset time series database in batches according to the preset time window. The redundancy reduction processing module 102 is used to perform redundancy reduction processing on the risk control flow data using the time series database to obtain redundancy-reduced data. Hotspot aggregation module 103 is used to perform redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data. The data query module 104 is used to aggregate hot data of the redundancy reduction data using the time-series database based on the pre-acquired historical query data, and obtain hot data. The real-time monitoring module 105 is used to receive query statements sent by a preset query application, and perform visualization information queries on the hot data and the redundancy reduction data in the time series database according to the query statements to obtain visualization information.

[0055] In one embodiment, when the data writing module 101 performs the step of batch writing the risk control flow data into a preset time-series database according to a preset time window, it is specifically used for: The risk control flow data is obtained through a preset message queue; The required fields of the risk control data are validated, and the validation results are obtained. Based on the verification results, determine whether the risk control data passes the verification of the required fields. If the operation fails, the risk control data will be sent to a preset error handling terminal. If successful, the risk control data will be converted to a timestamp format to obtain a converted timestamp. The risk control data is written into the time-series database based on the conversion timestamp.

[0056] In one embodiment, the redundancy reduction processing module 102, when performing the redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data, is specifically used for: Randomly sample the aforementioned risk control flow data to obtain sampled data; Count the number of times each field appears repeatedly in the sampled data; Redundant fields are determined based on the number of repetitions and a preset repetition threshold. After deduplicating the risk control data based on the redundant fields, the redundant fields are written into a dictionary table. An index is created for the deduplicated risk control data based on the dictionary table and the redundant fields to obtain the reduced redundancy data.

[0057] In one embodiment, the hotspot aggregation module 103, when performing the hotspot data aggregation of the de-redundancy data using the time-series database based on pre-acquired historical query data to obtain hotspot data, is specifically used for: Obtain the query logs of the time-series database, and obtain historical query data for a preset time period based on the query logs; The historical query data is divided into time windows to obtain window query data; Calculate the total amount of query data in each window of the window query data; Analyze the hot time range of the window query data based on the total amount of query data; Calculate the query frequency of each query field in the historical query data; Based on the query frequency, identify the high-frequency query fields in the historical query data; The time-series database is used to perform a pre-query based on the high-frequency query field to obtain the pre-query results; The pre-query results are aggregated based on the hotspot time range to obtain hotspot data.

[0058] In one embodiment, the hotspot aggregation module 103, when performing the analysis of the hotspot time range of the window query data based on the total amount of query data, is specifically used for: The window query data is filtered for invalid queries to obtain the window filtered query data; Calculate the average amount of query data based on the total amount of query data in each window of the window-filtered query data; Identify data windows in the window filter query data that are larger than the average amount of the query data to obtain excess query window data; Calculate the gradient value of the total amount of query data in adjacent windows in the excess query window data to obtain a set of gradient values; Based on the set of gradient values, the hotspot time range in the excess query window data is determined.

[0059] In one embodiment, the hotspot aggregation module 103, when performing the step of confirming the hotspot time range in the excess query window data based on the gradient value set, is specifically used for: The increase or decrease relationship of the total amount of query data in adjacent windows in the excess query window data is determined based on the sign of the gradient values ​​in the gradient value set. Based on the increase / decrease relationship, identify the hotspot start boundary and hotspot end boundary in the excess query window data; The hotspot range window in the excess query window data is determined based on the hotspot start boundary and the hotspot end boundary. Determine whether there are adjacent hotspot range windows in the excess query window data; If they exist, merge adjacent hotspot range windows, extract the time range corresponding to the merged hotspot range window, and obtain the hotspot time range. If it does not exist, extract the time range corresponding to each hotspot range window to obtain the hotspot time range.

[0060] In one embodiment, the data query module 104, when performing the visualization information query on the hot data and the redundancy reduction data in the time-series database according to the query statement to obtain visualization information, is specifically used for: The query fields of the query statement are extracted through data parsing; Determine whether the hot data contains the query field; If the hot data contains the query field, then a visualization information query is performed on the hot data according to the query statement to obtain the visualization information; If the hot data does not contain the query field, then a visualization information query is performed on the reduced redundancy data according to the query statement to obtain the visualization information.

[0061] This invention provides a large-scale risk control transaction data processing device. It acquires risk control transaction data (including timestamps, amounts, etc.) via message queues or APIs. After mandatory field validation (checking the existence and validity of core fields to prevent data from being sent to erroneous terminals) and timestamp conversion, the data is batch-written to a time-series database (such as TDengine) according to rolling / session time windows, reducing frequent writes and improving performance. The time-series database reduces redundancy in the transaction data by randomly sampling to identify highly repetitive fields, deduplicating them, and storing redundant fields in a dictionary table with an index, reducing storage costs. It analyzes the time-series database query logs, divides time windows to determine hot time ranges, statistically analyzes frequently queried fields, pre-queries and aggregates them to obtain hot data, improving subsequent query efficiency. When receiving query statements, it prioritizes querying hot data; if no match is found, it queries reduced-redundancy data and obtains status data. It visualizes the status data using tools such as Grafana, monitors in real time, and issues warnings for abnormal data exceeding thresholds. This improves the efficiency of risk control transaction data management. Specific limitations regarding the large-scale risk control flow data processing device can be found in the limitations of the large-scale risk control flow data processing method described above, and will not be repeated here. Each module in the aforementioned large-scale risk control flow data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0062] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a large-scale risk control flow data processing method on the server side.

[0063] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a large-scale risk control flow data processing method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire risk control flow data generated by preset flow observation and statistics applications, and write the risk control flow data into a preset time series database in batches according to a preset time window; The risk control flow data is processed using the time-series database to reduce redundancy, resulting in redundancy-reduced data. The time-series database is used to aggregate hotspot data based on pre-acquired historical query data to obtain hotspot data; Receive a query statement sent by a preset query application, and perform a visualization information query on the hot data and the redundancy reduction data in the time series database according to the query statement to obtain visualization information; The visualization information is displayed using a preset visualization application, and the visualization information is monitored in real time and anomaly warnings are issued.

[0064] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire risk control flow data generated by preset flow observation and statistics applications, and write the risk control flow data into a preset time series database in batches according to a preset time window; The risk control flow data is processed using the time-series database to reduce redundancy, resulting in redundancy-reduced data. The time-series database is used to aggregate hotspot data based on pre-acquired historical query data to obtain hotspot data; Receive a query statement sent by a preset query application, and perform a visualization information query on the hot data and the redundancy reduction data in the time series database according to the query statement to obtain visualization information; The visualization information is displayed using a preset visualization application, and the visualization information is monitored in real time and anomaly warnings are issued.

[0065] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0068] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for large-scale risk control flow data processing, characterized in that, include: Acquire risk control flow data generated by preset flow observation and statistics applications, and write the risk control flow data into a preset time series database in batches according to a preset time window; The risk control flow data is processed using the time-series database to reduce redundancy, resulting in redundancy-reduced data. Obtain the query log of the time series database, obtain historical query data for a preset time period based on the query log, and divide the historical query data into time windows to obtain window query data; Identify data windows in the query data that are larger than the average amount of query data to obtain excess query window data; Calculate the gradient value of the total amount of query data in adjacent windows in the excess query window data to obtain a set of gradient values; Based on the gradient value set, the hotspot time range in the excess query window data is determined; The time-series database is used to perform a pre-query based on high-frequency query fields to obtain the pre-query results; The pre-query results are aggregated based on the hotspot time range to obtain hotspot data; Receive a query statement sent by a preset query application, and perform a visualization information query on the hot data and the redundancy reduction data in the time series database according to the query statement to obtain visualization information; The visualization information is displayed using a preset visualization application, and the visualization information is monitored in real time and anomaly warnings are issued.

2. The large-scale risk control flow data processing method as described in claim 1, characterized in that, The step of writing the risk control flow data into a preset time-series database in batches according to a preset time window includes: The risk control flow data is obtained through a preset message queue; The required fields of the risk control data are validated, and the validation results are obtained. Based on the verification results, determine whether the risk control data passes the verification of the required fields. If the operation fails, the risk control data will be sent to a preset error handling terminal. If successful, the risk control data will be converted to a timestamp format to obtain a converted timestamp. The risk control data is written into the time-series database based on the conversion timestamp.

3. The large-scale risk control flow data processing method as described in claim 1, characterized in that, The process of using the time-series database to perform redundancy reduction processing on the risk control flow data to obtain redundancy-reduced data includes: Randomly sample the aforementioned risk control flow data to obtain sampled data; Count the number of times each field appears repeatedly in the sampled data; Redundant fields are determined based on the number of repetitions and a preset repetition threshold. After deduplicating the risk control data based on the redundant fields, the redundant fields are written into a dictionary table. An index is created for the deduplicated risk control data based on the dictionary table and the redundant fields to obtain the reduced redundancy data.

4. The large-scale risk control flow data processing method as described in claim 1, characterized in that, The step of using the time-series database to perform a pre-query based on high-frequency query fields to obtain pre-query results includes: Calculate the total amount of query data in each window of the window query data; Calculate the query frequency of each query field in the historical query data; Based on the query frequency, identify the high-frequency query fields in the historical query data; The time-series database is used to perform a pre-query based on the high-frequency query field to obtain the pre-query results.

5. The large-scale risk control flow data processing method as described in claim 1, characterized in that, Before identifying data windows in the window query data that are larger than the average amount of query data, and obtaining excess query window data, the process further includes: The window query data is filtered for invalid queries to obtain the window filtered query data; The average amount of query data is calculated based on the total amount of query data in each window of the window-filtered query data.

6. The large-scale risk control flow data processing method as described in claim 1, characterized in that, The step of determining the hotspot time range in the excess query window data based on the gradient value set includes: The increase or decrease relationship of the total amount of query data in adjacent windows in the excess query window data is determined based on the sign of the gradient values ​​in the gradient value set. Based on the increase / decrease relationship, identify the hotspot start boundary and hotspot end boundary in the excess query window data; The hotspot range window in the excess query window data is determined based on the hotspot start boundary and the hotspot end boundary. Determine whether there are adjacent hotspot range windows in the excess query window data; If they exist, merge adjacent hotspot range windows, extract the time range corresponding to the merged hotspot range window, and obtain the hotspot time range. If it does not exist, extract the time range corresponding to each hotspot range window to obtain the hotspot time range.

7. The large-scale risk control flow data processing method as described in claim 1, characterized in that, The step of performing a visualization information query on the hotspot data and the redundancy reduction data in the time-series database according to the query statement to obtain visualization information includes: The query fields of the query statement are extracted through data parsing; Determine whether the hot data contains the query field; If the hot data contains the query field, then a visualization information query is performed on the hot data according to the query statement to obtain the visualization information; If the hot data does not contain the query field, then a visualization information query is performed on the reduced redundancy data according to the query statement to obtain the visualization information.

8. A large-scale risk control flow data processing device, characterized in that, include: The data writing module is used to acquire the risk control flow data generated by the preset flow observation and statistics application, and write the risk control flow data into the preset time series database in batches according to the preset time window; The redundancy reduction processing module is used to perform redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data. The hotspot aggregation module is used to perform redundancy reduction processing on the risk control flow data using the time-series database to obtain redundancy-reduced data. The data query module is used for: Obtain the query log of the time series database, obtain historical query data for a preset time period based on the query log, and divide the historical query data into time windows to obtain window query data; Identify data windows in the query data that are larger than the average amount of query data to obtain excess query window data; Calculate the gradient value of the total amount of query data in adjacent windows in the excess query window data to obtain a set of gradient values; Based on the gradient value set, the hotspot time range in the excess query window data is determined; The time-series database is used to perform a pre-query based on high-frequency query fields to obtain the pre-query results; The pre-query results are aggregated based on the hotspot time range to obtain hotspot data; The real-time monitoring module is used to receive query statements sent by preset query applications, and perform visualization information queries on the hot data and the redundancy reduction data in the time series database according to the query statements to obtain visualization information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the large-scale risk control flow data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large-scale risk control flow data processing method as described in any one of claims 1 to 7.

Images