Risk control flow data real-time processing method, system, device and storage medium
By preprocessing and expiration judgment of risk control flow data, and using preset processing windows and templates for data processing, the problem of high memory requirements in the risk control system is solved, and real-time updates and efficient processing of flow data are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MERCHANTS BANK
- Filing Date
- 2023-10-24
- Publication Date
- 2026-07-24
AI Technical Summary
Existing risk control systems require frequent definition of variables and updates to risk control rules, resulting in a large memory storage requirement and impacting system performance and efficiency.
By acquiring risk control flow data from business scenarios for preprocessing, data processing is performed using preset processing windows and templates, and expiration judgment and result updates are performed based on data expiration time, reducing the memory requirements of custom templates.
It enables real-time updating and efficient processing of streaming data without increasing memory, reducing memory requirements and improving the processing capacity of the risk control system.
Smart Images
Figure CN117407425B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic data processing technology, and in particular to a method, system, device, and computer storage medium for real-time processing of risk control flow data. Background Technology
[0002] With the growth of online data, higher demands are being placed on the real-time analysis capabilities of online data traffic. For example, in the risk control systems of financial institutions, in order to intercept abnormal transactions in a timely manner, protect customer assets, and not disrupt normal transaction processes, a real-time big data risk control system is needed to provide certain risk control capabilities. In risk management, the real-time risk control system first defines variables based on input from risk control analysis personnel, then calculates relevant variables using risk control rules and outputs the variable values for use in risk control decisions.
[0003] However, financial crimes are becoming increasingly complex and varied. To ensure the risk control capabilities of real-time risk control systems, these systems need to frequently define variables and update risk control rules. As a result, traditional risk control methods require a large amount of memory to store risk control rules and variables. Summary of the Invention
[0004] This invention proposes a real-time processing method, system, device, and computer storage medium for risk control flow data, aiming to solve the technical problem that existing risk control methods rely on large amounts of memory.
[0005] To address the above problems, this invention proposes a real-time processing method for risk control flow data, which includes:
[0006] Obtain the business scenario and preprocess the risk control flow data of the business scenario to obtain effective flow data, wherein the effective flow data includes data expiration time;
[0007] The valid stream data is added to a preset processing window, and the valid stream data in the processing window is processed by the processing template corresponding to the business scenario to obtain the processing result, and the processing result is stored.
[0008] The valid stream data in the processing window is expired based on the data expiration time to obtain an expiration result. A new processing result is obtained based on the expiration result, the processing template, and the valid stream data. The new processing result is then used to update the processing result.
[0009] Optionally, the preprocessing includes: format conversion, data filtering, and data cleaning. The step of preprocessing the risk control flow data of the business scenario to obtain effective flow data includes:
[0010] The risk control flow data of the business scenario is converted into a format to obtain the first intermediate flow data, wherein the first intermediate flow data is in JSON format;
[0011] Based on the business scenario, the first intermediate stream data is filtered to obtain the second intermediate stream data, and the second intermediate stream data is cleaned to obtain the effective stream data.
[0012] Optionally, before the step of determining the expiration of valid stream data in the processing window based on the data expiration time to obtain an expiration result, the method further includes:
[0013] Get the current time;
[0014] The step of determining the expiration of valid stream data in the processing window based on the data expiration time to obtain an expiration result includes:
[0015] The current time is compared with the data expiration time in the processing window. If the current time is greater than or equal to the data expiration time, the expiration result is set to expired; otherwise, the expiration result is set to not expired.
[0016] Optionally, the step of obtaining a new processing result based on the expired result, the processing template, and the valid stream data includes:
[0017] When the expired result is detected as expired, the valid stream data corresponding to the data expiration time in the processing window is deleted, and the valid stream data in the processing window after deletion is processed by the processing template to obtain a new processing result;
[0018] When the expired result is detected as not expired, the processing result is taken as the new processing result.
[0019] Optionally, the step of processing the valid stream data in the deleted processing window using the processing template to obtain a new processing result includes:
[0020] An update job is constructed based on the valid stream data in the deleted processing window and the processing template, and the update job is executed to obtain the execution result;
[0021] The update job progress is obtained based on the execution result, and a new processing result is obtained based on the job progress.
[0022] Optionally, the step of obtaining the job progress of the update job based on the execution result, and obtaining a new processing result based on the job progress, includes:
[0023] When the execution result indicates that the update job has failed, the job progress of the update job is stored, and the update job is restored using the job progress to obtain a new processing result;
[0024] When the execution result indicates that the update job has been successfully executed, the job progress of the update job is set to completed, and the processing result is used as the new processing result.
[0025] Optionally, the method further includes:
[0026] Obtain the interruption time and interruption handling result, wherein the interruption handling result is the handling result when the update job fails to execute;
[0027] The step of restoring the update job based on the job progress to obtain a new processing result includes:
[0028] The unprocessed stream data in the processing window is found through the job progress, and the data expiration time of the unprocessed stream data is compared with the interruption time.
[0029] When the data expiration time is detected to be less than or equal to the interruption time, the unprocessed data stream corresponding to the data expiration time in the processing window is deleted, and the unprocessed data stream in the processing window is processed by the processing template to obtain an intermediate result. The intermediate result and the interruption processing result are then merged to obtain a new processing result.
[0030] To address the above problems, this invention also proposes a real-time processing system for risk control flow data, which includes:
[0031] A data preprocessing module is used to acquire business scenarios and preprocess the risk control flow data of the business scenarios to obtain effective flow data, wherein the effective flow data includes data expiration time;
[0032] The new data triggering module is used to add the valid stream data to a preset processing window, process the valid stream data in the processing window through the processing template corresponding to the business scenario, obtain the processing result, and store the processing result;
[0033] The processing result update module is used to determine the expiration of valid stream data in the processing window based on the data expiration time, obtain an expiration result, obtain a new processing result based on the expiration result, the processing template, and the valid stream data, and update the processing result with the new processing result.
[0034] To address the aforementioned problems, this invention also proposes a real-time processing device for risk control flow data. The real-time processing device for risk control flow data includes: a memory, a processor, and a real-time processing program for risk control flow data stored in the memory and executable on the processor. When the real-time processing program for risk control flow data is executed by the processor, it implements the steps of the real-time processing method for risk control flow data as described above.
[0035] To address the aforementioned problems, the present invention also proposes a computer storage medium storing a real-time processing program for risk control flow data. When the real-time processing program for risk control flow data is executed by a processor, it implements the steps of the real-time processing method for risk control flow data as described above.
[0036] In this embodiment of the invention, by acquiring a business scenario and preprocessing the streaming data of the business scenario to obtain valid streaming data, wherein the valid streaming data includes data expiration time; invalid data can be deleted, so that only valid streaming data participates in real-time processing; then, by adding the valid streaming data to a preset processing window, and processing the valid streaming data in the processing window through the processing template corresponding to the business scenario, the processing result is obtained and stored; it is possible to process the valid streaming data in the processing window based on the actual business scenario of the streaming data, and to process the streaming data based on the existing template, thereby eliminating the need for custom templates and reducing the memory requirements of custom templates; then, based on the data expiration time, the valid streaming data in the processing window is expired to obtain an expiration result, and a new processing result is obtained based on the expiration result, the processing template, and the valid streaming data, and the processing result is updated with the new processing result, which can update the processing result without increasing memory, when new streaming data enters or when there is expired streaming data.
[0037] Compared to traditional real-time streaming data processing methods, this invention processes streaming data based on a pre-defined processing template corresponding to the business scenario. Therefore, this invention does not require setting up a storage area for custom processing templates, thus reducing the memory requirements of real-time streaming data processing methods. Furthermore, this invention achieves real-time updates of streaming data processing results by setting a processing window and determining whether expired streaming data exists in the processing window based on an expiration time, and triggering an update operation when expired streaming data is present. Attached Figure Description
[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a schematic diagram of the hardware operating environment of the real-time processing equipment for risk control flow data involved in the embodiments of the present invention;
[0041] Figure 2 This is a flowchart illustrating the first embodiment of the real-time processing method for risk control flow data of the present invention.
[0042] Figure 3 This is a schematic diagram of the effective flow data acquisition process in an embodiment of the real-time processing method for risk control flow data of the present invention;
[0043] Figure 4 This is a schematic diagram of the data flow direction in an embodiment of the real-time processing method for risk control flow data of the present invention;
[0044] Figure 5 This is a flowchart illustrating the result storage process of an embodiment of the real-time processing method for risk control flow data of the present invention.
[0045] Figure 6 This is a flowchart illustrating the expired data processing of an embodiment of the real-time processing method for risk control flow data of the present invention.
[0046] Figure 7 This is a schematic diagram showing the result output of an embodiment of the real-time processing method for risk control flow data of the present invention;
[0047] Figure 8 This is a flowchart illustrating the calculation process of an embodiment of the real-time processing method for risk control flow data of the present invention.
[0048] Figure 9 This is a functional block diagram of an embodiment of the real-time processing system for risk control flow data of the present invention. Detailed Implementation
[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0051] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0052] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0053] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.
[0054] like Figure 1 As shown, Figure 1 This is a schematic diagram of the hardware operating environment of the real-time processing equipment for risk control flow data involved in the embodiments of the present invention.
[0055] like Figure 1 As shown, in the hardware operating environment of the real-time processing device for risk control flow data, the device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0056] Those skilled in the art will understand that Figure 1 The real-time processing equipment structure for risk control flow data shown does not constitute a limitation on the terminal equipment. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0057] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a real-time processing program for risk control flow data.
[0058] exist Figure 1 In the device shown, network interface 1004 is mainly used to connect to the backend server and communicate with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate with it; while processor 1001 can be used to call the real-time processing program for risk control flow data stored in memory 1005 and perform the following operations:
[0059] Obtain the business scenario and preprocess the risk control flow data of the business scenario to obtain effective flow data, wherein the effective flow data includes data expiration time;
[0060] The valid stream data is added to a preset processing window, and the valid stream data in the processing window is processed by the processing template corresponding to the business scenario to obtain the processing result, and the processing result is stored.
[0061] The valid stream data in the processing window is expired based on the data expiration time to obtain an expiration result. A new processing result is obtained based on the expiration result, the processing template, and the valid stream data. The new processing result is then used to update the processing result.
[0062] Based on the above hardware structure, the overall concept of various embodiments of the real-time processing method for risk control flow data of the present invention is proposed.
[0063] In this embodiment of the invention, the increasing volume of online data places higher demands on the real-time analysis capabilities of online data flow. For example, in the risk control systems of financial institutions, in order to promptly intercept abnormal transactions, protect customer assets, and not disrupt normal transaction processes, a real-time big data risk control system is required to provide certain risk control capabilities. During risk management, the real-time risk control system first defines variables based on input from risk control analysis personnel, then calculates relevant variables using risk control rules and outputs the variable values for risk control decision-making.
[0064] However, financial crimes are becoming increasingly complex and varied. To ensure the risk control capabilities of real-time risk control systems, these systems need to frequently define variables and update risk control rules. As a result, traditional risk control methods require a large amount of memory to store risk control rules and variables.
[0065] To address the above problems, this invention proposes a real-time processing method for risk control flow data.
[0066] Based on the overall concept of the above embodiments of the real-time processing method for risk control flow data of the present invention, various embodiments of the real-time processing method for risk control flow data of the present invention are proposed.
[0067] It should be noted that the execution subject of the real-time processing method for risk control flow data in this invention is a real-time processing device for risk control flow data, which can be a server, computer, mobile phone, or other devices. For ease of explanation, the execution subject has been omitted from the description in the following embodiments.
[0068] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the real-time processing method for risk control flow data according to the present invention. It should be noted that although the logical order is shown in the flowchart, in certain situations, the steps of the real-time processing method for risk control flow data according to the present invention can be executed in a different order.
[0069] In this embodiment, the real-time processing method for the risk control flow data includes:
[0070] Step S10: Obtain the business scenario and preprocess the risk control flow data of the business scenario to obtain effective flow data, wherein the effective flow data includes the data expiration time;
[0071] It should be noted that "business scenario" refers to the application scenario of streaming data, such as a transfer scenario, a storage scenario, or a loan scenario. "Risk control streaming data" refers to streaming data used for risk control.
[0072] In this embodiment, it is necessary to obtain the application scenario of the streaming data waiting to be analyzed, and to preprocess the streaming data used for risk control in the application scenario to obtain the effective streaming data in the business scenario. The effective streaming data includes the data expiration time.
[0073] As an example, the business scenario could be a money transfer scenario. In this case, the money transfer details obtained in the money transfer scenario would be used as risk control flow data, and the money transfer transaction details would be preprocessed to obtain valid flow data in the money transfer scenario. Each valid flow data would include a data expiration time.
[0074] Optionally, in a feasible embodiment, the preprocessing includes: format conversion, data filtering, and data cleaning, and the above step S10 includes:
[0075] Step S101: Convert the format of the risk control flow data of the business scenario to obtain the first intermediate flow data, wherein the first intermediate flow data is in JSON format;
[0076] Since the business data comes from various user terminals, and the data output formats of each user terminal are inconsistent, after obtaining the streaming data corresponding to the business scenario, it is necessary to convert the streaming data into a format, that is, to convert streaming data in other formats into JSON format data, where the JSON format data is the first intermediate streaming data.
[0077] Step S102: Filter the first intermediate stream data according to the business scenario to obtain the second intermediate stream data, and clean the second intermediate stream data to obtain effective stream data.
[0078] It should be noted that data filtering refers to screening data based on one or more parameters. Data cleaning refers to removing data that does not conform to preset rules, which are specified by technical personnel.
[0079] After obtaining the first intermediate stream data, select one or more filtering parameters according to the business scenario, and filter the first intermediate stream data based on one or more parameters to obtain the second intermediate stream data. Then, remove the data in the second intermediate stream data that does not conform to the preset rules according to the preset rules to obtain the second intermediate stream data that conforms to the preset rules. The second intermediate stream data that conforms to the preset rules is the valid stream data.
[0080] As an example, in a business scenario involving money transfers and the streaming data consisting of transfer details, the process first converts the transfer details collected from various user terminals into JSON format to obtain the first intermediate stream data. Then, filtering parameters are determined based on the business scenario. For example, a filtering parameter could be that the transfer amount exceeds 10,000. Based on these parameters, the first intermediate stream data can be filtered to obtain the second intermediate stream data. Then, preset rules are obtained, and data in the second intermediate stream that does not conform to these rules is removed, resulting in valid streaming data. For example, if the preset rule is: "When a transfer fails, the streaming data is processed," then based on this preset rule, successfully transferred data in the second intermediate stream data can be removed, ensuring that the valid streaming data consists entirely of transfers exceeding 10,000 that failed.
[0081] In the process of streaming data processing, different jobs correspond to different fields in different business scenarios, which makes the workload of redesigning data formats and developing access when developing new jobs quite large. Therefore, the present invention can obtain effective streaming data in the following way:
[0082] Step a: Register data source forwarding information and data source information in MySQL. The data source includes the Kafka instance that sends data to the business scenario. The data source forwarding information includes the original Kafka cluster and topic, and the target Kafka cluster and topic (the topic used to connect to Flink). The data source information includes the data format, a list of data field names, and information such as the type and length of each field.
[0083] Step b: Convert the data into a JSON data format acceptable to the built-in Flink connector, and filter and clean the data;
[0084] Step c: Obtain data source forwarding information by querying the MySQL database, and forward the data to the Kafka topic pre-configured by the data source;
[0085] Step d: Obtain the data source fields, Kafka cluster information, and corresponding topic information by querying the MySQL database;
[0086] Step f: In response to the user's action, select the specific field in Kafka, generate the Flink data source table creation SQL, and submit it to the Flink cluster to create the data source table;
[0087] Kafka stores risk control flow data. The risk control flow data is converted into JSON format to obtain the first intermediate flow data. After selecting a specific field in the first intermediate flow data and generating a data source table, the valid flow data for this business scenario is obtained.
[0088] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the effective stream data acquisition process in an embodiment of the real-time processing method for risk control stream data of the present invention. In this embodiment, the data forwarding application can forward the data source information in MySQL and the upstream data of other applications in the current scenario to Kafka connected to the Flink computing engine. The data source management service can query the data source information in MySQL, and MySQL can register the data source information in the data source management service. In addition, the Flink table management service can also obtain the source table related information from the data source management service, and then pass the table creation SQL and the source table creation SQL to the Flink computing engine, thereby obtaining effective stream data based on the data accessed by the Flink computing engine and Kafka.
[0089] Step S20: Add the valid stream data to a preset processing window, process the valid stream data in the processing window using the processing template corresponding to the business scenario, obtain the processing result, and store the processing result;
[0090] It should be noted that the processing window can be a data structure such as an array, linked list, or stack, and the processing templates include: function computation modules, filtering information, deduplication flags, etc. Furthermore, when processing valid streaming data using processing templates, it can also be processed through Flink's Over window.
[0091] After converting, filtering, and cleaning the risk control flow data to obtain valid flow data, the valid flow data is added to a preset data structure, and the processing template corresponding to the business scenario is obtained. The processing template stores the operation logic of the flow data, such as whether a certain attribute value in the flow data needs to be filtered or deduplicated. Then, when valid data is detected to arrive at the Over window, the valid flow data is processed by the operation logic stored in the processing template to obtain the processing result.
[0092] As an example, in a money transfer scenario, the array can be encapsulated using Flink's Over window. The encapsulated array can then be used as a processing window to add valid transfer details to the array. Since the Over window will trigger data processing after new data arrives, after adding valid transfer details to the array, the valid transfer details in the array can be calculated based on the processing template corresponding to the money transfer scenario to obtain the processing result. To facilitate data access, the processing result also needs to be stored in a preset storage area.
[0093] Please refer to Figure 4 , Figure 4This is a data flow diagram illustrating an embodiment of the real-time processing method for risk control flow data of the present invention. Kafka filters the data from the Source and Sink tables and then passes the data to the FlinkOver window. The data is then calculated using function calculation templates (such as total, sum, maximum, average, table information, filtering conditions, and whether to remove duplicates) within the Flink Over window. At this point, the filtering conditions, Source table name, Sink table name, and relevant fields are filled into the preset SQL positions to generate SQL code. The SQL's validity is then verified using Flink's built-in SQL validation function. Next, a Kafka consumer group is created to consume the Flink job data source Kafkatopic and obtain fixed configurations such as job fault recovery strategies and parameter measurement storage. Since different data sources have different traffic volumes and require different memory resources, the memory resource configuration corresponding to the job is obtained based on the source table's corresponding data source. Finally, the job is submitted to the Flink cluster via a RESTful interface for computation, and the computation results are stored in Redis. Users can access the variable query application via HTTP query requests and retrieve variable results from Redis through the variable query application. Furthermore, this invention can also use the Flink-Kafka connector to consume Kafka data, perform over-aggregation operations on input operators, and input the results to Redis for querying by users.
[0094] The Over window triggers a computation each time it receives a stream of data, and the results can be output in real time. Furthermore, the Over window maintains only one copy of the computational data in memory, thus keeping computational resource requirements within a reasonable range even with large data volumes. This invention uses the Over window as the variable aggregation operator, combined with low-latency components such as Redis and Kafka. These components are deployed in a distributed cluster, creating a highly stable, low-latency computational pipeline capable of supporting large data volumes.
[0095] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating the result storage process of an embodiment of the real-time processing method for risk control flow data of the present invention. Before storing the calculation results, the data storage engine needs to be connected via a connector configured in the Flink cluster to access the target database. MySQL performs registration or queries on the Sink table through the Flink table service. The computation job management service can obtain the Sink table from the Flink table via an HTTP request, allowing the Flink table service to output the calculation results to Redis based on the connector configured in the Flink cluster. The connector can be flink-connector-redis.
[0096] Step S30: Based on the data expiration time, the valid stream data in the processing window is judged to be expired to obtain an expiration result, and a new processing result is obtained based on the expiration result, the processing template and the valid stream data, and the new processing result is used to update the processing result.
[0097] After obtaining the processing result, if new valid stream data enters the processing window, the processing window will continue to process the valid stream data in the processing window according to the processing template; after the valid stream data expires, it is necessary to recalculate the other valid stream data in the processing window that has not expired, so as to obtain a new processing result.
[0098] In this embodiment, the valid stream data in the processing window is expired based on its expiration time to obtain an expiration result. Then, the expiration information represented by the expiration result (expired stream data name, expired stream data ID, etc.) is used to calculate a new processing result for the valid stream data that has not expired in the processing window. It should be understood that the calculation for valid stream data that has not expired is still performed using the processing template. After obtaining the new processing result, the previously stored processing result is overwritten.
[0099] As an example, in a transfer scenario, after new valid data streams arrive at the processing window, processing result A will be obtained. When the valid data streams in the processing window are expired based on the expiration time, the non-expired valid data streams in the processing window can also be processed based on the expiration information in the processing window and the processing template of the transfer scenario to obtain a new processing result B. Then, the new processing result B replaces processing result A.
[0100] In this embodiment, compared to traditional real-time streaming data processing methods, since the present invention processes streaming data based on a preset processing template corresponding to the business scenario, it eliminates the need to set up a storage area for custom processing templates, thereby reducing the memory requirements of the real-time streaming data processing method. Furthermore, the present invention achieves real-time updating of streaming data processing results by setting a processing window, determining whether expired streaming data exists in the processing window based on an expiration time, and triggering an update operation on the processing results when expired streaming data exists.
[0101] Based on the first embodiment of the real-time processing method for risk control flow data of the present invention described above, a second embodiment of the real-time processing method for risk control flow data of the present invention is proposed.
[0102] In this embodiment, before step S30 above, the method further includes:
[0103] Step S40: Obtain the current time;
[0104] Based on this, step S30 above includes:
[0105] Step S301: Compare the current time with the data expiration time in the processing window, and if the current time is detected to be greater than or equal to the data expiration time, set the expiration result to expired; otherwise, set the expiration result to not expired.
[0106] It should be noted that, to improve the accuracy of data comparison, time values can be compared by numerical values, or timestamps can be converted for comparison. Furthermore, since there are multiple valid streams of data in the processing window, there are multiple data expiration times. Therefore, the current time needs to be compared with each of these multiple expiration times.
[0107] In this embodiment, before comparing the current time and the data expiration time, the current time and the data expiration time can be converted into timestamps to obtain the current timestamp and the data expiration timestamp. Then, the current timestamp is compared with multiple data expiration timestamps in the processing window. If a current timestamp is found to be greater than or equal to an expiration timestamp, it indicates that there is expired data in the processing window, and the expiration result is set to expired. If the current timestamp is less than each expiration timestamp in the processing window, the expiration result is set to not expired.
[0108] As an example, suppose there are three valid stream data in the processing window, and their expiration times are 2023-09-22 20:55:04, 2023-09-22 21:55:04, and 2023-09-22 22:00:04 respectively. These expiration times, converted to timestamps in seconds, would be 1695387304, 1695390904, and 1695391204 respectively. Assuming the current time is 2023-09-30 21:55:04, the current timestamp would be 1696082104. Then, the current timestamp is compared with the three expiration timestamps. If all three valid stream data are found to be expired, the expiration result is set to expired. If the current actual time is 2023-09-22... If the timer is 21:00:04, the first valid stream data will be detected as expired, and the expired result will be set to expired; if the current timestamp is detected as being less than the above 3 timestamps, the expired result will be set to not expired.
[0109] Optionally, in a feasible embodiment, step S30 above further includes:
[0110] Step S305: When the expired result is detected to be expired, the valid stream data corresponding to the data expiration time in the processing window is deleted, and the valid stream data in the processing window after deletion is processed by the processing template to obtain a new processing result;
[0111] Step S306: When the expired result is detected to be not expired, the processing result is taken as the new processing result.
[0112] It should be noted that processing valid stream data through a processing template refers to processing the valid stream data through various functions and judgment conditions within the processing template.
[0113] When expired data is detected, to prevent it from affecting risk control decisions, the expired valid stream data needs to be deleted from the processing window. Then, the non-expired valid stream data in the processing window after deleting the expired data is processed using a processing template. That is, the non-expired valid stream data is processed using various functions and judgment conditions in the processing template to obtain a new processing result. If the detected expired result is not expired, it means that there is no expired valid stream data in the processing window, and the processing result obtained in the previous processing window can be used as the new processing result.
[0114] As an example, suppose there are 3 valid stream data in the processing window, and the previous processing result is A. When an expired result is detected as expired, the expired valid stream data in the processing window is deleted. The processing result obtained by various functions and judgment conditions in the processing window is B, and B is taken as the new processing result. If the expired result is detected as not expired, A is still taken as the new processing result.
[0115] Please refer to Figure 6 , Figure 6 This is a flowchart of expired data processing according to an embodiment of the real-time processing method for risk control flow data of the present invention. T is a time detection segment, which is the time interval for expiration detection. t1 to t4 are flow data. When it is detected that t4 has expired and there is flow data without valid data in the processing window, the flow data without valid flow data can be deleted together when t4 is deleted.
[0116] Furthermore, in Flink, when data arrives at the processing window, a timer is registered via TimerService to trigger the onTimer function, which processes the data within the window. The processing steps include state cleanup, expired data removal, accumulator calculation, and result output. The expired data removal step removes data whose expiration timestamp is less than the start time of the time window from the accumulated state value. The accumulator calculation step adds newly arriving data to the accumulated state value, yielding the final output result.
[0117] In the Over window aggregation operator, the timer registration and triggering process only occurs when new data arrives or all data expires. To ensure that the calculation result is updated and output is triggered every time a piece of data expires, a `quitTsState` state (exit window status identifier) can be added to the aggregation operator processing class. This queue stores the exit window timestamps, and the exit window timer is registered simultaneously when processing data. The `onTimer` function is modified to include a condition that triggers the calculation when the current timestamp coincides with the timestamp at the head of the queue (the smallest timestamp in the window), only removing expired data and not accumulating new data.
[0118] Additionally, when registering a timer to trigger normal calculations, check if there are any window-out calculation timers with overlapping times. If so, delete the window-out calculation timers to avoid conflicts that could affect normal calculation results.
[0119] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the result output of an embodiment of the real-time processing method for risk control flow data of the present invention. In this embodiment, the data expiration time can be obtained by the data arrival time and the running time of the processing window. A Timer is registered based on the data expiration time, and then the state cleanup or calculation process is activated through the onTimer function to output the calculation result of the current window.
[0120] Optionally, in a feasible embodiment, step S305 above further includes:
[0121] Step S3051: Construct an update job based on the valid stream data in the deleted processing window and the processing template, and execute the update job to obtain the execution result;
[0122] Step S3052: Obtain the job progress of the update job based on the execution result, and obtain a new processing result based on the job progress.
[0123] In this embodiment, when an expired result is detected, the expired valid stream data in the processing window needs to be deleted to obtain the unexpired valid stream data in the processing window. Then, based on each processing position in the processing template, the unexpired valid stream data is matched with that processing position to obtain an update job. This update job is then executed via a preset script file or a background management service, and the execution result is obtained based on whether the update job fails. The job progress is then determined based on the execution result, and the update job is then restored or its status modified according to the progress to obtain a new processing result.
[0124] As an example, suppose the processing template includes an accumulation function that accumulates each detailed data point in the valid stream data. Assuming the accumulation function has multiple parameter positions, all non-expired valid stream data in the processing window are mapped to these parameter positions, resulting in an update job that includes parameter information and calculation logic. This update job is then executed using a pre-defined script. The execution result (success or failure) is then obtained based on the update job's performance. The job progress is then determined from the execution result (recording progress upon failure, indicating completion upon success). Based on the job progress, the update job is either restored (in case of failure) or its status is modified (in case of success), resulting in a new processing result.
[0125] As another example, when processing streaming data using a processing template, it is also possible to construct computational jobs (update jobs). Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a flowchart of a computational job in an embodiment of the real-time processing method for risk control flow data of the present invention. In this embodiment, MySQL can query source tables, result tables, and SQL template information, while the computational job management service can generate job SQL, obtain relevant parameter configurations, generate dynamic resource parameters, and then assemble job submission configurations for processing by the Flink cluster.
[0126] Optionally, in a feasible embodiment, step S3052 above further includes:
[0127] Step X: When the execution result indicates that the update job has failed, the job progress of the update job is stored, and the update job is restored using the job progress to obtain a new processing result;
[0128] Step Y: When the execution result indicates that the update job has been successfully executed, the job progress of the update job is set to completed, and the processing result is used as the new processing result.
[0129] If the execution result is obtained and it is detected that the execution result indicates that the update job has failed, the job progress of the update job is stored, and the update job is restored by using the stored job progress to obtain a new processing result. If the execution result indicates that the update job has been successfully executed, the job progress of the update job is set to completed, and the processing result of the update job with the completed job progress is used as the new processing result.
[0130] Optionally, in one feasible embodiment, the method further includes:
[0131] Step S100: Obtain the interruption time and interruption handling result, wherein the interruption handling result is the handling result when the update job fails to execute;
[0132] It should be noted that the interruption time refers to the time when the update job of the processing window fails to execute. In this embodiment, the format of the time is not restricted.
[0133] In this embodiment, when the update job fails to execute, the time of the failure and the processing result of the update job failure can also be obtained so as to recover the failed update job.
[0134] Based on this, step X above includes:
[0135] Step X1: Locate the unprocessed stream data in the processing window through the job progress, and compare the data expiration time of the unprocessed stream data with the interruption time;
[0136] Step X2: When the data expiration time is detected to be less than or equal to the interruption time, the unprocessed data stream corresponding to the data expiration time in the processing window is deleted, and the unprocessed data stream in the processing window is processed by the processing template to obtain an intermediate result. The intermediate result and the interruption processing result are then merged to obtain a new processing result.
[0137] In this embodiment, since there is a certain time interval between the failure of the update job and the repair of the update job, after obtaining the job progress of the failed update job, it is possible to find the unprocessed stream data in the processing window that has not been processed by the processing template based on the job progress, and compare the expiration time of the unprocessed stream data with the interruption time. If the expiration time is less than or equal to the interruption time, it is possible to detect that there is expired data in the processing window. The expired unprocessed data needs to be deleted, and the unexpired unprocessed stream data in the processing window is processed by the processing template to obtain an intermediate result. The intermediate result and the interruption processing result are then merged to obtain a new processing result.
[0138] As an example, when processing streaming data using a processing template, the streaming data before and after processing can be identified by an identifier, which can then be used to obtain the job progress. After obtaining the job progress, the unprocessed streaming data (e.g., data 1-3) in the processing window can be found using the identifier representing the job progress, and the expiration time of data 1-3 can be compared with the interruption time. If it is detected that the expiration time of data 3 is greater than the interruption time, and the expiration time of data 1-2 is less than the interruption time, then data 1-2 is deleted from the processing window, and data 3 is processed using the processing template to obtain an intermediate result. Then, the interruption processing result and the intermediate result are merged to obtain a new processing result.
[0139] In this embodiment, the present invention processes expired data in the processing window, updates the processing results based on the expired data, and updates the processing results based on the execution status of update jobs. This enables timely processing of real-time streaming data and ensures that the processing results are always up-to-date. Furthermore, since the processing of streaming data is confined to the processing window, the present invention can reduce the memory requirements of the device while processing streaming data.
[0140] In addition, this invention also proposes a real-time processing system for risk control flow data.
[0141] Please refer to Figure 9 The real-time processing system for the risk control flow data includes:
[0142] The data preprocessing module 10 is used to acquire business scenarios and preprocess the risk control flow data of the business scenarios to obtain effective flow data, wherein the effective flow data includes data expiration time;
[0143] The new data triggering module 20 is used to add the valid stream data to a preset processing window, process the valid stream data in the processing window through the processing template corresponding to the business scenario, obtain the processing result, and store the processing result.
[0144] The processing result update module 30 is used to determine the expiration of the valid stream data in the processing window based on the data expiration time, obtain an expiration result, obtain a new processing result based on the expiration result, the processing template and the valid stream data, and update the processing result with the new processing result.
[0145] Optionally, the preprocessing includes: format conversion, data filtering, and data cleaning. The data preprocessing module 10 includes:
[0146] The format conversion unit is used to convert the risk control flow data of the business scenario to obtain the first intermediate flow data, wherein the first intermediate flow data is in JSON format;
[0147] The filtering and cleaning unit is used to filter the first intermediate stream data according to the business scenario to obtain the second intermediate stream data, and to clean the second intermediate stream data to obtain effective stream data.
[0148] Optionally, the real-time processing system for the risk control flow data includes:
[0149] The time acquisition module is used to obtain the current time;
[0150] Processing result update module 30 includes:
[0151] The expiration result judgment unit is used to compare the current time with the data expiration time in the processing window, and set the expiration result to expired when the current time is detected to be greater than or equal to the data expiration time; otherwise, set the expiration result to not expired.
[0152] Optionally, the processing result update module 30 also includes:
[0153] The data expiration unit is used to delete the valid stream data corresponding to the data expiration time in the processing window when the expiration result is detected as expired, and to process the valid stream data in the processing window after deletion through the processing template to obtain a new processing result;
[0154] The data not expired unit is used to take the processing result as the new processing result when the expired result is detected to be not expired.
[0155] Optionally, the data expiration unit includes:
[0156] The job construction subunit is used to construct an update job based on the valid stream data in the deleted processing window and the processing template, and execute the update job to obtain the execution result;
[0157] The progress processing subunit is used to obtain the job progress of the update job based on the execution result, and to obtain a new processing result based on the job progress.
[0158] Optionally, the schedule processing subunit includes:
[0159] The progress recovery unit is used to store the progress of the update job when the execution result indicates that the update job has failed, and to recover the update job by using the progress to obtain a new processing result.
[0160] The progress completion unit is used to set the progress of the update job to "completed" and use the processing result as the new processing result when the execution result indicates that the update job has been successfully executed.
[0161] Optionally, the real-time processing system for risk control flow data includes:
[0162] An interruption information acquisition module is used to acquire the interruption time and interruption processing result, wherein the interruption processing result is the processing result when the update job fails to execute;
[0163] Progress recovery molecular unit, including:
[0164] An interruption confirmation atomic unit is used to find unprocessed stream data in the processing window through the job progress and compare the data expiration time of the unprocessed stream data with the interruption time.
[0165] The data update atomic unit is used to delete the unprocessed data stream corresponding to the data expiration time in the processing window when the data expiration time is detected to be less than or equal to the interruption time, process the unprocessed data stream in the processing window through the processing template to obtain an intermediate result, and merge the intermediate result and the interruption processing result to obtain a new processing result.
[0166] The functions of each module in the real-time processing system for the aforementioned risk control flow data correspond to the steps in the embodiment of the real-time processing method for the aforementioned risk control flow data. Their functions and implementation processes will not be described in detail here.
[0167] Furthermore, the present invention also proposes a real-time processing device for risk control flow data, which includes: a memory, a processor, and a real-time processing program for risk control flow data stored in the memory and executable on the processor. When the real-time processing program for risk control flow data is executed by the processor, it implements the steps of the real-time processing method for risk control flow data of the present invention as described above.
[0168] The specific embodiments of the real-time processing device for risk control flow data of the present invention are basically the same as the embodiments of the real-time processing method for risk control flow data described above, and will not be repeated here.
[0169] Furthermore, the present invention also proposes a storage medium storing a real-time processing program for risk control flow data, which, when executed by a processor, implements the steps of the real-time processing method for risk control flow data of the present invention as described above.
[0170] The specific embodiments of the storage medium of the present invention are basically the same as the embodiments of the real-time processing method of the risk control flow data described above, and will not be repeated here.
[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be an in-vehicle computer, smartphone, computer, or server, etc.) to execute the methods described in the various embodiments of this application.
[0174] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for real-time processing of risk control flow data, characterized in that, The real-time processing method for the risk control flow data includes: Obtain the business scenario and preprocess the risk control flow data of the business scenario to obtain effective flow data, wherein the effective flow data includes data expiration time; The valid stream data is added to a preset processing window, and the valid stream data in the processing window is processed by the processing template corresponding to the business scenario to obtain the processing result, and the processing result is stored. The valid stream data in the processing window is expired based on the data expiration time to obtain an expiration result. A new processing result is obtained based on the expiration result, the processing template, and the valid stream data. The new processing result is then used to update the processing result. The step of obtaining a new processing result based on the expired result, the processing template, and the valid stream data includes: When the expired result is detected as expired, the valid stream data corresponding to the data expiration time in the processing window is deleted, and the valid stream data in the processing window after deletion is processed by the processing template to obtain a new processing result; When the expired result is detected as not expired, the processing result is taken as the new processing result; The step of processing the valid stream data in the deleted processing window using the processing template to obtain a new processing result includes: An update job is constructed based on the valid stream data in the deleted processing window and the processing template, and the update job is executed to obtain the execution result; The update job's progress is obtained based on the execution result, and a new processing result is obtained based on the job progress. The step of obtaining the job progress of the update job based on the execution result, and obtaining a new processing result based on the job progress, includes: When the execution result indicates that the update job has failed, the job progress of the update job is stored, and the update job is restored using the job progress to obtain a new processing result; When the execution result indicates that the update job has been successfully executed, the job progress of the update job is set to completed, and the processing result is used as the new processing result. The method further includes: Obtain the interruption time and interruption handling result, wherein the interruption handling result is the handling result when the update job fails to execute; The step of restoring the update job based on the job progress to obtain a new processing result includes: The unprocessed stream data in the processing window is found through the job progress, and the data expiration time of the unprocessed stream data is compared with the interruption time. When the data expiration time is detected to be less than or equal to the interruption time, the unprocessed data stream corresponding to the data expiration time in the processing window is deleted, and the unprocessed data stream in the processing window is processed by the processing template to obtain an intermediate result. The intermediate result and the interruption processing result are then merged to obtain a new processing result.
2. The real-time processing method for risk control flow data as described in claim 1, characterized in that, The preprocessing includes: format conversion, data filtering, and data cleaning. The step of preprocessing the risk control flow data of the business scenario to obtain effective flow data includes: The risk control flow data of the business scenario is converted into a format to obtain the first intermediate flow data, wherein the first intermediate flow data is in JSON format; Based on the business scenario, the first intermediate stream data is filtered to obtain the second intermediate stream data, and the second intermediate stream data is cleaned to obtain the effective stream data.
3. The real-time processing method for risk control flow data as described in claim 1, characterized in that, Before the step of determining the expiration of valid stream data in the processing window based on the data expiration time to obtain an expiration result, the method further includes: Get the current time; The step of determining the expiration of valid stream data in the processing window based on the data expiration time to obtain an expiration result includes: The current time is compared with the data expiration time in the processing window. If the current time is greater than or equal to the data expiration time, the expiration result is set to expired; otherwise, the expiration result is set to not expired.
4. A real-time processing system for risk control flow data, characterized in that, The real-time processing system for the risk control flow data includes: A data preprocessing module is used to acquire business scenarios and preprocess the risk control flow data of the business scenarios to obtain effective flow data, wherein the effective flow data includes data expiration time; The new data triggering module is used to add the valid stream data to a preset processing window, process the valid stream data in the processing window through the processing template corresponding to the business scenario, obtain the processing result, and store the processing result; The processing result update module is used to determine the expiration of valid stream data in the processing window based on the data expiration time, obtain an expiration result, obtain a new processing result based on the expiration result, the processing template, and the valid stream data, and update the processing result with the new processing result; The processing result update module further includes: The data expiration unit is used to delete the valid stream data corresponding to the data expiration time in the processing window when the expiration result is detected as expired, and to process the valid stream data in the processing window after deletion through the processing template to obtain a new processing result; The data not expired unit is used to take the processing result as the new processing result when the expired result is detected to be not expired. The data expiration unit includes: The job construction subunit is used to construct an update job based on the valid stream data in the deleted processing window and the processing template, and execute the update job to obtain the execution result; The progress processing subunit is used to obtain the job progress of the update job based on the execution result, and to obtain a new processing result based on the job progress; The progress processing subunit includes: The progress recovery unit is used to store the progress of the update job when the execution result indicates that the update job has failed, and to recover the update job by using the progress to obtain a new processing result. The progress completion unit is used to set the progress of the update job to "completed" and use the processing result as the new processing result when the execution result indicates that the update job has been successfully executed. The real-time processing system for the risk control flow data includes: An interruption information acquisition module is used to acquire the interruption time and interruption processing result, wherein the interruption processing result is the processing result when the update job fails to execute; Progress recovery molecular unit, including: An interruption confirmation atomic unit is used to find unprocessed stream data in the processing window through the job progress and compare the data expiration time of the unprocessed stream data with the interruption time. The data update atomic unit is used to delete the unprocessed data stream corresponding to the data expiration time in the processing window when the data expiration time is detected to be less than or equal to the interruption time, process the unprocessed data stream in the processing window through the processing template to obtain an intermediate result, and merge the intermediate result and the interruption processing result to obtain a new processing result.
5. A real-time processing device for risk control flow data, characterized in that, The real-time processing device for risk control flow data includes: a memory, a processor, and a real-time processing program for risk control flow data stored in the memory and executable on the processor. When the real-time processing program for risk control flow data is executed by the processor, it implements the steps of the real-time processing method for risk control flow data as described in any one of claims 1 to 3.
6. A computer storage medium, characterized in that, The computer storage medium stores a real-time processing program for risk control flow data. When the real-time processing program for risk control flow data is executed by the processor, it implements the steps of the real-time processing method for risk control flow data as described in any one of claims 1 to 3.