General decision log real-time library falling method based on flink

Abstract

The invention relates to a general decision log real-time library falling method based on flink, and the method comprises the steps: firstly, writing a newly-added log of a risk control decision system into kafka in real time by using an OGG tool, merging a decision configuration input parameter field and a non-decision input parameter field, synchronizing the merged fields to a target configuration table, and then writing the merged fields into a kafka specific topic; then, target configuration table data are spliced into a long character string by using flink, and the long character string is written into an oracle dimension table; and finally, the flink program associates the kafka log data with the oracle dimension table in real time, dynamically analyzes key information, generates a json field and writes the json field into a Dores analysis library, so that real-time and efficient library falling of the log is realized, and the problems of log content redundancy, low development efficiency and poor data timeliness in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data stream processing, and in particular to a real-time storage method for general decision logs based on Flink. Background Art

[0002] The risk control decision system will call a large number of data interfaces from both inside and outside, and each interface itself has a large number of interface fields. The risk control decision system will record all node logs of the decision flow in the decision log general table in JSON format. However, since this table will record all fields in the external call interface, regardless of whether they are used in the decision, the JSON content is very large, which makes it impossible to analyze and use directly, and the batch processing performance cannot keep up. In order for the downstream to be able to use the analysis log data, the risk control decision system needs to set up a separate table for the logs of each different channel, and since the policy parameters of each platform are inconsistent, each platform needs to be customized. Finally, the data of the data warehouse T+1 is synchronized to the analysis library for business use. Therefore, there are major problems in development efficiency and data timeliness.

[0003] In the risk control decision system, in order to fully record and track the decision process, the system will call a large number of data interfaces from both inside and outside. These interfaces usually contain a lot of field information, which is used to describe the data and status related to various decisions. In order to maintain the transparency and traceability of the decision, the risk control decision system will record the logs of all nodes in the decision flow in detail in JSON format and store them in a unified decision log table.

[0004] At present, risk control decision systems usually process decision logs in the following way: all decision-related data, regardless of whether it is actually used, will be recorded in the decision log master table. This results in the JSON format content being extremely large and containing a lot of redundant information. In order to meet the needs of downstream analysis systems, the risk control decision system needs to generate a separate log table for each different channel or platform. This is because the strategy parameters and focus of each platform are different, requiring customized log content. The generated customized log table will be synchronized to the analysis library through the data warehouse's T+1 (next day) data synchronization mechanism for business personnel to conduct subsequent data analysis and mining.

[0005] Although the existing risk control decision log processing method meets business needs to a certain extent, it has the following defects: 1) Performance issues: Since the decision log total table contains a large amount of field information, the table size is huge and the query and analysis performance is poor. When running batch processing, the system resources are consumed hugely and the processing time is long. 2) Low development efficiency: In order to generate customized log tables, developers need to write specific code and logic for each platform. This not only increases the development workload, but also reduces the reusability and maintainability of the code. 3) Poor data timeliness: Due to the use of the T+1 data synchronization mechanism, there is a one-day delay between the data in the analysis library and the real-time decision data. This is unacceptable for business scenarios that require real-time insights into decision effects and optimization strategies.

[0006] The existing risk control decision log processing methods have obvious problems and defects in performance, development efficiency and data timeliness. These problems limit the further optimization and upgrading of the risk control decision system and also affect the decision-making efficiency and accuracy of business personnel. Summary of the invention

[0007] In view of this, it is necessary to provide a real-time storage method for general decision logs based on Flink to solve the above-mentioned defects of the prior art.

[0008] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a real-time storage method for general decision logs based on Flink, comprising:

[0009] Use the OGG tool to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time;

[0010] Use the OGG tool to synchronize the decision configuration input parameter fields in the risk control decision system configuration table to the target configuration table in real time, add the non-decision input parameter fields to the target configuration table, and merge the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table;

[0011] Write the merged data in the target configuration table into a specific topic of Kafka in real time through the OGG tool;

[0012] Use Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string and write it into the Oracle dimension table;

[0013] The log data in Kafka is associated with the Oracle dimension table in real time through the flink program, key information is obtained by dynamic parsing, and JSON fields are generated and written into the Doris analysis library.

[0014] Preferably, the use of the OGG tool to write the newly added data of the original decision log table oracle in the risk control decision system database into kafka in real time includes:

[0015] Use the OGG tool to capture new call logs from the original decision log table of the risk control decision system database Oracle in real time;

[0016] Convert the newly added call logs into JSON format and write them to a fixed topic of Kafka.

[0017] Preferably, after using the OGG tool to write the newly added data of the original decision log table oracle in the risk control decision system database into kafka in real time, the method further includes:

[0018] Configure the Flink task to subscribe to the Kafka log as the mainstream data processed by Flink.

[0019] Preferably, the method of using flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string includes:

[0020] Use the UDAF function of Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string; UDAF is a user-defined aggregation function.

[0021] Preferably, the writing into the Oracle dimension table includes:

[0022] The changed data in the target configuration table is captured and processed by Flink in real time; when new parameters are added to the target configuration table, Flink updates the long string in the Oracle dimension table in real time.

[0023] Preferably, the log data in Kafka is associated with the Oracle dimension table in real time through the flink program, key information is obtained by dynamic parsing, JSON fields are generated and written into the Doris analysis library, specifically including:

[0024] The flink program associates the decision log data in kafka with the oracle dimension table in real time;

[0025] The flink program dynamically obtains the fields to be parsed from the oracle dimension table, and parses the decision configuration input parameters and non-decision input parameters from the decision log data in real time;

[0026] Write the parsed information into the Doris analysis library in json field format.

[0027] In a second aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0028] The memory is used to store programs;

[0029] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the real-time storage method for general decision logs based on Flink as described in the embodiment of the first aspect of the present invention.

[0030] In a third aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the real-time storage method for general decision logs based on Flink as described in the embodiment of the first aspect of the present invention.

[0031] Compared with the prior art, the real-time storage method for general decision logs based on Flink provided by the present invention has the following beneficial effects:

[0032] 1) The present invention utilizes the high throughput and low latency characteristics of flink to collect and process the decision logs generated by the risk control decision system in real time. The flink task directly accesses the decision log summary table, reads the data in real time, performs necessary conversion and processing, and immediately writes the processing results to the log analysis library table of each platform. This improves the traditional T+1 log storage time to seconds, significantly enhancing the real-time performance of the data.

[0033] 2) The present invention configures the different storage requirements of each platform, and there is no need to write independent code for each platform to handle its specific storage requirements. These requirements are stored in the form of configuration in a configuration table or configuration file. When it is necessary to process logs of different platforms, the system only needs to read the corresponding configuration and parse and process the data according to the configuration.

[0034] 3) Through real-time configuration reading and Flink's real-time processing capabilities, the present invention implements the function of processing logs of multiple platforms with one set of code. This not only greatly reduces the development workload, but also improves the reusability and maintainability of the code. Developers no longer need to write independent code for each platform, but only need to focus on the writing and maintenance of the configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of risk control decision log analysis scenario provided by existing technology;

[0036] Figure 2 A flow chart of the real-time storage method for general decision logs based on Flink provided by the present invention;

[0037] Figure 3 The architecture diagram of the real-time storage method for general decision logs based on Flink provided by the present invention;

[0038] Figure 4 The architecture diagram of the general decision log real-time storage system based on Flink provided by the present invention;

[0039] Figure 5 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] Figure 1 Schematic diagram of risk control decision log analysis scenario provided by existing technology, refer to Figure 1 At present, the risk control decision system usually processes the decision log in the following way: all decision-related data, whether or not it is actually used, will be recorded in the decision log master table. This results in the JSON format content being extremely large and containing a lot of redundant information. In order to meet the needs of the downstream analysis system, the risk control decision system needs to generate a separate log table for each different channel or platform. This is because the strategy parameters and focus of each platform are different, and customized log content is required. The generated customized log table will be synchronized to the analysis library through the T+1 (next day) data synchronization mechanism of the data warehouse for business personnel to conduct subsequent data analysis and mining.

[0043] Although the existing risk control decision log processing method meets business needs to a certain extent, it has the following defects: First, the decision log summary table records the logs of all nodes in the decision flow, including a large number of external interface fields that are not actually used by the decision, resulting in huge JSON content, which is difficult to analyze and use directly, and the batch processing performance is also affected. Secondly, in order to meet the needs of downstream analysis, the system needs to store data separately for different channels and platforms, and due to differences in policy parameters, each platform requires customized processing, which greatly increases the development workload. Finally, data synchronization to the analysis library depends on the T+1 process of the data warehouse, which leads to a lag in data timeliness and cannot meet real-time analysis needs. In summary, the existing technology has problems such as content redundancy, low development efficiency and poor data timeliness in log processing. A more efficient, flexible and real-time log processing solution is urgently needed to optimize and improve it.

[0044] In view of this, the present invention provides a real-time storage method for general decision logs based on Flink, which uses Flink technology to access the original decision log summary table to achieve streaming and real-time data processing. The present invention configures the different storage requirements of each platform, and does not need to write independent code for each platform to handle its specific storage requirements. Through real-time reading of configuration and Flink's real-time processing capabilities, the present invention implements the function of processing multiple platform logs with one set of code, greatly reducing the development workload. The following will be explained and introduced through multiple embodiments.

[0045] Figure 2 The flowchart of the real-time storage method for general decision logs based on Flink provided by the present invention is as follows: Figure 3 This is an architecture diagram of the real-time storage method for general decision logs based on Flink provided by the present invention. Figure 2 The real-time storage method of general decision logs based on Flink includes the following steps:

[0046] Step S1, using the OGG tool, write the new data of the original decision log table oracle in the risk control decision system database into kafka in real time.

[0047] OGG (Oracle GoldenGate) is a data replication and integration tool used to capture, convert and transmit data changes between Oracle databases and other systems. In this embodiment, Oracle database is the core database used by the risk control decision system to store and manage these decision logs.

[0048] In this embodiment, the OGG tool is first used to capture the newly added call log from the original decision log table of the risk control decision system database Oracle in real time. Then, the newly added call log is converted into json format and written to the fixed topic of kafka ( Figure 3 kafka topic1) for subsequent processing.

[0049] Kafka is a distributed stream processing platform. In the present invention, Oracle's original decision log table is used as the data source, and the newly added call log is written into the fixed Topic of Kafka in real time in JSON format through the OGG (Oracle GoldenGate) tool. Kafka, as a message queue, ensures the real-time transmission and efficient processing of log data. In Kafka, a topic is a specific data stream or channel used to organize and classify messages.

[0050] The format of the decision log written to Kafka is as follows. Generally, a decision log contains more than 4000 key-value pairs:

[0051] {"param1":"value1",

[0052] "param2":"value2",

[0053] "param3":"value3",

[0054] "param4":"value4", ...

[0056] "param4000":"value4000"}

[0057] Furthermore, after using the OGG tool to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time, configure the flink task to subscribe to the Kafka log as the mainstream data processed by flink. Among them, flink is an open source computing platform for distributed data stream processing and batch data processing.

[0058] The present invention utilizes the high throughput and low latency characteristics of flink to collect and process the decision logs generated by the risk control decision system in real time. The flink task directly accesses the decision log summary table, reads the data in real time, performs necessary conversion and processing, and immediately writes the processing results to the log analysis library table of each platform. This improves the traditional T+1 log storage time to seconds, significantly enhancing the real-time performance of the data.

[0059] Step S2: Use the OGG tool to synchronize the decision configuration input parameter fields in the risk control decision system configuration table to the target configuration table in real time, add the non-decision input parameter fields to the target configuration table, and merge the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table.

[0060] Specifically, the risk control decision system configuration table is a table that stores the input parameters required for risk control system decisions, and is used as a reference for the risk control system when performing risk assessment and decision-making.

[0061] Decision configuration input parameters refer to the variables actually used for decision flow calculation configured in the risk control system according to the different storage requirements of each platform. Decision configuration input parameters come from the risk control decision system configuration table.

[0062] "Decision configuration input parameter fields" refer to the specific fields that constitute "decision configuration input parameters". Decision configuration input parameters play a key role in the decision flow, and they determine which data will be used for decision calculations. At the same time, during real-time processing, the Flink task will dynamically parse the relevant fields in the log data based on these decision configuration input parameter fields and non-decision input parameter fields.

[0063] In this embodiment, the OGG tool is used to copy the decision input parameter fields in the risk control system configuration table to another target configuration table in real time. This means that when the data in the original configuration table changes, these changes will be immediately reflected in the target configuration table.

[0064] In the embodiment of the present invention, by synchronizing data to another configuration table in real time, the online processing part (i.e., the real-time decision-making part) of the risk control system is decoupled from the data processing part (i.e., the management and maintenance of the configuration table). Since the online system directly relies on the synchronized target configuration table rather than the original configuration table, when the original configuration table needs to be updated, maintained, or repaired, these operations will not have a direct impact on the operation of the online system. This improves the stability and availability of the system.

[0065] It is understandable that in the decision log of the risk control decision system, in addition to the fields that serve as decision input parameters, there may be some other fields, namely non-decision input parameter fields. Although these fields are not directly involved in the decision process, they may be important for business analysis or monitoring. In order to ensure that these non-input parameter fields can also be recorded and stored, the system allows users or administrators to add the information of these fields to the configuration table in some way, such as manually inserting them into the target configuration table.

[0066] Furthermore, the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table are merged so as to integrate all relevant field information together for subsequent processing and analysis.

[0067] The configuration table plays a role of a bridge in the present invention, which connects the original decision log table and the target data warehouse (such as Doris analysis library). Through the configuration table, the system can know which fields need to be extracted and stored, even if these fields are not direct inputs in the decision process.

[0068] Step S3, write the merged data in the target configuration table into a specific topic of Kafka in real time through the OGG tool.

[0069] In this embodiment, after merging the decision configuration input parameter field and the non-decision input parameter field in the target configuration table, the merged data in the target configuration table is written to a specific topic of Kafka in real time through the OGG tool ( Figure 3 In kafka topic2), the flink task can subscribe to this topic and process the data in real time. The merged data is shown in Table 1.

[0070] Table 1

[0071] Input parameter type Input parameter name Decision configuration input param1 Decision configuration input param3 Manually configure input parameters param4

[0072] Step S4: Use Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string and write it into the Oracle dimension table.

[0073] Specifically, use Flink's UDAF (User-Defined Aggregate Function) to concatenate multiple columns of data that change in real time in the target configuration table into a long string. UDAF is a user-defined aggregate function. Through UDAF, these real-time changing column data can be concatenated into a long string according to a certain logic (such as comma separation). This long string will contain all the field names that need to be parsed and will be used for field extraction in subsequent steps.

[0074] The concatenated long character string is written into the Oracle dimension table, which is used as a real-time dimension table to provide necessary dimension information in the subsequent data processing flow. The embodiment of the present invention utilizes the powerful storage and query capabilities of Oracle, while ensuring the persistence and consistency of data.

[0075] The changed data in the target configuration table is captured and processed by Flink in real time; when new parameters are added to the target configuration table, Flink updates the long string in the Oracle dimension table in real time to ensure that the subsequent data processing process can use the latest field information.

[0076] Through the processing of step S3, the parameters in the target configuration table are converted into the format of "param1, param2, param3, param4". Due to the real-time processing capability of Flink, when a new parameter (such as param5) is added to the upstream configuration table, this long string can be updated in real time to "param1, param2, param3, param4, param5". This real-time update capability ensures that the system can reflect changes in the configuration table in a timely manner, thereby maintaining the accuracy and integrity of the data.

[0077] Step S5, through the flink program, the log data in kafka is associated with the oracle dimension table in real time, key information is obtained by dynamic parsing, json fields are generated and written into the Doris analysis library.

[0078] Specifically, refer to Figure 2 and Figure 3 , the flink program will subscribe to the decision log data in Kafka as the mainstream input. In this embodiment, first, the flink program associates the decision log data in Kafka with the oracle dimension table in real time.

[0079] Then, the flink program dynamically obtains the fields to be parsed from the oracle dimension table, and parses the decision configuration input parameters and non-decision input parameters from the decision log data in real time;

[0080] Finally, the parsed information is written into the Doris analysis library in json field format. Doris is an efficient column storage analysis database suitable for real-time analysis and query of large-scale data. In this embodiment, by writing key information into the Doris analysis library in real time, the system can provide timely and accurate data support for downstream businesses.

[0081] Figure 3 The architecture diagram of the real-time storage method for general decision logs based on Flink provided by the present invention is shown in Figure 1. Figure 3 According to the annotation in, in a preferred embodiment of the present invention, the real-time storage method of general decision logs based on Flink may include the following 9 steps:

[0082] 1. Use the OGG (Oracle GoldenGate) tool to capture the newly added call logs from Oracle's original decision log table in real time and write them to Kafka topic 1 in JSON format.

[0083] 2. Configure the flink task to subscribe to Kafka topic1 as the main input source for the data processing flow, ready to be associated with the data in subsequent steps.

[0084] 3. Use the OGG tool to synchronize the decision configuration input parameter fields of the risk control decision system configuration table to the target configuration table in real time.

[0085] 4. For non-decision input parameter fields, you can manually insert these fields into the target configuration table for subsequent processing.

[0086] 5. Merge the decision-making configuration input parameters in the configuration table and the manually configured non-decision-making input parameters, and write them to Kafka topic2 in real time through the OGG tool for further processing by the Flink task.

[0087] 6. Use Flink's UDAF (user-defined aggregate function) function to concatenate multiple columns of data that change in real time into a long string and write it into an Oracle table as a real-time dimension table. This step ensures real-time capture of input parameter changes and the timeliness of data.

[0088] 7. The flink task converts the parameters of the target configuration table into the format of "param1, param2, param3, param4..." and updates them in real time.

[0089] 8. The flink program associates the large JSON log data in Kafka with the dimension table generated in step 6 in real time, dynamically parses the required fields, and generates new JSON fields.

[0090] 9. Write the processed json fields into the Doris analysis library for business use. Through specific functions (such as json_muti_get), extract the decision configuration input parameters and business-related variables from the json data of the original log.

[0091] Figure 4 This is the architecture diagram of the real-time storage system for general decision logs based on Flink provided by the present invention. Figure 4 The system integrates OLTP (online transaction processing) and OLAP (online analytical processing) environments to support efficient and accurate decision-making processes. The system includes:

[0092] Risk control decision system: Located at the core of the OLTP environment, it is responsible for processing real-time decision tasks. The risk control decision system interacts closely with the decision log table to record key information in the decision-making process in real time to ensure the timeliness and accuracy of the data.

[0093] Decision log summary table: As a central storage unit, it is located between the OLTP and OLAP environments and is responsible for receiving and storing decision logs from the risk control decision system. These logs are the basis for subsequent analysis and support real-time acquisition and backtracking of data.

[0094] Real-time acquisition mechanism: This mechanism ensures that the log data in the decision log table can be quickly extracted and transmitted to the subsequent processing module. Through efficient real-time processing technology, the system can maintain sensitivity to the latest decision logs and provide support for rapid response.

[0095] Flink dynamic analysis: In the OLAP environment, Flink, as the core processing engine, dynamically analyzes the log data obtained in real time. Flink's stream processing capabilities enable the system to process large amounts of log data in real time and generate structured, easy-to-analyze data formats, providing strong support for subsequent data analysis.

[0096] Platform A / B / C decision log analysis tables: These tables store data dynamically parsed by Flink, supporting the decision log analysis requirements of platforms A, B, and C. Each table is optimized for the business logic of a specific platform, supports multi-dimensional data query and analysis, and provides accurate data support for decision-making on different platforms.

[0097] The general decision log real-time storage system based on Flink provided by the present invention realizes the full process coverage from real-time decision-making to deep data analysis by integrating OLTP and OLAP environments. The risk control decision system ensures real-time and accuracy, the decision log summary table provides central storage and backtracking capabilities, the real-time acquisition mechanism and Flink dynamic analysis ensure the rapid processing and structuring of data, and finally supports complex decision analysis requirements through the platform A / B / C decision log analysis library table. This design enables the system to efficiently process large amounts of real-time data and provide powerful data analysis capabilities to meet the decision support needs of different business platforms.

[0098] In summary, the real-time storage method and system for general decision logs based on Flink provided by the present invention have the following beneficial effects compared with the prior art:

[0099] 1) The present invention utilizes the high throughput and low latency characteristics of flink to collect and process the decision logs generated by the risk control decision system in real time. The flink task directly accesses the decision log summary table, reads the data in real time, performs necessary conversion and processing, and immediately writes the processing results to the log analysis library table of each platform. This improves the traditional T+1 log storage time to seconds, significantly enhancing the real-time performance of the data.

[0100] 2) The present invention configures the different storage requirements of each platform, and there is no need to write independent code for each platform to handle its specific storage requirements. These requirements are stored in the form of configuration in a configuration table or configuration file. When it is necessary to process logs of different platforms, the system only needs to read the corresponding configuration and parse and process the data according to the configuration.

[0101] 3) Through real-time configuration reading and Flink's real-time processing capabilities, the present invention implements the function of processing logs of multiple platforms with one set of code. This not only greatly reduces the development workload, but also improves the reusability and maintainability of the code. Developers no longer need to write independent code for each platform, but only need to focus on the writing and maintenance of the configuration.

[0102] Figure 5 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 5 As shown, the present invention also provides an electronic device, and the electronic device 500 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device 500 includes a processor 501 and a memory 502, wherein the memory 502 stores a general decision log real-time storage program 503 based on Flink.

[0103] In some embodiments, the memory 502 may be an internal storage unit of a computer device, such as a hard disk or memory of a computer device. In other embodiments, the memory 502 may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory 502 may also include both an internal storage unit of a computer device and an external storage device. The memory 502 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 502 can also be used to temporarily store data that has been output or is to be output. In one embodiment, when the flink-based general decision log real-time storage program 503 is executed by the processor 501, the following steps are implemented:

[0104] Use the OGG tool to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time;

[0105] Use the OGG tool to synchronize the decision configuration input parameter fields in the risk control decision system configuration table to the target configuration table in real time, add the non-decision input parameter fields to the target configuration table, and merge the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table;

[0106] Write the merged data in the target configuration table into a specific topic of Kafka in real time through the OGG tool;

[0107] Use Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string and write it into the Oracle dimension table;

[0108] The log data in Kafka is associated with the Oracle dimension table in real time through the flink program, key information is obtained by dynamic parsing, and JSON fields are generated and written into the Doris analysis library.

[0109] In some embodiments, the processor 501 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 502 or process data, such as executing a general decision log real-time storage program based on Flink.

[0110] This embodiment further provides a computer-readable storage medium, on which a general decision log real-time storage program based on Flink is stored. When the general decision log real-time storage program based on Flink is executed by a processor, the following steps are implemented:

[0111] Use the OGG tool to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time;

[0112] Use the OGG tool to synchronize the decision configuration input parameter fields in the risk control decision system configuration table to the target configuration table in real time, add the non-decision input parameter fields to the target configuration table, and merge the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table;

[0113] Write the merged data in the target configuration table into a specific topic of Kafka in real time through the OGG tool;

[0114] Use Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string and write it into the Oracle dimension table;

[0115] The log data in Kafka is associated with the Oracle dimension table in real time through the flink program, key information is obtained by dynamic parsing, and JSON fields are generated and written into the Doris analysis library.

[0116] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time storage method for general decision logs based on Flink, characterized in that: include: Use the OGG tool to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time; Use the OGG tool to synchronize the decision configuration input parameter fields in the risk control decision system configuration table to the target configuration table in real time, add the non-decision input parameter fields to the target configuration table, and merge the decision configuration input parameter fields and non-decision input parameter fields in the target configuration table; Write the merged data in the target configuration table into a specific topic of Kafka in real time through the OGG tool; Use Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string and write it into the Oracle dimension table; The log data in Kafka is associated with the Oracle dimension table in real time through the flink program, key information is obtained by dynamic parsing, and JSON fields are generated and written into the Doris analysis library.

2. According to the flink-based general decision log real-time storage method of claim 1, it is characterized by: The OGG tool is used to write the new data of the original decision log table oracle in the risk control decision system database into Kafka in real time, including: Use the OGG tool to capture new call logs from the original decision log table of the risk control decision system database Oracle in real time; Convert the newly added call logs into JSON format and write them to a fixed topic of Kafka.

3. According to the flink-based general decision log real-time storage method of claim 1, it is characterized in that: After using the OGG tool to write the newly added data of the original decision log table oracle in the risk control decision system database into kafka in real time, the method further includes: Configure the Flink task to subscribe to the Kafka log as the mainstream data processed by Flink.

4. According to the flink-based general decision log real-time storage method of claim 1, it is characterized in that: The method of using Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string includes: Use the UDAF function of Flink to concatenate multiple columns of data that change in real time in the target configuration table into a long string; UDAF is a user-defined aggregation function.

5. According to the flink-based general decision log real-time storage method of claim 4, it is characterized in that: The writing into the Oracle dimension table includes: The changed data in the target configuration table is captured and processed by Flink in real time; when new parameters are added to the target configuration table, Flink updates the long string in the Oracle dimension table in real time.

6. The real-time storage method for general decision logs based on Flink according to claim 3 is characterized in that: The flink program is used to associate the log data in kafka with the oracle dimension table in real time, dynamically parse to obtain key information, generate json fields and write them into the Doris analysis library, specifically including: The flink program associates the decision log data in kafka with the oracle dimension table in real time; The flink program dynamically obtains the fields to be parsed from the oracle dimension table, and parses the decision configuration input parameters and non-decision input parameters from the decision log data in real time; Write the parsed information into the Doris analysis library in json field format.

7. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the real-time storage method for general decision logs based on Flink as described in any one of claims 1 to 6 above.

8. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the real-time storage method for general decision logs based on Flink as described in any one of claims 1 to 6 above.

Images