Data processing method based on pre-credit granting stage of credit project

Abstract

The invention discloses a data processing method based on a credit project pre-credit granting stage. The data processing method comprises the following steps: collecting real-time data and offline data; data circulation transmission is carried out through the Kafka message middleware; performing hierarchical storage in the circulation transmission process; data layering processing is carried out, processed data of each layer is transmitted to Kafka, and a real-time result is obtained; and transmitting the real-time result to a service layer. The Flink + Kafka + Hudi combined architecture is introduced in the pre-credit granting stage of the credit project, so that off-line data and real-time data can be combined, and the problem that the cost of maintaining the two sets of architectures is too high is avoided.

Description

Technical Field

[0001] The present invention relates to the field of credit data processing, and in particular to a data processing method based on the pre-credit approval stage of a credit project. Background Art

[0002] At present, the pre-credit stage of credit projects in the financial industry mostly processes data in real time or offline, which meets the needs of some enterprises. However, with the development of enterprises and increasing needs, processing real-time data or offline data alone can no longer meet the growing business needs.

[0003] At present, the pre-credit stage of credit projects is mostly based on Java, Flink, Spark or SQL to collect and transfer data, so as to support the determination of decision-making strategies and risk control rules. The usual technical architecture is only a single Java, or a real-time architecture design of Flink combined with Kafka, or a quasi-real-time architecture design of Spark and Kafka, and a hierarchical design of data warehouses performed by offline HSQL, etc. The present invention discloses a real-time data processing method based on the pre-credit stage, which adopts the real-time processing engine Flink and the data lake Hudi storage, and uses the Kafka message middleware to simultaneously process the required real-time and offline data in the pre-credit stage of the credit project, thereby ensuring that the credit project can be applied to a wider range of more complex and more demanding business scenarios.

[0004] Disadvantages of existing technology:

[0005] Using Spark, Java, and HSQL alone to process credit project data can only process single real-time data or offline data, and cannot meet the requirements of both real-time data and offline data participation in the business.

[0006] Maintaining two sets of engines at the same time, real-time and offline, high operation and maintenance costs

[0007] Real-time offline Two platforms need to maintain two sets of frameworks, but the business logic is the same, the code development cost is high, the data has two different links, which easily causes data inconsistency

[0008] Although Flink can process real-time and offline data, it is highly dependent on message middleware. Message middleware such as Kafka cannot support massive data storage and does not support update / upsert.

[0009] Since Kafka needs to save historical data, it has natural deficiencies in batch processing.

[0010] The scenario is not universal, and the backtracking cost of large amounts of data is high

[0011] In view of the drawbacks of the existing architecture of these credit projects, the invention discloses a method for real-time data processing of credit projects. Summary of the invention

[0012] In view of the above problems, the present invention is proposed to provide a data processing method based on the pre-credit approval stage of a credit project, which overcomes the above problems or at least partially solves the above problems.

[0013] According to one aspect of the present invention, a data processing method based on the pre-credit approval stage of a credit project is provided, the data processing method comprising:

[0014] Collect real-time data and offline data;

[0015] Data is transferred and transmitted through Kafka message middleware;

[0016] Performing hierarchical storage during the flow transmission process;

[0017] Perform data layer processing and transfer each layer of processed data to Kafka to obtain real-time results;

[0018] The real-time results are transmitted to the service layer.

[0019] Optionally, the collecting of real-time data and offline data specifically includes:

[0020] The real-time data and the offline data are collected in the business log or business database through Flink's CDC technology.

[0021] Optionally, the data flow transmission through Kafka message middleware specifically includes:

[0022] The data collected through Flink's CDC technology is transmitted and transferred through the Kafka message middleware, and the data is transferred to the downstream for further circulation.

[0023] Optionally, performing hierarchical storage during the streaming transmission specifically includes:

[0024] While data is flowing, the data in the Kafka messages of each layer is stored in Hudi for hierarchical storage.

[0025] Optionally, performing data layer processing and transferring each layer of processed data to Kafka to obtain real-time results specifically includes: while the data is flowing, using FlinkSQL to perform data layer processing through the business logic of the pre-credit stage of the credit project, and transferring each layer of processed data to Kafka.

[0026] Optionally, the transmitting the real-time result to the service layer specifically includes: transmitting the final real-time result to the service layer through Flink, and also transmitting the near real-time result in Hudi to the service layer.

[0027] Optionally, the data processing method further includes: displaying the data on a front-end interface through the service layer.

[0028] The present invention provides a data processing method based on the pre-credit stage of a credit project, the data processing method comprising: collecting real-time data and offline data; transferring data through Kafka message middleware; performing layered storage during the transfer process; performing layered data processing, transferring each layer of processed data to Kafka to obtain real-time results; transferring the real-time results to the service layer. By introducing the Flink+Kafka+Hudi combined architecture in the pre-credit stage of a credit project, it is possible to combine offline data with real-time data and avoid the problem of high cost of maintaining two sets of architectures.

[0029] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 A flow chart of a data processing method based on the pre-credit approval stage of a credit project provided by an embodiment of the present invention;

[0032] Figure 2 A specific architecture implementation diagram of a data processing method based on the pre-credit approval stage of a credit project provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0036] In order to meet the needs of various business scenarios and avoid the shortcomings of Lambda and Kappa architectures, the present invention uses Hudi as a unified storage layer for batches and streams, and finally decides to adopt the Flink+Kafka+Hudi architecture solution.

[0037] like Figure 1 As shown, a data processing method based on the pre-credit approval stage of a credit project includes:

[0038] Real-time data and offline data come from business logs or business databases. First, Flink's CDC technology is used to efficiently and accurately collect real-time data or offline data.

[0039] The data collected by Flink's CDC technology is transmitted and transferred through the Kafka message middleware. Due to the real-time and consistency of Kafka, the data can be transferred to the downstream for further circulation.

[0040] While data is flowing, the data in the Kafka messages of each layer needs to be stored in Hudi for hierarchical storage.

[0041] While the data is flowing, FlinkSQL is used to process the data in layers through the business logic of the pre-credit approval stage of the credit project, and the processed data at each layer is transferred to Kafka.

[0042] The final real-time results are passed to the service layer through Flink, and the near real-time results in Hudi are also passed to the service layer.

[0043] Finally, the data is displayed on the front-end interface through the service layer.

[0044] At this point, the data transmission and processing of the pre-credit approval stage of the credit project is completed, and the problem that cannot be solved by single Flink, Spark, Java or HSQL is solved. The specific architecture implementation diagram is as follows Figure 2 shown.

[0045] Beneficial effects: Flink is used as a real-time computing framework because Flink has an API for processing basic offline data. Hudi is used as the storage layer to effectively avoid the inability to perform effective update / upsert operations. Hudi itself does not generate any business data. It can access relational databases, logs, and message queues through tools such as Spark and Flink for large-capacity storage, and finally provide it to the upper-level engine for query. At the same time, Hudi can support many common components of big data. This advantage greatly meets the needs of various business scenarios of credit projects. By introducing the Flink+Kafka+Hudi combined architecture in the pre-credit stage of the credit project, it can not only combine offline data with real-time data, avoiding the problem of high cost of maintaining two sets of architectures, but also has a very strong versatility in business scenarios, bringing more benefits to enterprises.

[0046] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method based on the pre-credit stage of a credit project, characterized in that: The data processing method comprises: Collect real-time data and offline data; Data is transferred and transmitted through Kafka message middleware; Performing hierarchical storage during the flow transmission process; Perform data layer processing and transfer the processed data of each layer to Kafka to obtain real-time results; The real-time results are transmitted to the service layer.

2. A data processing method based on the pre-credit stage of a credit project according to claim 1, characterized in that: The collection of real-time data and offline data specifically includes: The real-time data and the offline data are collected in the business log or business database through Flink's CDC technology.

3. A data processing method based on the pre-credit stage of a credit project according to claim 1, characterized in that: The data transmission through Kafka message middleware specifically includes: The data collected through Flink's CDC technology is transmitted and transferred through the Kafka message middleware, and the data is transferred to the downstream for further circulation.

4. A data processing method based on the pre-credit stage of a credit project according to claim 1, characterized in that: The layered storage during the flow transmission specifically includes: While data is flowing, the data in the Kafka messages of each layer is stored in Hudi for hierarchical storage.

5. The data processing method based on the pre-credit stage of a credit project according to claim 1 is characterized in that: The data is processed in layers, and each layer of processed data is transferred to Kafka to obtain real-time results. Specifically, while the data is flowing, FlinkSQL is used to process the data in layers through the business logic of the pre-credit stage of the credit project, and each layer of processed data is transferred to Kafka.

6. A data processing method based on the pre-credit stage of a credit project according to claim 1, characterized in that: The transmitting the real-time result to the service layer specifically includes: transmitting the final real-time result to the service layer through Flink, and also transmitting the near real-time result in Hudi to the service layer.

7. A data processing method based on the pre-credit stage of a credit project according to claim 1, characterized in that: The data processing method also includes: displaying the data on a front-end interface through the service layer.

Images