A method for real-time data processing visualization based on Spark

Abstract

The present invention provides a method for real-time data processing visualization based on Spark, including: Kafka pushes source data in Oracle to Spark program in real time, Spark program listens to Kafka and consumes data pushed by Kafka in real time, Spark program stores data in mongoDB database, Websocket pushes data in mongoDB database to front-end interface, and real-time data in the front-end is dynamically displayed in charts on the client through Echart framework. Actively pushing data solves the problem of resource waste caused by polling and improves the feasibility of data query operation; and real-time visualization of data makes data more intuitive and convenient to view data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for real-time data processing visualization based on Spark. Background Art

[0002] In the context of the cashless era, the number of merchants joining the third-party payment companies and the number of transactions through the third-party payment platform are increasing day by day, especially the number of daily transactions, which has reached tens of millions. In front of the massive amount of merchant information and transaction data, business personnel need to conduct real-time supervision and management of merchants, agents and branches from multiple dimensions, such as the compliance rate of merchants joining the network and the proportion of risky transactions. When the risk ratio reaches the set threshold, the agents or branches should be inspected to achieve the purpose of controlling risky transactions.

[0003] The current transaction management system only provides T+1 statistics for most business data. Business personnel can only see the compliance transaction information of agents and branches on the next day. Moreover, the statistical results are statically displayed on the client interface in the form of numbers. The data volume is large, the fields are many, and it is not intuitive. It is difficult for business personnel to make quick and accurate judgments. They have to rely on the retrieval of a large amount of data, which greatly increases the workload of business personnel and affects their work efficiency. In the prior art, Ajax is used to poll data, but this method will perform query operations even when the data is not updated. In the case of large data volume, it will cause a lot of resource waste. In today's era of big data, it cannot meet actual needs. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method for real-time data processing visualization based on Spark, which actively pushes data to avoid the waste of resources caused by polling, so as to solve the above technical problems.

[0005] The technical method adopted by the present invention to solve its technical problem is: a method for real-time data processing visualization based on Spark, and the improvement thereof lies in: comprising S1, Kafka pushing source data in Oracle to the Spark program in real time; S2, the Spark program listening to Kafka, and consuming the data pushed by Kafka in real time; S3, the Spark program storing the data in the mongoDB database; S4, Websocket pushing the data in the mongoDB database to the front end; S5, using the Echart framework to dynamically display the real-time data of the front end in charts on the client.

[0006] In the above method, step S2 comprises the following steps:

[0007] S21. The Spark program obtains the updated data pushed by Kafka in real time;

[0008] S22, Spark program consumes the data pushed by Kafka one by one;

[0009] S23. The Spark program encapsulates the last updated time label into the field.

[0010] In the above method, the updated data pushed by Kafka in real time in step S21 includes a merchant table, an online order table, an offline flow table, a risk check list table and a risk merchant table.

[0011] In the above method, step S22 includes the following steps:

[0012] S221, Spark program cleans and removes duplicate data every time it gets a piece of data;

[0013] S222. The Spark program summarizes and aggregates the data.

[0014] In the above method, step S3 comprises the following steps:

[0015] S31. The Spark program stores the aggregated data in the mongoDB database in json format;

[0016] S31. The Spark program inserts the last updated time tag into the mongoDB database.

[0017] In the above method, step S4 comprises the following steps:

[0018] S41. Perform a handshake with the server through the Websocket protocol to generate a data channel between the browser and the server;

[0019] S42. The server pushes the updated data in the mongoDB database to the front-end interface in real time.

[0020] In the above method, the client in step S5 is developed using Node.js.

[0021] In the above method, the client is developed using the single thread and asynchronous I / O of Node.js.

[0022] The beneficial effects of the present invention are as follows: the method of the present invention uses Kafka to push source data in Oracle to the Spark program in real time, uses the Spark program to aggregate the data in real time, actively pushes the data to the mongoDB database after aggregation, pushes the data in the mongoDB database to the front end in real time by Websocket, uses Echart to visualize the real-time data of the front end on the client, and actively pushes data to solve the problem of resource waste caused by polling, and improves the feasibility of data query operations; and the real-time visualization display of data makes the data more intuitive and convenient to view the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Attached Figure 1 Flowchart of the present invention

[0024] Attached Figure 2 for Figure 1 Specific flow diagram of step S2 DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0027] Reference Figure 1The method of the present invention includes: the data generated by the transaction system, operation management system, risk control system, etc. are all present in the Oracle database of each business system, and Kafka pushes the source data in Oracle to the Spark program in real time; the Spark program listens to Kafka and consumes the data pushed by Kafka in real time; then the Spark program stores the summarized data in the mongoDB database in json format, and inserts the last update time tag (last-time-up) into the mongoDB database; Websocket pushes the data in the mongoDB database to the browser (front-end interface), specifically, a handshake is performed with the server through the Websocket protocol, and a data channel is generated between the browser and the server, so that the server can push the updated data in the mongoDB database to the browser (front-end interface) in real time, and it is not necessary to send a request every time data is sent, which can greatly reduce the consumption of network bandwidth; finally, the real-time data pushed to the front end is displayed in a dynamic chart on the client through the Echart framework, and the client is developed by Node.js, and specifically, the client is developed using the single-threaded, asynchronous I / O of Node.js. Kafka is used to push source data in Oracle to the Spark program in real time. The Spark program aggregates the data in real time and actively pushes the aggregated data to the mongoDB database. Websocket pushes the data in the mongoDB database to the browser (front-end interface) in real time. Finally, Echart is used to visualize the real-time data of the front end on the client. Actively pushing data solves the problem of resource waste caused by polling and improves the feasibility of data query operations.

[0028] Reference Figure 2 , the Spark program listens to Kafka and consumes the data pushed by Kafka in real time, which is implemented through the following steps: the Spark program obtains the updated data pushed by Kafak in real time, the Spark program consumes the data pushed by Kafka one by one, and the Spark program cleans and removes duplicate data every time it gets a piece of data, and then summarizes and aggregates it to ensure the accuracy of data aggregation statistics, and encapsulates the last updated time label into the field.

[0029] The method of the present invention uses Kafka to push source data in Oracle to the Spark program in real time, uses the Spark program to aggregate the data in real time, actively pushes the data to the mongoDB database after aggregation, and pushes the data in the mongoDB database to the browser (front-end interface) in real time by Websocket, and uses Echart to visualize the real-time data of the front end on the client. Actively pushing data solves the problem of resource waste caused by polling and improves the feasibility of data query operations; and the real-time visualization display of data makes the data more intuitive and convenient for viewing data.

[0030] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for real-time data processing visualization based on Spark, characterized by: The method comprises the following steps: S1. Kafka pushes source data in Oracle to the Spark program in real time; S2, Spark program monitors Kafka and consumes the data pushed by Kafka in real time; The step S2 comprises the following steps: S21. The Spark program obtains the updated data pushed by Kafka in real time; S22, Spark program consumes the data pushed by Kafka one by one; S23, Spark program encapsulates the last updated time label into the field; The updated data pushed by Kafka in real time in step S21 includes the merchant table, online order table, offline flow table, risk check list table and risk merchant table; The step S22 comprises the following steps: S221, Spark program cleans and removes duplicate data every time it gets a piece of data; S222, Spark program aggregates the data once; S3 and Spark programs store data in the mongoDB database; The step S3 comprises the following steps: S31. The Spark program stores the aggregated data in json format into the mongoDB database; S31, Spark program inserts the last updated time tag into the mongoDB database; S4 and Websocket push data from the mongoDB database to the front end; The step S4 comprises the following steps: S41. Perform a handshake with the server through the Websocket protocol to generate a data channel between the browser and the server; S42, the server pushes the updated data in the mongoDB database to the front-end interface in real time; S5. Use the Echart framework to display the real-time data of the front end in a dynamic chart on the client.

2. The method for real-time data processing visualization based on Spark according to claim 1, characterized in that: The client in step S5 is developed using Node.js.

3. The method for real-time data processing visualization based on Spark as claimed in claim 2, characterized in that: The client is developed using the single thread and asynchronous I / O of Node.js.

Images