An ETC real-time blacklist processing method and system based on flink

Abstract

The application discloses an ETC real-time black list processing method and system based on flink, and the method comprises the following steps: S1, a service data source is accessed, when a service occurs, a log content containing service data is generated while the service data is inserted or updated in a Mysql database, and a collection program of a Binlog module is triggered; S2, the service data in the log content is processed by the Binlog module, a message is generated, and the message is written into a Kafka module; S3, the service data is transmitted from the Binlog module to the Kafka module, a data stream is formed, and the data stream is stored in the Kafka module; and S4, a FlinkAQL module sequentially consumes the data stream of the Kafka module, a new data stream is formed, and finally, the result data stream is written into a data table. The application solves the delay problem of three-party data exchange existing in the ETC black list.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a Flink-based ETC real-time blacklist processing method and system. Background Art

[0002] The current ETC blacklist management system in the industry requires that blacklisting and removal operations take effect the next day. If a user finds themselves unable to use the ETC due to unpaid fees, they will have to pay their fees the next day before they can use the ETC normally. Because ETC involves collaboration between the ETC Administration and the bank, each maintains a copy of the bill and a copy of the blacklist. The blacklist system, built by both parties based on traditional batch data processing, is limited in performance. The ETC Administration sends a daily list of vehicle crossings to the bank for deduction. After the deduction is complete, the bank returns the results to the ETC Administration and maintains the blacklist with the ETC Administration. Since only one batch is processed daily, each batch contains ETC deduction details for tens of thousands of vehicles. The bank only notifies the ETC Administration of the addition of a user to the blacklist after the deduction fails. Once the customer has paid the outstanding fees, the bank notifies the ETC Administration of the removal of the user from the blacklist after the deduction is successful.

[0003] Existing technologies have the following technical issues: With existing ETC blacklist technology, ETC service is only blocked after one or two days after a customer's account is blacklisted. This puts banks at risk of capital loss. Similarly, customers wait one or two days after payment before using the ETC function, potentially impacting the user experience. Furthermore, existing batch processing of accounts carries a high cost of failure. Summary of the Invention

[0004] The purpose of the present invention is to provide a flink-based ETC real-time blacklist processing method and system to solve the technical problem of how to process the ETC blacklist in real time.

[0005] The present invention is implemented by adopting the following technical solution: a flink-based ETC real-time blacklist processing method, comprising the following steps:

[0006] S1: Connect to the business data source. When a business occurs, the business data is inserted or updated in the MySQL database, and a log containing the business data is generated, triggering the collection program of the Binlog module.

[0007] S2: The Binlog module processes the business data in the log content, generates a message, and writes it to the Kafka module;

[0008] S3: Business data is transferred from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module.

[0009] S4: The FlinkAQL module consumes the data stream of the Kafka module in sequence to form a new data stream, and finally writes the resulting data stream into the data table.

[0010] Furthermore, the Binlog module is responsible for collecting the vehicle ETC road crossing detailed information table, and the vehicle ETC road crossing detailed information table includes vehicle information and fee information.

[0011] Furthermore, the Kafka module includes data sink TopicA, data sink TopicB and data sink TopicC, among which data sink TopicA is responsible for storing vehicle ETC crossing details information table data, data sink TopicB is responsible for storing ETC customer status data stream information, and data sink TopicC is responsible for storing blacklist status change information exchanged between banks and the high-level management bureau.

[0012] Furthermore, in the process of business data moving from the Binlog module to the Kafka module, each message has strict real-time and sequence characteristics, and a message can only be triggered once. As business continues to occur, due to the characteristics of the message, business data moves from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module. The data stream includes the vehicle ETC crossing details data stream.

[0013] Furthermore, step S4 includes the following data processing tasks:

[0014] Task A: Convert the vehicle ETC toll details data stream into a deduction result data stream after bank deduction.

[0015] Task B, combines customer information to process the deduction result data stream and converts it into the customer ETC status data stream;

[0016] Task C: Continuous processing of customer ETC status data stream;

[0017] Task D: Send SMS reminders to ETC customers and output them to the data sink Topic C.

[0018] Task E uses the data in Topic C as a trigger to update the blacklist information of the Management Bureau in real time.

[0019] Furthermore, the customer ETC status data stream includes the following states: normal, low balance, arrears, blacklisted, and unblacklisted. The customer ETC status data stream needs to be maintained in the data sink Topic B.

[0020] Further, task D is specifically: ETC customers are reminded by short message, with the help of Flink state processing mechanism, so that each customer receives low balance, arrears, blacklist and unblacklist state change reminders every day, and the data output of the first blacklist and unblacklist is filtered out to the data sink Topic C.

[0021] An ETC real-time blacklist processing system based on flink, comprising a Binlog module, a Kafka module and a FlinkAQL module, wherein,

[0022] The Binlog module is responsible for collecting vehicle ETC toll information table, and the vehicle ETC toll information table comprises vehicle information and fee information.

[0023] The Kafka module is responsible for storing the data stream written by the Binlog module.

[0024] The FlinkAQL module sequentially consumes the data stream of the Kafka module, forms a new data stream, and finally writes the result data stream into a data table.

[0025] A computer program product comprising a computer program, which, when executed by a processor, implements the above-mentioned ETC real-time blacklist processing method based on flink.

[0026] A computer-readable storage medium, the storage medium stores a computer program, the computer program is executed by the processor, and the above-mentioned ETC real-time blacklist processing method based on flink is realized.

[0027] The beneficial effects of the present application are that in the present application, the addition of the customer ETC account to the blacklist and the removal of the blacklist are real-time, and even if the customer temporarily pays the fee, the ETC can be used normally again in a very short time, and on the basis of the big data platform and the Flink distributed operation framework, the processing speed is slow during the peak period, and the function of emergency freezing of the customer ETC can be realized.

[0028] The present application solves the delay problem of three-party data exchange existing in ETC blacklist, shortens the business time from 1-2 days to a few seconds, and saves the customer time. Based on the big data platform and the Flink distributed operation framework, the processing speed is only related to the data entering the system at the same time during the peak period, without considering the data backlog caused by the travel peak during the holiday, and the processing speed is greatly improved. During the travel peak of the holiday, the original batch processing time may be extended to 3-4 days and the error probability is very large, and the error will affect all customers for a day. However, the present application processes all data in parallel, without data backlog, and can correct single pen after error, or correct a certain hour for a few minutes, greatly reducing the operation and maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0030] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0032] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0033] The following embodiments of the present invention are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other. Example

[0034] See also Figure 1 , a flink-based ETC real-time blacklist processing method, including the following steps:

[0035] S1: Connect to the business data source. When a business occurs, the business data is inserted or updated in the MySQL database, and a log containing the business data is generated, triggering the collection program of the Binlog module.

[0036] S2: The Binlog module processes the business data in the log content, generates a message, and writes it to the Kafka module;

[0037] S3: Business data is transferred from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module.

[0038] S4: The FlinkAQL module consumes the data stream of the Kafka module in sequence to form a new data stream, and finally writes the resulting data stream into the data table.

[0039] In this embodiment, the Binlog module is responsible for collecting the vehicle ETC road crossing details information table, which includes vehicle information and fee information, and stores it in the data pool TopicA of the Kafka module. The Binlog module collects MySQL logs in real time for analysis, and uses the insert and update operations of the database as event messages. Each operation generates a message, which is sent to the Kafka topic and waits for consumption processing. The Kafka module includes data pool TopicA, data pool TopicB and data pool TopicC, among which data pool TopicA is responsible for storing vehicle ETC road crossing details information table data, data pool TopicB is responsible for storing ETC customer status data flow information, and data pool TopicC is responsible for storing blacklist status change information exchanged between banks and the management bureau. In the process of business data moving from the Binlog module to the Kafka module, each message has strict real-time and sequential requirements, and a message can only be triggered once. As business continues to occur, due to the characteristics of the message, business data moves from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module. The data stream includes the vehicle ETC crossing details data stream.

[0040] In this embodiment, step S4 includes the following data processing tasks:

[0041] Task A, based on the data sink TopicA, converts the vehicle ETC toll details data stream into a deduction result data stream after bank deductions, and stores it in the data sink TopicA_1;

[0042] Task B, based on the data collection Topic A_1, processes the deduction result data stream in combination with customer information and converts it into a customer ETC status data stream. The data is divided into three statuses: "Normal", "Low Balance", "Arrears", "Blacklisted", and "Unblacklisted". This data needs to be maintained in the customer ETC status data collection Topic B;

[0043] Task C, based on the data sink Topic B, continuously processes the customer ETC status data stream and stores the processing results in the MySQL database and Hive database respectively;

[0044] Task D sends SMS reminders to ETC customers based on the data collection topic B. Leveraging Flink's stateful processing mechanism, an algorithm is designed to ensure that each customer only receives daily reminders for status changes such as "low balance," "arrears," "blacklisted," and "unblacklisted." Customers with unchanged status will not receive SMS reminders. The algorithm then filters out the first "blacklisted" and "unblacklisted" data and outputs it to the data collection topic C.

[0045] Task E, based on the data collection Topic C, uses the data in the data collection Topic C as a trigger to update the blacklist information of the High-Level Administration in real time and store the updated results in the MySQL database.

[0046] The present invention also provides a flink-based ETC real-time blacklist processing system, which is used to implement the above-mentioned flink-based ETC real-time blacklist processing method. The system includes a Binlog module, a Kafka module, and a FlinkAQL module, wherein:

[0047] The Binlog module is responsible for collecting the vehicle ETC road crossing detailed information table, which includes vehicle information and fee information;

[0048] The Kafka module is responsible for storing the data stream written by the Binlog module;

[0049] The FlinkAQL module consumes the data stream of the Kafka module in sequence, forms a new data stream, and finally writes the resulting data stream into the data table.

[0050] Specifically, the Binlog module collects MySQL logs in real time for analysis, treating database insert and update operations as event messages. Each operation generates a message and sends it to a Kafka topic for consumption and processing. The Kafka module is a distributed publish-subscribe-based messaging system, often used in real-time data processing systems. The storage medium is a Topic, the write end is called a producer, and the output end is called a consumer. Kafka supports distributed data consumption and can ensure the sequential transmission of data within each partition. The FlinkAQL module is a distributed processing engine used for stateful computing on unbounded data streams, capable of distributed real-time data stream processing.

[0051] The connection and data flow between the Binlog module, Kafka module, and FlinkAQL module are as follows: The Binlog module is the collection system, connecting to the business data source. When business data is inserted or updated in the MySQL database, a log entry containing the business data is generated, triggering the Binlog collection program. The collection program processes the business data in the log and generates a message, which is written to the Kafka module's topic. Each message has strict real-time and order requirements and can only be triggered once. As business transactions continue, due to the nature of messages, business data flows from the Binlog module (source) to the Kafka module (stream), forming a data stream and stored in the Kafka module. The Kafka module is a strictly ordered data stream consisting of messages. The FlinkAQL module must consume the Kafka module's data stream in sequence. The FlinkAQL module processes the data stream to form a new data stream. The FlinkAQL module has multiple tasks, which process the data stream multiple times before ultimately writing the resulting data stream to the data table.

[0052] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned Flink-based ETC real-time blacklist processing method.

[0053] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned Flink-based ETC real-time blacklist processing method.

[0054] Based on the above embodiments, the present invention has at least the following technical effects:

[0055] In the present invention, the blacklisting and removal of customer ETC accounts are both real-time. Even if the customer makes a temporary payment, he or she can use ETC normally again in a very short time. In addition, based on the big data platform and the Flink distributed computing framework, there will be no slow processing speed during peak hours, and the function of emergency freezing of customer ETC can also be realized.

[0056] The present invention solves the delay problem of three-party data exchange in the ETC blacklist, shortening the business that takes 1-2 days to a few seconds, saving customers time. In terms of performance, it is based on the big data platform and the Flink distributed computing framework. The processing speed is only related to the data entering the system at the same time as the peak. There is no need to consider the data backlog caused by the holiday travel peak, and the processing speed is greatly improved. During the holiday travel peak, the original batch processing time may be extended to 3-4 days and the probability of error is very high. If an error occurs, customers of the entire day will be affected. However, the present invention processes all data in parallel, and there will be no data backlog. After an error, corrections can be made for a single transaction, or for a few minutes or an hour, greatly reducing operation and maintenance costs.

[0057] For the sake of simplicity, the aforementioned embodiments are described as a series of actions. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are preferred embodiments, and the actions involved are not necessarily required by this application.

[0058] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Without departing from the spirit and scope of the present invention, modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention should be within the scope of protection of the appended claims.

Claims

1. A flink-based ETC real-time blacklist processing method, characterized in that: The steps include: S1: Connect to the business data source. When a business occurs, the business data is inserted or updated in the MySQL database, and a log containing the business data is generated, triggering the collection program of the Binlog module. S2: The Binlog module processes the business data in the log content, generates a message, and writes it to the Kafka module. The Kafka module includes data sinks TopicA, TopicB, and TopicC. Data sink TopicA is responsible for storing vehicle ETC road crossing details table data, data sink TopicB is responsible for storing ETC customer status data stream information, and data sink TopicC is responsible for storing blacklist status change information exchanged between banks and the Highway Administration. S3: Business data is transferred from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module. S4: The FlinkAQL module consumes the data stream of the Kafka module in sequence, forming a new data stream and eventually writing the resulting data stream into the data table. Step S4 includes the following data processing tasks: Task A: Convert the vehicle ETC toll details data stream into a deduction result data stream after bank deduction. Task B, combines customer information to process the deduction result data stream and converts it into the customer ETC status data stream; Task C: Continuous processing of customer ETC status data stream; Task D: Send SMS reminders to ETC customers and output them to the data sink Topic C. Task E uses the data in Topic C as a trigger to update the blacklist information of the Management Bureau in real time.

2. A flink-based ETC real-time blacklist processing method as claimed in claim 1, characterized in that: The Binlog module is responsible for collecting the vehicle ETC road crossing detailed information table, which includes vehicle information and fee information.

3. A flink-based ETC real-time blacklist processing method as claimed in claim 1, characterized in that: In the process of business data moving from the Binlog module to the Kafka module, each message has strict real-time and sequential requirements, and a message can only be triggered once. As business continues to occur, due to the characteristics of the message, business data moves from the Binlog module to the Kafka module, forming a data stream and stored in the Kafka module. The data stream includes the vehicle ETC crossing details data stream.

4. A flink-based ETC real-time blacklist processing method as claimed in claim 1, characterized in that: The customer ETC status data stream includes the following states: normal, low balance, arrears, blacklisted, and unblacklisted. The customer ETC status data stream needs to be maintained in the data sink Topic B.

5. A flink-based ETC real-time blacklist processing method as claimed in claim 1, characterized in that: Task D is specifically about sending SMS reminders to ETC customers. By leveraging Flink's state-based processing mechanism, each customer receives only daily reminders for low balance, arrears, blacklisting, and unblacklisting status changes. The data for the first blacklisting and unblacklisting status changes is filtered out and output to the data sink Topic C.

6. A flink-based ETC real-time blacklist processing system, used to implement the flink-based ETC real-time blacklist processing method according to any one of claims 1 to 5, characterized in that: Including Binlog module, Kafka module and FlinkAQL module, among which, The Binlog module is responsible for collecting the vehicle ETC road crossing detailed information table, which includes vehicle information and fee information; The Kafka module is responsible for storing the data stream written by the Binlog module; The FlinkAQL module consumes the data stream of the Kafka module in sequence, forms a new data stream, and finally writes the resulting data stream into the data table.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the Flink-based ETC real-time blacklist processing method described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the flink-based ETC real-time blacklist processing method according to any one of claims 1 to 5 is implemented.

Images