A fast real-time early warning method based on Spark Streaming

Abstract

The application provides a quick real-time early warning method based on Spark Streaming, wherein the vehicle track data to be processed is stored in different storage partitions according to license plate numbers, all real-time monitoring events are respectively established with a real-time comparison thread, and the vehicle track data to be processed is sent into all real-time comparison threads for parallel calculation each time; meanwhile, because the vehicle track data to be processed is stored according to license plate numbers, the data is sent into the corresponding vehicle data cache queue of the storage partition by the vehicle cache queue consumption thread each time, and in each real-time comparison thread, the data of the corresponding license plate number range is extracted from the basic database and the suspect vehicle blacklist according to the license plate range corresponding to the vehicle track data to be processed, so that the hit rate of data comparison and search is improved, the calculation efficiency of the system is greatly improved, and the calculation speed is further improved because the amount of data to be calculated in each thread each time is limited.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic control, and in particular to a fast real-time early warning method based on Spark Streaming. Background Art

[0002] By the end of June 2022, China had a total of 406 million motor vehicles, including 310 million cars; 492 million motor vehicle drivers, including 454 million car drivers. In the first half of 2022, 16.57 million new motor vehicles were registered nationwide, and 11.03 million new drivers were licensed. This generates massive amounts of complex and dynamically changing traffic data in real time every day.

[0003] At the same time, traffic management demands swift investigation and handling of various traffic incidents, enabling large-scale vehicle searches and early warning interceptions, and improving highway and road management capabilities. For example, it's necessary to find and locate specific vehicles in real time within a city or nationwide, or to identify vehicles suspected of license or permit violations in real time, and then track, warn, and intercept them. However, existing big data computing methods for traffic data, due to the sheer volume and real-time requirements, place extremely high demands on server performance, leading to prohibitive costs. Summary of the Invention

[0004] In order to solve the problem that the existing big data computing methods for traffic data calculation have too high performance requirements for servers, the present invention provides a fast real-time early warning method based on Spark Streaming, which can reduce the requirements for server performance on the basis of meeting the computing requirements, thereby reducing system costs.

[0005] The technical solution of the present invention is as follows: a fast real-time early warning method based on Spark Streaming, characterized in that it includes the following steps:

[0006] S1: Determine the vehicle trajectory data that needs to be processed in real time, and record it as the vehicle trajectory data to be processed;

[0007] S2: In the Kafka system, a vehicle passing topic is created for the vehicle trajectory data to be processed.

[0008] The to-be-processed vehicle trajectory data is stored in different storage partitions of the passing vehicle topic according to the license plate number;

[0009] S3: Based on the Spark Streaming distributed stream processing framework, the warning comparison processing program is distributed to each worker node. Each worker node starts the executor to run the warning comparison processing program;

[0010] The relationship between the working node and the executor is 1:n, where n is a positive integer greater than or equal to 1;

[0011] S4: The early warning comparison processing program determines all real-time monitoring events to be calculated in real time, as well as all basic database data and suspect vehicle blacklist data used in each real-time monitoring event;

[0012] The basic database is a database containing basic vehicle information used when comparing vehicle information in the real-time monitoring event;

[0013] The suspected vehicle blacklist records the suspected vehicles that need to be found and warned in the real-time monitoring event;

[0014] S5: Each of the early warning comparison processing procedures includes all the real-time monitoring events processed in parallel;

[0015] Each of the real-time monitoring events corresponds to an early warning rule and a real-time comparison thread;

[0016] Each of the real-time monitoring events corresponds to a warning cache queue as data output. After the real-time comparison thread finds the vehicle trajectory data to be processed that meets the conditions, it pushes it to the warning cache queue;

[0017] S6: Build the vehicle passing cache queue consumption thread and the warning cache queue consumption thread;

[0018] S7: After each of the warning comparison processing programs is started, the license plate number range corresponding to the real-time calculated vehicle data of this calculation is obtained, which is recorded as: license plate extraction range;

[0019] According to the license plate extraction range, data corresponding to the license plate number range is extracted from the basic database and the suspected vehicle blacklist, which are respectively recorded as: comparison basic data and comparison blacklist data;

[0020] S8: The early warning comparison processing program obtains the data involved in this calculation from the passing vehicle topic according to the preset data extraction amount, which is recorded as real-time calculated vehicle data; encapsulates the real-time calculated vehicle data into a discrete data stream DStream structure, and passes it to all the real-time comparison threads in the early warning comparison processing program for comparison calculation;

[0021] The real-time comparison thread performs collision comparison on the input vehicle data to be processed, the comparison basic data and the comparison blacklist data according to the warning rule corresponding to the monitoring event, and finds the vehicle data to be processed that meets the warning conditions and pushes it to the warning cache queue;

[0022] S9: The warning cache queue consumer thread reads data from the warning cache queue and publishes warning information.

[0023] It is further characterized by:

[0024] It also includes step S10: saving the offset of the vehicle topic partition P corresponding to the real-time calculated vehicle data participating in each calculation;

[0025] In step S2, the method for storing the to-be-processed vehicle trajectory data specifically includes the following steps:

[0026] a1: In the Kafka system, create a vehicle passing topic for the vehicle trajectory data to be processed;

[0027] Assume that the total number of partitions included in the passing vehicle topic is Z, where Z is a natural number;

[0028] a2: Take out each piece of the to-be-processed vehicle trajectory data one by one and confirm whether there is a license plate number;

[0029] If there is no license plate number, perform step a3 to calculate the corresponding storage partition;

[0030] Otherwise, record the license plate number as: the license plate number to be calculated, and implement step a4;

[0031] a3: Based on the random function, calculate an integer between [0, (Z-1)], which is the partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed, where P is a natural number less than or equal to Z;

[0032] Execute step a5;

[0033] a4: Calculate the hash value H corresponding to the license plate number to be calculated based on the hash function. The partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed is:

[0034] P = H mod Z;

[0035] a5: storing the text data corresponding to the vehicle trajectory data to be processed into the corresponding storage partition P;

[0036] The number of the executors is greater than or equal to 1;

[0037] The corresponding relationship between the storage partition and the executor is: m:1, where m is a natural number greater than or equal to 1.

[0038] The present invention provides a fast real-time early warning method based on Spark Streaming, which stores the vehicle trajectory data to be processed into different storage partitions according to the license plate number, establishes a real-time comparison thread for each real-time monitoring event, and simultaneously sends the vehicle trajectory data to all real-time comparison threads for parallel calculation, thereby improving processing efficiency; at the same time, because the vehicle trajectory data to be processed is stored according to the license plate number, each time the calculation is performed, the vehicle cache queue consumer thread sends the data to the vehicle data cache queue corresponding to the storage partition, and in each real-time comparison thread, because the license plate range corresponding to the vehicle trajectory data to be compared is extracted from the basic database and the suspect vehicle blacklist according to the license plate range, the hit rate of data comparison search is improved, greatly improving the calculation efficiency of the system, and at the same time, the amount of data required to be calculated each time in each thread is limited, further improving the calculation speed. Based on the technical solution of the present invention, not only can the real-time early warning requirements be met, but also the performance requirements for the server are reduced. At the same time, the method reduces system cost by improving server resource utilization and concurrent processing capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the real-time early warning method implemented based on Spark Streaming in the present invention;

[0040] Figure 2 This is a flowchart of the real-time comparison thread in the executor of the present invention;

[0041] Figure 3 A flowchart of an embodiment of the present invention providing rapid real-time push of early warning data;

[0042] Figure 4 Schematic diagram of the positional relationship between the bayonet device and the vehicle in the embodiment;

[0043] Figure 5 Schematic diagram of the logical relationship between the passing vehicle topic and the three storage partitions in the embodiment;

[0044] Figure 6 Schematic diagram of the process of the executor reading data based on the offset in an embodiment. DETAILED DESCRIPTION

[0045] like Figures 1 to 3 As shown, the present invention includes a fast real-time early warning method based on Spark Streaming, which is characterized by comprising the following steps:

[0046] S1: Determine the vehicle trajectory data that needs to be processed in real time, and record it as the vehicle trajectory data to be processed.

[0047] In specific implementation, obtaining the vehicle trajectory data to be processed includes three processes: data collection, data cleaning and data aggregation.

[0048] Based on unified data format requirements, front-end checkpoint systems, electronic police, and other equipment are responsible for collecting text and images of passing vehicles, completing the data collection process. Existing technologies are used to handle missing fields, fill in default values, and verify license plate number rules to complete the data cleansing process. Due to the mismatch between data collection and processing speeds, which can lead to continuous data backlogs and delays in back-end business data push, Kafka messaging middleware is added as a buffer to facilitate data aggregation. Messages are encrypted, compressed, and sent to the corresponding partitions based on the distribution strategy to prevent data skew in Kafka.

[0049] like Figure 4 As shown in the figure, 6 vehicles passed through checkpoints 1 and 2 respectively. The text data and images collected by the checkpoints correspond to 12 pieces of vehicle passing data, as shown in Table 1 below:

[0050] Table 1: Example of vehicle passing data collected at the checkpoint

[0051]

[0052] After data cleaning, missing fields are processed, default values ​​are filled, and license plate number rules are verified. In the data in Table 1, if the license plate number is empty, it is filled with "-", and if the license plate type is empty, it is filled with "41" (41 means no license plate). For details, please refer to Table 2 below:

[0053] Table 2: Example of cleaned data:

[0054]

[0055] S2: In the Kafka system, create a vehicle passing topic for the vehicle trajectory data to be processed.

[0056] The vehicle trajectory data to be processed is stored in different storage partitions of the passing vehicle topic according to the license plate number.

[0057] In step S2, the method for storing the vehicle trajectory data to be processed specifically includes the following steps:

[0058] a1: In the Kafka system, create a vehicle passing topic for the vehicle trajectory data to be processed;

[0059] Assume that the total number of partitions included in the topic of passing vehicles is Z, where Z is a natural number;

[0060] a2: Take out each piece of vehicle trajectory data to be processed one by one to confirm whether there is a license plate number;

[0061] If there is no license plate number, perform step a3 to calculate the corresponding storage partition;

[0062] Otherwise, record the license plate number as: the license plate number to be calculated, and implement step a4;

[0063] a3: Based on the random function, calculate an integer between [0, (Z-1)], which is the partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed. P is a natural number less than or equal to Z;

[0064] Execute step a5;

[0065] a4: Calculate the hash value H corresponding to the license plate number to be calculated based on the hash function. The partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed is:

[0066] P = H mod Z;

[0067] a5: Store the text data corresponding to the vehicle trajectory data to be processed into the corresponding storage partition P.

[0068] For example, if Z is set to 10, then, using an existing random function, an integer between [0, 9] is calculated for each unnumbered license plate, which serves as the storage partition number P for the unnumbered license plate data. This ensures that unnumbered vehicles are evenly stored in the storage partition of the passing vehicle topic. In actual applications, all unnumbered license plates are generally included in the unnumbered vehicle warning in the business logic. In this embodiment, the specific calculation results are shown in Table 3 below:

[0069] Table 3 Example of calculating partition P based on license plate

[0070] License plate number Total number of partitions Z Hash value H Partition P Unlicensed 10 - Random number [0-9] Su A1111 10 33797 7 Su A1112 10 33798 8

[0071] The data in Table 2 is stored in the topic of passing vehicles. In this embodiment, Z=3. The corresponding partition number is calculated based on the license plate number field in the vehicle trajectory text data, and the corresponding vehicle trajectory text data is stored in the corresponding partition. Refer to Table 4:

[0072] Table 4: Example of storing the to-be-processed vehicle trajectory data in the vehicle passing topic:

[0073]

[0074] The logical diagram of the aggregated data in the Kafka system is as follows: Figure 5As shown in the figure, the vehicle passing topic includes three storage partitions: partition 0, partition 1, and partition 2. The process data to be processed is stored in the three storage partitions of the vehicle passing topic according to the license plate number. When reading, the data is read according to the offset corresponding to each vehicle trajectory data to be processed.

[0075] S3: If Figure 1 As shown in the figure, based on the Spark Streaming distributed streaming processing framework, the warning comparison processing program is distributed to each worker node, and a single worker node starts the executor to run the warning comparison processing program;

[0076] The relationship between the worker node and the executor is 1:n, where n is a positive integer greater than or equal to 1.

[0077] In specific implementation, the number of executors is greater than or equal to 1; the corresponding relationship between storage partitions and executors is: m:1, where m is a natural number greater than or equal to 1; the specific corresponding relationship between the number of executors and storage partitions is determined according to the number of servers and computing requirements.

[0078] S4: The early warning comparison processing program determines all real-time monitoring events for real-time calculation, as well as all basic database data and suspect vehicle blacklist data used in each real-time monitoring event;

[0079] The basic database is a database containing basic vehicle information used when comparing vehicle information in real-time monitoring events;

[0080] The suspected vehicle blacklist records suspected vehicles that need to be found and warned during real-time monitoring events.

[0081] S5: Each early warning comparison processing program includes all real-time monitoring events processed in parallel;

[0082] Each real-time monitoring event corresponds to an early warning rule and a real-time comparison thread;

[0083] Each real-time monitoring event corresponds to an early warning cache queue as data output. After the real-time comparison thread finds the vehicle trajectory data that meets the conditions, it is pushed to the early warning cache queue.

[0084] S6: Build the vehicle passing cache queue consumption thread and the warning cache queue consumption thread.

[0085] S7: After each early warning comparison processing program is started, the license plate number range corresponding to the real-time calculated vehicle data of this calculation is obtained, which is recorded as: license plate extraction range;

[0086] According to the license plate extraction range, data corresponding to the license plate number range is extracted from the basic database and the suspect vehicle blacklist, which are respectively recorded as: comparison basic data and comparison blacklist data.

[0087] S8: The early warning comparison processing program extracts data from the passing vehicle topic according to the preset data extraction volume, which is recorded as real-time vehicle data. The real-time vehicle data is encapsulated into a discrete data stream DStream structure and passed to all real-time comparison threads in the early warning comparison processing program for comparison calculation.

[0088] The real-time comparison thread performs collision comparison on the input vehicle data to be processed, the comparison basic data and the comparison blacklist data according to the warning rules corresponding to the monitoring event, and finds the vehicle data to be processed that meets the warning conditions and pushes it to the warning cache queue.

[0089] In the specific implementation, Kafka's simple consumer API is used to read data. Kafka is used as the data source, Spark Streaming and Kafka are integrated, and batch consumption message list batch mode is enabled to transmit data. After the offsetRange (topic, partition, starting address from Offset, ending address until Offset) metadata of the batch data is transmitted to the Spark driver, the batch task is triggered, and the executor uses the pull mode to directly pull the data within the corresponding offset range. Figure 1 In the Direct mode marked in the figure, data is sent to the executor in parallel, business processing begins, and suspicious vehicles are quickly compared and analyzed in real time.

[0090] Attachment Figure 5 In the embodiment shown, based on the Spark Streaming vehicle consumption task, three executors (executors 1 to 3) are configured to consume and process the three partitions of the vehicle topic. The executors and partitions are consumed in a 1:1 mode.

[0091] When reading data, obtain the consumption offset of the partition, that is, the latest offset saved after the previous batch of data is processed;

[0092] If offset > 0, data needs to be fetched from the middle of the partition. The data for that partition is fetched starting at the specified offset, resulting in a DStream. If not, data needs to be fetched from the very beginning of the partition. The data for that partition is fetched starting at the latest offset, resulting in a DStream. For specific implementation steps, refer to the consumer API read method.

[0093] In this embodiment, the specified offset is a number greater than 0. Figure 6 , executor1 pulls data from partition 0 (offset 859-869), executor2 pulls data from partition 1 (offset 866-869), and executor3 pulls data from partition 2 (offset 867-869).

[0094] The real-time comparison thread performs collision comparison on the input vehicle data to be processed, the comparison basic data and the comparison blacklist data according to the warning rules corresponding to the monitoring event, and finds the vehicle data to be processed that meets the warning conditions and pushes it to the warning cache queue.

[0095] When the system is running and the warning comparison process is started, it first checks whether the basic data is loaded. If not, it loads the corresponding data for the license plate number range into memory. It then checks whether the blacklist of suspected vehicles is loaded. If not, it loads the corresponding data for the license plate number range into memory. It then checks whether the consumer threads for the vehicle cache queue and the warning cache queue are running. If not, it starts the corresponding threads. After ensuring that both threads are running, it traverses and processes the DStream data to compare and analyze suspected vehicles.

[0096] In this embodiment, the basic database includes: databases of departments, roads, personnel, checkpoints, system internal parameters, etc.; the suspected vehicle blacklist includes: databases of overdue annual inspection, overdue scrapping, should be processed but not processed, local and superior search data, etc.

[0097] like Figure 2 As shown in the figure, during the comparison process, after the decrypted pending vehicle data is pushed to the vehicle passing cache queue, real-time comparison threads (labeled in the figure: Comparison Thread 1 to Comparison Thread N) batch-write the data into a pre-built vehicle passing table in HBase. Each thread initiates a real-time warning comparison thread based on its own warning rules. If the warning conditions are met, the thread pushes the warning data to the warning cache queue. In other words, the thread batch-writes the warning data that meets the conditions to the HBase warning table. If the data does not meet the warning conditions, the comparison is terminated immediately.

[0098] S9: While the real-time comparison thread thread is operating, the warning cache queue consumption thread reads data from the warning cache queue in real time and publishes the warning information to Redis.

[0099] As Figure 3 shown, on the application side, the application side background subscribes to the data of the Redis warning topic and pushes it to the web page in real time based on WebSocket, ensuring that the warning data can be published in real time and meeting the requirements of real-time warning.

[0100] In this embodiment, each warning comparison processing program traverses and processes each vehicle passing text data in the DStream data for comparison. For example, for the text data of the vehicle trajectory data to be processed in Table 5 below, assuming there are problems that need to be warned:

[0101] Table 5 Text Data Embodiment of Vehicle Trajectory Data to be Processed

[0102]

[0103] The comparison process in the real-time comparison thread thread is as follows:

[0104] thread1. Comparing un-inspected vehicles, the license plate number Shanghai *A333D exists in the overdue un-inspected blacklist database, generating an overdue un-inspected warning.

[0105] thread2. Comparing un-scrapped vehicles, the license plate number Shanghai *A333D exists in the overdue un-scrapped blacklist database, generating an overdue un-scrapped warning.

[0106] thread3. Comparing red-eye buses, it is judged that the vehicle information of Shanghai *A333D is tourist passenger transport, but the passing time is not between 2 am and 5 am, so no red-eye bus warning is generated.

[0107] thread4. Comparing fatigue driving, pulling the trajectory of Shanghai *A333D in the previous 4 hours, judging whether it has continuously driven for more than 4 hours, meeting the condition for generating a fatigue driving warning.

[0108] thread5. Comparing local and superior search data, if the vehicle exists in the search data, a warning of the corresponding search type (fields from the search data) is generated.

[0109] thread6......

[0110] S10: Every time the early warning comparison processing program processes a batch of data, it saves the offset of the vehicle topic partition corresponding to the real-time calculated vehicle trajectory data participating in each calculation; so that the next batch of data processing can start pulling data after the offset of the current batch of data, thereby ensuring zero data loss during the processing process. By saving the offset, it ensures that each data can be consumed once and only once.

[0111] After using the technical solution of the present invention, it is possible to process vehicle trajectory data in real time, with high throughput and fault tolerance, and to accurately and in real time issue warnings for suspicious vehicles. The technical solution of the present invention performs data batch processing based on the Spark Streaming framework. Spark Streaming is a coarse-grained, quasi-real-time processing framework that processes data after reading it once or asynchronously, and its calculations can be performed based on large memory, thus having high throughput. At the same time, during the business processing process, when obtaining data from the vehicle passing topic, the offset data will be pulled. After the data processing is completed in the real-time comparison thread, the offset information is reliably stored. Once the processing fails due to an accident, these offset information can be directly read when it is restored, ensuring zero loss during the data processing process and consuming it once and only once. During the comparison and analysis of suspicious vehicles, multi-threaded comparison is used in the warning comparison processing program to ensure that the comparison logic of each type of suspicious warning does not affect each other. The warning data is written to HBase and published to Redis in an asynchronous manner to ensure that once a warning is found, it can be pushed to the application end in real time.

Claims

1. A fast real-time early warning method based on Spark Streaming, characterized in that: It includes the following steps: S1: Determine the vehicle trajectory data that needs to be processed in real time, and record it as the vehicle trajectory data to be processed; S2: In the Kafka system, a vehicle passing topic is created for the vehicle trajectory data to be processed. The to-be-processed vehicle trajectory data is stored in different storage partitions of the passing vehicle topic according to the license plate number; S3: Based on the Spark Streaming distributed stream processing framework, the warning comparison processing program is distributed to each worker node. Each worker node starts the executor to run the warning comparison processing program; The relationship between the working node and the executor is 1:n, where n is a positive integer greater than or equal to 1; S4: The early warning comparison processing program determines all real-time monitoring events to be calculated in real time, as well as all basic database data and suspect vehicle blacklist data used in each real-time monitoring event; The basic database is a database containing basic vehicle information used when comparing vehicle information in the real-time monitoring event; The suspected vehicle blacklist records the suspected vehicles that need to be found and warned in the real-time monitoring event; S5: Each of the early warning comparison processing procedures includes all the real-time monitoring events processed in parallel; Each of the real-time monitoring events corresponds to an early warning rule and a real-time comparison thread; Each of the real-time monitoring events corresponds to a warning cache queue as data output. After the real-time comparison thread finds the vehicle trajectory data to be processed that meets the conditions, it pushes it to the warning cache queue; S6: Build the vehicle passing cache queue consumption thread and the warning cache queue consumption thread; S7: After each of the warning comparison processing programs is started, the license plate number range corresponding to the real-time calculated vehicle data of this calculation is obtained, which is recorded as: license plate extraction range; According to the license plate extraction range, data corresponding to the license plate number range is extracted from the basic database and the suspected vehicle blacklist, which are respectively recorded as: comparison basic data and comparison blacklist data; S8: The early warning comparison processing program obtains the data involved in this calculation from the passing vehicle topic according to the preset data extraction amount, which is recorded as real-time calculated vehicle data; encapsulates the real-time calculated vehicle data into a discrete data stream DStream structure, and passes it to all the real-time comparison threads in the early warning comparison processing program for comparison calculation; The real-time comparison thread performs collision comparison on the input vehicle data to be processed, the comparison basic data and the comparison blacklist data according to the warning rule corresponding to the monitoring event, and finds the vehicle data to be processed that meets the warning conditions and pushes it to the warning cache queue; S9: The warning cache queue consumer thread reads data from the warning cache queue and publishes warning information.

2. The Spark Streaming-based rapid real-time early warning method according to claim 1, characterized in that: It also includes step S10: saving the offset of the vehicle topic partition P corresponding to the real-time calculated vehicle data participating in the calculation each time.

3. The Spark Streaming-based rapid real-time early warning method according to claim 1, characterized in that: In step S2, the method for storing the to-be-processed vehicle trajectory data specifically includes the following steps: a1: In the Kafka system, create a vehicle passing topic for the vehicle trajectory data to be processed; Assume that the total number of partitions included in the passing vehicle topic is Z, where Z is a natural number; a2: Take out each piece of the to-be-processed vehicle trajectory data one by one and confirm whether there is a license plate number; If there is no license plate number, perform step a3 to calculate the corresponding storage partition; Otherwise, record the license plate number as: the license plate number to be calculated, and implement step a4; a3: Based on the random function, calculate an integer between [0, (Z-1)], which is the partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed, where P is a natural number less than or equal to Z; Execute step a5; a4: Calculate the hash value H corresponding to the license plate number to be calculated based on the hash function. The partition number P corresponding to the storage partition corresponding to the vehicle trajectory data to be processed is: P = H mod Z; a5: Storing the text data corresponding to the to-be-processed vehicle trajectory data into the corresponding storage partition P.

4. The Spark Streaming-based rapid real-time early warning method according to claim 1, characterized in that: The number of the executors is greater than or equal to 1; The corresponding relationship between the storage partition and the executor is: m:1, where m is a natural number greater than or equal to 1.

Images