Method for realizing low-delay convergence of large-scale positioning data across different network intervals by using Kafka

Abstract

The invention relates to the technical field of big data processing and communication, in particular to a method for realizing low-delay convergence of large-scale positioning data across different network intervals by using Kafka, which comprises the following steps: firstly, transmitting position information of terminal equipment to a mobile police service network for convergence, and after the mobile police service network gathers position information of a plurality of terminal equipment, transmitting the position information of the terminal equipment to the mobile police service network; according to the method, a public security network boundary is passed through Kafka to a public security network user domain Kafka, the public security network user domain gathers positioning data from a mobile police service network and city platforms, the positioning data are pushed to a public security network T area through a Kafka cluster boundary, and the public security network T area realizes position rendering and data storage of all positioning data. And further, external service support is provided for each service system by relying on the micro-service platform, so that low-delay convergence of the large-scale positioning data across different network intervals is realized, and the technical problem of relatively high delay during processing of data convergence of the large-scale positioning data across the network intervals in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing and communication technology, and in particular to a method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka. Background Art

[0002] With the rapid development of Internet of Things technology, in the era of big data and Internet of Things, positioning data plays an important role in many fields. Positioning data refers to the location information of equipment or objects obtained through positioning systems (such as GPS, Beidou, etc.); for example, in the public security system, by using positioning data, it is possible to more effectively maintain social order, combat criminal activities, and improve the efficiency and accuracy of police work. Vehicle positioning data can also be used to monitor traffic flow and road conditions in real time, providing a scientific basis for traffic management and guidance. The above functions all require the aggregation and processing of positioning data to achieve.

[0003] Traditional methods have the problem of high latency when processing large-scale positioning data aggregation across network intervals. Summary of the invention

[0004] The purpose of the present invention is to provide a method for low-latency aggregation of large-scale positioning data across different network intervals using Kafka, aiming to solve the technical problem of high latency in the prior art when processing data aggregation of large-scale positioning data across network intervals.

[0005] To achieve the above purpose, the present invention adopts a method for low-latency aggregation of large-scale positioning data across different network intervals using Kafka.

[0006] The following steps are involved:

[0007] Step 1: The terminal device location information is transmitted to the mobile police network for collection;

[0008] Step 2: After the mobile police network collects the location information of several terminal devices, it passes through the public security network boundary to the public security network user domain Kafka through Kafka;

[0009] Step 3: The user domain of the public security network aggregates the location data from the mobile police network and local city platforms, and then pushes it across the border to the T zone of the public security network through the Kafka cluster;

[0010] Step 4: The T zone of the public security network realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to various business systems, thereby realizing low-latency aggregation of large-scale positioning data across different network intervals.

[0011] Among them, the specific implementation method of transmitting the terminal device location information to the mobile police network for collection is:

[0012] After the device's own positioning information is corrected for position accuracy by differential services, it is transmitted from the Internet side to the mobile police network through the government cloud to achieve location aggregation on the mobile police network side;

[0013] At the same time, the certified devices are directly connected to the mobile police network, and their built-in SDK analyzes the differential data transmitted from the Internet side to optimize the positioning accuracy;

[0014] In addition, equipment with high-precision positioning capabilities uses Beidou inertial navigation technology + 5G base station assisted positioning technology to directly achieve high-precision positioning effects.

[0015] Among them, after the mobile police network collects the location information of several terminal devices, the specific implementation method of passing through the public security network boundary to the public security network user domain Kafka through Kafka is as follows:

[0016] Build a mobile police network location aggregation platform, Kafka cluster, Zookeeper cluster and Nginx cluster in the mobile police network;

[0017] The mobile police network location aggregation platform aggregates terminal data from the Internet and mobile police network, and simultaneously responds to the needs of the mobile police network business system for positioning information;

[0018] After the mobile police network location aggregation platform aggregates the information, it transmits the data from the mobile police network to the public security network user domain in real time through the Kafka cluster.

[0019] The Zookeeper cluster is used to register the Kafka cluster and monitor the working status of the Kafka cluster.

[0020] Among them, the Nginx cluster is used to achieve load balancing of business system requests on the mobile police network side.

[0021] Among them, the specific implementation method of the public security network user domain to aggregate the positioning data from the mobile police network and local municipal platforms, and then push it across the boundary to the public security network T zone through the Kafka cluster is:

[0022] The public security network user domain is deployed with the public security network city data processing platform, Kafka cluster, Zookeeper cluster and Nginx cluster;

[0023] The location data pushed from the mobile police network and the location data reported by the city-level location platform on the public security network are aggregated through the city-level data processing platform of the public security network, so as to realize the cross-network and full coverage integration and aggregation of the positioning data of various terminals in the covered area, and then carry out standardized processing of the location data, and then push it into the T zone of the public security network using Kafka.

[0024] Among them, the T zone of the public security network realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to various business systems, thereby realizing the low-latency aggregation of large-scale positioning data across different network intervals. The specific implementation method is:

[0025] A comprehensive positioning service platform and external service platform are built in Zone T of the public security network. They are deployed in containers and integrate gateway clusters, service registration centers, authentication centers, and distributed task scheduling functions to achieve storage, management, visualization, and rendering of location data in the covered areas, as well as service support.

[0026] The comprehensive positioning service platform pulls data pushed from the user domain of the public security network to the Kafka cluster in the T zone and stores it in the PG database in real time; it also provides device management, real-time positioning monitoring, historical trajectory playback and electronic fences;

[0027] The external service platform provides multiple map rendering services including multi-location normalization, thermal analysis, density analysis, trajectory anomaly, and residence analysis.

[0028] The present invention uses Kafka to achieve low-latency aggregation of large-scale positioning data across different network intervals. First, the terminal device location information is transmitted to the mobile police network for aggregation. After the mobile police network aggregates the location information of several terminal devices, it is transmitted through Kafka across the boundary of the public security network to the public security network user domain Kafka. The public security network user domain aggregates the positioning data from the mobile police network and various municipal platforms, and then pushes it to the public security network T zone across the boundary through the Kafka cluster. The public security network T zone realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to each business system, thereby achieving low-latency aggregation of large-scale positioning data across different network intervals. In this way, the technical problem of high latency in the prior art when processing data aggregation of large-scale positioning data across network intervals is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 This is a network architecture diagram of the present invention's method for implementing low-latency aggregation of large-scale positioning data across different network intervals using Kafka.

[0031] Figure 2It is a schematic diagram of the data transmission architecture from the user domain of the public security network to the T zone of the public security network of the present invention.

[0032] Figure 3 It is a schematic diagram of the design architecture of the comprehensive positioning service platform of the present invention.

[0033] Figure 4 It is a detailed schematic diagram of the time consumption of each link of the method of the present invention using Kafka to realize low-latency aggregation of large-scale positioning data across different network intervals. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0035] See also Figure 1 to Figure 4 ,in Figure 1 This is a network architecture diagram of the method for implementing low-latency aggregation of large-scale positioning data across different network intervals using Kafka in the present invention. Figure 2 This is a schematic diagram of the data transmission architecture from the user domain of the public security network to the T zone of the public security network of the present invention. Figure 3 is a schematic diagram of the design architecture of the comprehensive positioning service platform of the present invention, Figure 4 It is a detailed schematic diagram of the time consumption of each link of the method of the present invention using Kafka to realize low-latency aggregation of large-scale positioning data across different network intervals.

[0036] The present invention provides a method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka, comprising the following steps:

[0037] S101, the terminal device location information is transmitted to the mobile police network for collection;

[0038] According to this specific implementation method, after the positioning information of the device itself is corrected for position accuracy by differential service, it is transmitted from the Internet side to the mobile police network through the government cloud to achieve position aggregation on the mobile police network side;

[0039] At the same time, the authenticated devices are directly connected to the mobile police network, and their built-in SDK analyzes the differential data transmitted from the Internet side to optimize the positioning accuracy. The optimized positioning data is directly aggregated on the mobile police network side. For example, the differential service of the certificate center adopts this method.

[0040] In addition, equipment with high-precision positioning capabilities uses Beidou inertial navigation technology + 5G base station assisted positioning technology to directly achieve high-precision positioning effects; in this way, high-precision positioning data on the Internet side is transmitted to the mobile police network through the government cloud to achieve location aggregation; in the next step, certified high-precision positioning equipment is also directly connected to the mobile police network.

[0041] S102, after the mobile police network collects the location information of several terminal devices, it passes through the public security network boundary through Kafka to the public security network user domain Kafka;

[0042] For this specific implementation, a mobile police network location aggregation platform, a Kafka cluster, a Zookeeper cluster, and an Nginx cluster are built in the mobile police network;

[0043] The mobile police network location aggregation platform aggregates terminal data from the Internet and mobile police network, and simultaneously responds to the needs of the mobile police network business system for positioning information;

[0044] The mobile police network location aggregation platform aggregates the information and transmits the data from the mobile police network to the public security network user domain in real time through the Kafka cluster;

[0045] The work efficiency of this link can reach 400,000 messages per second, with a delay as low as milliseconds; during the data transmission process, the parsed location data is encrypted and decrypted using Snappy compression or LZ4 compression algorithm to improve data transmission security and efficiency.

[0046] S103, the public security network user domain aggregates the positioning data from the mobile police network and local municipal platforms, and then pushes it across the border to the public security network T zone through the Kafka cluster;

[0047] For this specific implementation, a public security network city data processing platform, a Kafka cluster, a Zookeeper cluster, and an Nginx cluster are deployed in the public security network user domain;

[0048] The location data pushed from the mobile police network and the location data reported by the city-level location platform on the public security network are aggregated through the city-level data processing platform of the public security network, so as to realize the cross-network and full coverage integration and aggregation of the positioning data of various terminals in the covered area, and then carry out standardized processing of the location data, and then push it into the T zone of the public security network using Kafka.

[0049] S104, the T zone of the public security network realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to various business systems, thereby realizing the low-latency aggregation of large-scale positioning data across different network intervals;

[0050] For this specific implementation method, a comprehensive positioning service platform and an external service platform are built in the T zone of the public security network, which adopts containerized deployment, integrates gateway clusters, service registration centers, authentication centers, and distributed task scheduling functions, and realizes the storage, management, visualization and rendering of location data in the covered areas, and service support;

[0051] The comprehensive positioning service platform pulls data pushed from the user domain of the public security network to the Kafka cluster in the T zone and stores it in the PG database in real time; it also provides device management, real-time positioning monitoring, historical trajectory playback and electronic fences;

[0052] The external service platform provides multiple map rendering services including multi-location normalization, thermal analysis, density analysis, trajectory anomaly, and residence analysis;

[0053] The architecture of the comprehensive positioning service platform is as follows: Figure 3 , the functions are as follows:

[0054] Integrate multiple middleware: The comprehensive positioning service platform uses self-built modular service registration center nacos cluster, load balancing nginx, data cache redis cluster, message queue kafka cluster, file storage minio cluster and other middleware to ensure the efficiency and reliability of data processing.

[0055] Build multiple modular services: Each module is responsible for handling different functions of the platform, including:

[0056] 1. Basic service module for equipment, police officers and authority management;

[0057] 2. External service module that realizes functions such as multi-location normalization, online duration analysis, and device profiling;

[0058] 3. System service module that controls user login, sets user permissions and roles, and checks operation logs;

[0059] 4. Real-time positioning service module responsible for receiving and processing real-time location information;

[0060] 5. Comprehensive positioning service module that processes comprehensive positioning electronic fence data and historical data.

[0061] Release multiple microservices: Provide 12 externally supported microservices, including multi-location normalization, online duration analysis, device profiling, heat map, density analysis, trajectory anomaly, trajectory anomaly analysis, residence analysis, driving anomaly, police dispatch similarity, electronic fence, spatial statistical analysis service, and query service by jurisdiction. Register the above series of microservices to the unified microservice management platform of the provincial department to achieve standardized service management and control

[0062] A method for implementing low-latency aggregation of large-scale positioning data across different network intervals using Kafka in this embodiment is used in specific use.

[0063] The location information reported by high-precision terminal devices accessing the Class II area needs to be moved from the police network and the public security network user domain to the public security network T zone;

[0064] By setting up a complete set of simulation environment on the external network for testing and verification, ignoring the time spent on passing through the security boundary, the overall time spent is between 90-100 milliseconds; the time spent on each link is as follows Figure 4 As shown;

[0065] The specific analysis of the delay of the high-precision terminal positioning data reporting process by network segment is as follows:

[0066] 1. Time consumption analysis of mobile police network:

[0067] 1. Data upload: High-precision equipment connected to the Class II area uploads data to the mobile police network location aggregation platform through the JT808 protocol, which takes about 20-30ms.

[0068] 2. Data processing: The data is received, verified, parsed and packaged on the Class II area aggregation platform, which takes about 15ms, and then pushed to the message queue, which takes about 5ms.

[0069] (II) Time consumption analysis of the public security network user domain:

[0070] 1. Data cross-border transmission: The time taken for data to be transmitted from the mobile police network to the user domain of the public security network is estimated to be 50ms;

[0071] 2. Data reception and forwarding: Data is received through the message queue of the user domain of the public security network, and integrated and forwarded to the T zone of the public security network by the municipal data processing platform, which takes about 10ms.

[0072] (III) Time consumption analysis of T zone of public security network:

[0073] 1. Cross-border data transmission: Data is pushed from the user domain of the public security network to Kafka in zone T of the public security network. The estimated transmission time is 50ms.

[0074] 2. Data flow: The data arrives at the message queue of zone T, and then passes through the comprehensive location aggregation platform (5ms) in sequence, and is rendered for display. The time consumption of each step of the rendering display is the calculation of the rendering sequence number (10ms), the real-time data push (5ms), and finally the GPU rendering for display (20ms).

[0075] 5. Estimation of positioning data volume:

[0076] During the whole process of uploading location information from the device to the integrated positioning service platform, it is planned to build a Kafka cluster in the mobile police network and user domain, and use the Kafka service provided by the information center in the T area. To ensure the normal operation of the entire link, the required resources of the self-built Kafka cluster are calculated here:

[0077] 1. Estimation of data quantity:

[0078] Number of devices: According to the survey, there are about 300,000 various positioning devices in the coverage area, of which about 10%, or 30,000, actually use positioning capabilities. Among them, the number of high-precision positioning devices is estimated to be 3,000, and the number of ordinary devices is estimated to be 27,000.

[0079] Data reporting frequency: High-precision equipment reports about once every second; ordinary equipment: The frequency of reporting position is uncertain, and it is estimated that each device uploads once every 30 seconds on average;

[0080] Positioning data cache period: 3 days;

[0081] The size of a single data message: 300 bytes; each data message here is standardized, including the message header (including metadata such as message length and timestamp), partition and offset information;

[0082] Average daily access time of a single device: 8 hours;

[0083] Data compression ratio: 60%; using Snappy compression algorithm to maintain efficient compression and decompression speed, the average compression rate is about 60%;

[0084] (II) Data volume calculation:

[0085] Number of concurrent positioning data reports: number of high-precision terminals 3000*1 time / second + number of ordinary terminals 27000*1 / 30 times / second = 3900 times / second.

[0086] Three-day data cache capacity: 3,000 high-precision terminal devices generate data records * 3 days * 8 hours * 3,600 records / hour = 259,200,000 data records; 27,000 ordinary terminals generate data records * 3 days * 8 hours * 120 records / hour = 77,760,000 records.

[0087] Three-day data capacity: (259,200,000+77,760,000)*300=94.15GB

[0088] Cache space after compression: 94.15*60%=56.49GB.

[0089] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka, characterized in that: The following steps are involved: Step 1: The terminal device location information is transmitted to the mobile police network for collection; Step 2: After the mobile police network collects the location information of several terminal devices, it passes through the public security network boundary to the public security network user domain Kafka through Kafka; Step 3: The user domain of the public security network aggregates the location data from the mobile police network and local city platforms, and then pushes it across the border to the T zone of the public security network through the Kafka cluster; Step 4: The T zone of the public security network realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to various business systems, thereby realizing low-latency aggregation of large-scale positioning data across different network intervals.

2. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 1, characterized in that: The specific implementation method of transmitting the terminal device location information to the mobile police network for collection is as follows: After the device's own positioning information is corrected for position accuracy by differential services, it is transmitted from the Internet side to the mobile police network through the government cloud to achieve location aggregation on the mobile police network side; At the same time, certified devices are directly connected to the mobile police network, and their built-in SDK parses the differential data transmitted from the Internet side to optimize positioning accuracy.

3. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 2, characterized in that: After the mobile police network collects the location information of several terminal devices, the specific implementation method of passing through the public security network boundary to the public security network user domain Kafka through Kafka is as follows: Build a mobile police network location aggregation platform, Kafka cluster, Zookeeper cluster and Nginx cluster in the mobile police network; The mobile police network location aggregation platform aggregates terminal data from the Internet and mobile police network, and simultaneously responds to the needs of the mobile police network business system for positioning information; After the mobile police network location aggregation platform aggregates the information, it transmits the data from the mobile police network to the public security network user domain in real time through the Kafka cluster.

4. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 3, characterized in that: The Zookeeper cluster is used to register the Kafka cluster and monitor the working status of the Kafka cluster.

5. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 4, characterized in that: The Nginx cluster is used to achieve load balancing of business system requests on the mobile police network side.

6. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 5, characterized in that: The specific implementation method of the public security network user domain to aggregate the positioning data from the mobile police network and local municipal platforms and then push it across the boundary to the public security network T zone through the Kafka cluster is as follows: The public security network user domain is deployed with the public security network city data processing platform, Kafka cluster, Zookeeper cluster and Nginx cluster; The location data pushed from the mobile police network and the location data reported by the city-level location platform on the public security network are aggregated through the city-level data processing platform of the public security network, so as to realize the cross-network and full coverage integration and aggregation of the positioning data of various terminals in the covered area, and then carry out standardized processing of the location data, and then push it into the T zone of the public security network using Kafka.

7. The method for realizing low-latency aggregation of large-scale positioning data across different network intervals using Kafka as claimed in claim 6, characterized in that: The T zone of the public security network realizes the location rendering and data storage of all positioning data, and further relies on the microservice platform to provide external service support to various business systems, thereby realizing the low-latency aggregation of large-scale positioning data across different network intervals. The specific implementation method is: A comprehensive positioning service platform and external service platform are built in Zone T of the public security network. They are deployed in containers and integrate gateway clusters, service registration centers, authentication centers, and distributed task scheduling functions to achieve storage, management, visualization, and rendering of location data in the covered areas, as well as service support. The comprehensive positioning service platform pulls the data pushed from the public security network user domain to the T zone Kafka cluster and stores it in the PG database in real time; It also provides equipment management, real-time positioning monitoring, historical trajectory playback and electronic fencing; The external service platform provides multiple map rendering services including multi-location normalization, thermal analysis, density analysis, trajectory anomaly, and residence analysis.

Images