A real-time position calculation method based on large-scale trajectory information data stream
Patent Information
- Application Number
- CN202511177561.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-21
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2045-08-21
AI Technical Summary
[0004]传统的实时位置计算技术在处理大规模实时数据时,由于数据量过大,会导致性能下降,无法满足实时性的要求
[0021]本发明深度融合了Apache Flink流式计算引擎的超低延迟、高吞吐处理能力与大规模高性能大数据组件在分布式存储与计算上的优化特性。通过Flink高效的内存计算、状态管理和流水线执行模型,结合大规模高性能大数据组件底层针对海量数据IO和计算任务的深度优化,实现了对大规模实时位置数据的并行、高速处理。显著提升了位置计算的吞吐量和响应速度,能够满足每秒百万级甚至更高数据点的实时处理需求,有效克服了传统方案在面对海量实时数据流时的性能瓶颈。有效解决了流式数据中普遍存在的乱序问题,确保了位置状态的精确性。大幅减少了因数据延迟或无序到达导致的位置状态计算偏差,使得基于事件时间计算的“最后位置”及“最后时间”结果更加真实、可靠。本发明构建于Flink和大规模高性能大数据组件的分布式架构之上。能够无缝应对从中小规模到超大规模(如亿级/十亿级对象实时位置跟踪)的业务场景,满足未来业务扩张的需求,具备高度的弹性与灵活性。
Smart Images

Figure CN121301395B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, and mainly to a real-time location calculation method based on a large-scale trajectory information data stream. Background Technology
[0002] Currently, real-time location computing primarily relies on traditional databases and distributed computing frameworks such as MySQL, Hadoop, and MongoDB. These technologies often suffer from performance bottlenecks when processing large-scale real-time trajectory data, failing to meet the requirements for both real-time performance and accuracy. In the era of big data, streaming data processing is becoming increasingly important. With the development of technologies such as the Internet of Things and mobile internet, massive amounts of real-time data, such as sensor data and user behavior data, have been generated. This data is characterized by its high real-time nature, large volume, and high speed, necessitating efficient streaming data processing technologies for analysis and processing.
[0003] Flink, an open-source streaming computing framework, has been widely used in the field of streaming data processing. Flink provides powerful event-time processing capabilities, state management functions, and fault tolerance mechanisms, enabling it to handle large-scale real-time data. It supports various data formats and data sources, such as Kafka and HDFS, and can be easily integrated with other big data components.
[0004] Traditional real-time location computing technologies suffer from performance degradation when processing large-scale real-time data due to the sheer volume of data, failing to meet real-time requirements. Data out-of-order issues: In practical applications, network latency, data acquisition equipment malfunctions, and other factors can lead to data being out of order. Traditional processing methods cannot effectively handle out-of-order data, affecting the accuracy of calculation results. Storage and update efficiency issues: Traditional databases are inefficient at storing and updating large-scale real-time data, unable to meet the demands of high-concurrency write and update operations. Summary of the Invention
[0005] To address the above shortcomings, according to a first aspect of the present invention, a method for real-time calculation of location using large-scale trajectory information data streams is proposed, the specific steps of which are as follows:
[0006] Extracting massive amounts of trajectory data from the Kafka message queue, and performing duplicate data deletion on this massive amount of trajectory data using Flink, specifically including:
[0007] Assign a corresponding event timestamp to each trajectory data based on the event_time field;
[0008] Define a scrolling window with a specified threshold time, and partition the trajectory data within each window using [primary key element, window start, window end] as the partitioning condition;
[0009] Window functions are used to sort the trajectory data in each partition in ascending order by the event_time field, and a sequence number is assigned to each record in the trajectory data. By filtering the records with sequence number 1 in each partition, the deduplicated trajectory data is obtained. The compress() function is used to compress the deduplicated trajectory data to obtain the compressed data.
[0010] Based on Flink, a data table is constructed according to the primary key elements of the sorted trajectory data. The above steps are repeated to continuously update the data in the data table. The last active longitude, latitude and activity time of the primary key object are extracted to obtain the real-time location of the primary key object.
[0011] Preferably, the massive trajectory data in the message queue Kafka is stored in binary format; the binary format is converted using the Protobuf data structure, and the converted data is serialized into a binary stream, which is then sent to the message queue Kafka through the producer API of the message queue Kafka.
[0012] Preferably, the deduplication of the massive trajectory data based on Flink also includes a water level configuration strategy. The configuration strategy specifically includes: specifying the collection time field of the trajectory data as event_time; setting the maximum time difference for out-of-order data arrival; after the window runs, when the water level advances to the end of the window plus the additional waiting time allowedLateness, triggering the deduplication result of the current window and clearing the window state.
[0013] Preferably, the compressed data is further sorted based on Flink's operator chaining transfer, specifically including:
[0014] The compressed data is partitioned by keyBy according to the primary key element, and data with the same primary key are assigned to the same parallel task.
[0015] A time window is set based on the event time. The data within the time window is sorted in ascending order according to the event_time field. The sorting process uses a custom window function to sort the data set and output it in sequence when the window is triggered.
[0016] The sorting operator and the downstream sink are configured as chained operators, so that they are executed in the same task chain. This reduces the overhead of data serialization and network transmission between operators, and the sorting result is directly written to the data table of the downstream storage system.
[0017] Preferably, the data table needs to explicitly define a primary key field that is consistent with the keyBy partition field in Flink; if a composite primary key is used, the field order must be consistent with the Flink partitioning logic.
[0018] Preferably, the updated data table is updated based on the event_time field as the anchor point; if location extraction is performed within the selected time period, the data table is updated until convergence, and then the latitude and longitude coordinates of the primary key object are extracted; if location extraction is performed in real time, the latitude and longitude coordinates of the primary key object are tracked in real time while the data table is updated.
[0019] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.
[0020] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:
[0021] This invention deeply integrates the ultra-low latency and high throughput processing capabilities of the Apache Flink streaming engine with the optimized distributed storage and computing features of large-scale, high-performance big data components. Through Flink's efficient in-memory computing, state management, and pipelined execution model, combined with the deep optimizations of the underlying large-scale, high-performance big data components for massive data I / O and computational tasks, parallel and high-speed processing of large-scale real-time location data is achieved. This significantly improves the throughput and response speed of location computing, meeting the real-time processing needs of millions or even higher data points per second, effectively overcoming the performance bottlenecks of traditional solutions when facing massive real-time data streams. It effectively solves the out-of-order problem commonly found in streaming data, ensuring the accuracy of location status. It greatly reduces the deviation in location status calculation caused by data latency or out-of-order arrival, making the "last location" and "last time" results calculated based on event time more realistic and reliable. This invention is built on a distributed architecture of Flink and large-scale, high-performance big data components. It can seamlessly handle business scenarios from small to medium scale (such as real-time location tracking of hundreds of millions / billions of objects), meeting the needs of future business expansion, and possessing high elasticity and flexibility. Attached Figure Description
[0022] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.
[0023] Figure 1 A flowchart illustrating a real-time location calculation method based on a large-scale trajectory information data stream according to an embodiment of the present invention is shown.
[0024] Figure 2 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation
[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0027] Figure 1 A flowchart illustrating a real-time location calculation method based on a large-scale trajectory information data stream according to an embodiment of the present invention is shown, as follows: Figure 1 As shown:
[0028] S1. Massive trajectory data is aggregated into the message queue Kafka and stored in binary format to reduce the size of a single data entry and improve the data production and consumption rate.
[0029] Serialization is the process of transforming data structures into a format that is easy to store or transmit, and can be reconstructed when needed later. Effective data serialization is crucial for optimizing the throughput and latency of data transmission systems. The choice of serialization protocol significantly impacts the overall system performance, affecting key metrics such as serialization and deserialization time, total execution time, and the efficient use of network and storage resources.
[0030] This invention uses the Protobuf data structure for binary format conversion, and after conversion, the data is serialized into a binary stream, which is then sent to the Kafka message queue via the Kafka producer API. Using the Protobuf data structure reduces the amount of data that needs to be written / sent for a single message, and eliminates the need for any significant conversion between disk and network, thus achieving low latency and high throughput.
[0031] S2. Extract massive trajectory data from the message queue Kafka, and perform duplicate data deletion on the massive trajectory data based on Flink.
[0032] Window deduplication is a special type of deduplication that removes rows that are repeated across a set of columns, retaining the first or last row for each window and partition key.
[0033] For streaming queries, unlike regular deduplication on continuous tables, window deduplication does not emit intermediate results; it only emits the final result at the end of the window. Furthermore, window deduplication clears all intermediate states when no longer needed.
[0034] The Flink-based window deduplication algorithm is shown below:
[0035] Assign a corresponding event timestamp to each trajectory data based on the event_time field;
[0036] Define a scrolling window with a specified threshold time, and partition the trajectory data within each window using [primary key element, window start, window end] as the partitioning condition;
[0037] The window function `row_number()` is used to sort the trajectory data in each partition in ascending order by the `event_time` field, and a sequence number is assigned to each record in the trajectory data. The sequence number is the row number of each record. By filtering the records with the sequence number 1 in each partition, the deduplicated trajectory data is obtained. The `compress()` function is used to compress the deduplicated trajectory data to obtain the compressed data.
[0038] When using the `row_number()` window function, it assigns a unique row number to each record in the window based on the sorting criteria you specify (such as time, value size, etc.). After sorting, each record in the window has a numerical identifier; the first record is row number 1, the second is row number 2, and so on.
[0039] There may be multiple similar or duplicate data records within the same partition or window. By sorting and filtering the records with row number 1, duplicate data is avoided, thus achieving deduplication.
[0040] For example, sort the data table in the following window by event_time:
[0041] A 2024-02-2201:00 data_1 A 2024-02-2201:05 data_2 A 2024-02-2201:10 data_3 B 2024-02-2201:03 data_4 B 2024-02-2201:08 data_5
[0042] Remove duplicate data from the data table in the above window:
[0043] For data from device A, the row numbers are 1, 2, and 3; for data from device B, the row numbers are 1 and 2.
[0044] After filtering the records with row number 1, we obtain the deduplicated data, as shown in the table below:
[0045] A 2024-02-2201:00 data_1 B 2024-02-2201:03 data_4
[0046] Since the window deduplication algorithm based on Flink performs window deduplication based on event time, it is necessary to configure the watermark. If the watermark is not configured, Flink cannot correctly determine when the window will close, and thus cannot trigger the final calculation result of window deduplication. Especially in scenarios where out-of-order arrival is very common, such as trajectory data, the following situations will occur:
[0047] 1) If the data arrives with a delay but the window has already closed, this part of the data will be directly discarded;
[0048] 2) The window does not close for a long time, and the deduplication result is not output for a long time, which increases the pressure on state storage.
[0049] The watermark configuration strategy specifically includes:
[0050] Specify the collection time field of the trajectory data as event_time to ensure that the time zone and format are correct.
[0051] Set the maximum time difference for out-of-order data arrival; in trajectory data, GPS signals and network delays may cause data to arrive late for several seconds or even tens of seconds, so it is necessary to configure the tolerance for data arrival, and wait for a period of time before the window runs to wait for all out-of-order data to arrive.
[0052] After the window runs, when the watermark advances to the end of the window plus the additional waiting time allowedLateness, the deduplication result of the current window is triggered, and the window state is cleaned up.
[0053] allowedLateness, as a parameter of the Flink window mechanism, represents the time range during which late data can still be accepted after the window is normally closed. Under normal circumstances, Flink will trigger window calculation and output the result when the watermark is greater than or equal to the window end time, and then clean up the window state. If late data arrives at this time (event_time < window_end), it will be directly discarded. After allowedLateness is enabled, the window will not be completely closed immediately after being triggered. It will retain the state for an additional period of time (lateness), accept late data and re-trigger the calculation, and data exceeding this additional time will be discarded.
[0054] S3. Operator chain transmission based on Flink, set the time window according to time, sort the compressed trajectory data, and reduce the update performance problem caused by time difference.
[0055] Based on the deduplicated and compressed trajectory data stream, the system first performs keyBy partitioning according to primary key elements, ensuring that data with the same primary key is assigned to the same parallel task. Then, a rolling time window is set based on event time, and the data within the window is sorted in ascending order according to the event_time field. The sorting process can be performed by a custom ProcessWindowFunction window function, which sorts the data set and outputs it sequentially when the window is triggered. Finally, the sorting operator and the downstream sink are configured as chained operators, executing them in the same task chain to reduce data serialization and network transmission overhead between operators, and the sorting results are directly written to the downstream storage system.
[0056] If data with the same primary key arrives in the database out of order, the data that arrives earlier might be from a later time point, causing the update to be overwritten. The data that arrives later but from an earlier time point will trigger another update, resulting in wasted performance. After sorting, the data is stored in the database in chronological order, requiring only appending or sequential updates. Furthermore, after the window sorting component is modularized, data can be organized in batches, making the database batch processing more efficient.
[0057] S4. Based on Flink, construct a data table for the sorted trajectory data according to the primary key elements, repeat the above steps, continuously update the data in the data table, extract the last active longitude, latitude and activity time in the primary key object, and obtain the real-time location of the primary key object.
[0058] The data table must explicitly define a primary key field that matches the keyBy partitioning field in Flink; if a composite primary key is used, the field order must be consistent with Flink's partitioning logic. In batch sink scenarios, the primary key parameters must be mapped one-to-one with the sorted data objects to avoid batch errors.
[0059] The data table is updated based on the event_time field as the anchor point; if the location is extracted within the selected time period, the data table is updated until convergence, and then the latitude and longitude coordinates of the primary key object are extracted; if the location is extracted in real time, the latitude and longitude coordinates of the primary key object are tracked in real time while the data table is updated.
[0060] Complexity analysis is performed based on the algorithm described above in this invention:
[0061] Time Complexity: The main time overhead of the algorithm lies in data processing and sorting operations. Because Flink employs an efficient streaming processing mechanism, the time complexity can be kept within O(n), where n is the amount of data.
[0062] Space complexity: The space complexity of the algorithm mainly depends on Flink's state management and the data storage of large-scale, high-performance big data components. Because Flink provides efficient state management capabilities, the space complexity can be kept within a reasonable range.
[0063] The "Large-scale Real-time Position Calculation Algorithm" of this invention has profound application value and significant practical significance in the following key industries due to its high performance, high accuracy, and high scalability:
[0064] In the public security and safety sector: real-time monitoring of suspects' locations enables rapid deployment and precise arrests; electronic fence management of key monitoring targets (such as community correction personnel); real-time monitoring of crowd flow and density during major events; and tracking of rescue forces' locations in emergency command. This application has significant strategic importance for improving crime-fighting efficiency and maintaining social security and stability.
[0065] Smart logistics and supply chain management: Real-time tracking of goods in transit, providing accurate estimated times of arrival (ETA) to enhance customer experience; optimizing transportation route planning and vehicle scheduling to reduce empty runs; monitoring the movement of goods and forklift positions within warehouses to improve warehouse operational efficiency; ensuring the safe transportation of high-value or sensitive goods (such as cold chain pharmaceuticals and hazardous materials). Effectively improves logistics efficiency and supply chain management, and reduces costs.
[0066] Intelligent Transportation and Vehicle-to-Everything (V2X) Systems: Real-time monitoring of vehicle location, speed, and direction on roads; construction of city-level traffic flow heat maps for congestion prediction and mitigation; real-time monitoring and dispatching of commercial vehicles (taxis, ride-hailing vehicles, buses, and freight trucks); provision of location-based navigation and travel services (such as finding nearby parking lots and charging stations); support for V2X applications such as vehicle collision warning and emergency rescue. Significantly improving traffic safety and efficiency, and optimizing traffic resource allocation.
[0067] Social networks and location services: Real-time display of friends' / group members' locations within social apps (user authorization required), enhancing the sense of presence and interactivity in social interactions; core support for features such as "People Nearby" and "Nearby Activities"; precise push notifications of local business offers and event information based on the user's real-time location (LBS advertising / recommendations); core basis for supply and demand matching in sharing economy platforms (such as bike-sharing and ride-hailing). This greatly enriches the user experience and enhances the stickiness and commercial value of social platforms.
[0068] The following is for reference. Figure 2 It shows a schematic diagram of the structure of a computer system 200 suitable for implementing electronic devices according to embodiments of the present application. Figure 2 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0069] like Figure 2As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the system 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.
[0070] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a liquid crystal display (LCD) and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card and a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 210 as needed so that computer programs read from it can be installed into storage section 208 as needed.
[0071] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0072] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0074] The modules described in the embodiments of this application can be implemented in software or in hardware.
[0075] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the electronic device, the electronic device causes the following actions: It extracts massive trajectory data from a message queue (Kafka) and performs deduplication on the massive trajectory data using Flink. Specifically, this includes: assigning a corresponding event timestamp to each trajectory data based on the `event_time` field; defining a scrolling window with a specified threshold time and partitioning the trajectory data within each window using [primary key element, window start, window end] as the partitioning condition; sorting the trajectory data within each partition in ascending order by the `event_time` field using a window function and assigning a sequence number to each record in the trajectory data; obtaining deduplicated trajectory data by filtering records with sequence number 1 in each partition; compressing the deduplicated trajectory data using the `compress()` function to obtain compressed data; constructing a data table based on the primary key element using Flink on the sorted trajectory data; repeating the above steps; continuously updating the data in the data table; and extracting the last active longitude, latitude, and activity time from the primary key object to obtain the real-time location of the primary key object.
[0076] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A real-time position calculation method based on a large-scale trajectory information data stream, characterized in that, include: Extracting massive amounts of trajectory data from the Kafka message queue, and performing duplicate data deletion on this massive amount of trajectory data using Flink, specifically including: Assign a corresponding event timestamp to each trajectory data based on the event_time field; Define a scrolling window with a specified threshold time, and partition the trajectory data within each window using [primary key element, window start, window end] as the partitioning condition; Window functions are used to sort the trajectory data in each partition in ascending order by the event_time field, and a sequence number is assigned to each record in the trajectory data. By filtering the records with sequence number 1 in each partition, the deduplicated trajectory data is obtained. The compress() function is used to compress the deduplicated trajectory data to obtain the compressed data. It also includes water level configuration strategies, which specifically include: specifying the acquisition time field of trajectory data as event_time; setting the maximum time difference for out-of-order data arrival; after the window runs, when the water level advances to the end of the window plus the extra waiting time allowedLateness, triggering the deduplication result of the current window and clearing the window state; The compressed data is further sorted based on Flink's operator chaining process, specifically including: The compressed data is partitioned by keyBy according to the primary key element, and data with the same primary key are assigned to the same parallel task. A time window is set based on the event time. The data within the time window is sorted in ascending order according to the event_time field. The sorting process uses a custom window function to sort the data set and output it in sequence when the window is triggered. The sorting operator and the downstream sink are configured as chained operators so that they are executed in the same task chain, reducing the serialization and network transmission overhead of data between operators, and the sorting result is directly written to the data table of the downstream storage system. Based on Flink, a data table is constructed according to the primary key elements of the sorted trajectory data. The above steps are repeated to continuously update the data in the data table. The last active longitude, latitude and activity time of the primary key object are extracted to obtain the real-time location of the primary key object.
2. The real-time location calculation method according to claim 1, characterized in that, The massive trajectory data in the message queue Kafka is stored in binary format; the binary format is converted using the Protobuf data structure, and the converted data is serialized into a binary stream, which is then sent to the message queue Kafka through the producer API.
3. The real-time location calculation method according to claim 1, characterized in that, The data table must explicitly define a primary key field that is consistent with the keyBy partition field in Flink; if a composite primary key is used, the field order must be consistent with the Flink partitioning logic.
4. The real-time location calculation method according to claim 1, characterized in that, The updated data table is updated based on the event_time field as the anchor point; if the location is extracted within the selected time period, the data table is updated until convergence, and then the latitude and longitude coordinates of the primary key object are extracted; if the location is extracted in real time, the latitude and longitude coordinates of the primary key object are tracked in real time while the data table is updated.
5. The real-time position calculation method according to claim 1, characterized in that, The real-time location calculation algorithm is integrated into a big data component, which is configured on a distributed architecture.
6. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.
7. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-5.
Citation Information
Patent Citations
An express delivery data processing method
CN113486069A
Real-time data deduplication method and device based on Flink, electronic equipment and medium
CN119225644A
