A method and apparatus for real-time publishing of freight rate data based on streaming mode

Abstract

This invention discloses a method and apparatus for real-time fare data publishing based on a streaming model, belonging to the field of fare data publishing technology. The method includes: a fare acquisition component polling a data source containing fare data at a preset period to capture incremental fare data and storing the captured incremental fare data in a fare storage component; a fare query component continuously querying the fare storage component in a streaming model to obtain incremental fare data; and a fare publishing component encapsulating the incremental fare data obtained by the fare query component into a message and publishing it to a message middleware for downstream systems that have subscribed to the message to receive it in real time. This invention achieves incremental data querying and publishing in a "streaming model" through a message mechanism, continuously publishing data to downstream fare calculation or fare search systems without any redundancy or waiting, greatly improving the fare effectiveness time for airlines.

Description

Technical Field

[0001] This invention belongs to the field of freight rate data publishing technology, and specifically relates to a method and apparatus for real-time publishing of freight rate data based on a streaming model. Background Technology

[0002] Airfare pricing, or fare business, is a core business of airlines, and the fare data ecosystem is as follows: Figure 1 As shown, the data setter (airline) is responsible for setting fares. They can generate large amounts of data using their own software and then upload it in batches to the data collector, or they can perform small-scale data maintenance through the front-end or interface provided by the data collector. The data collector receives the data uploaded by the airline, merges it with the existing data according to the fare business rules, and then stores it in the database. Finally, the merged incremental data is released to the data users. The data users use the fare data through the fare search system or fare calculation system and are provided with fare search or fare calculation services. Service users use the fare services through service calls or interface calls.

[0003] The most common mode of uploading and publishing fare data currently used is as follows: formatted text files are used as the data interface, FTP is used for transmission, and the text files are usually compressed to reduce transmission volume; the upload or publishing operation is initiated through scheduled timeouts, with scheduling cycles typically ranging from 10 to 30 minutes. This mode is adopted by leading companies in the international and domestic markets, such as ATPCO and TravelSky. In recent years, to cope with emergencies, data collectors have also provided some airlines with a method for urgently uploading very small amounts of error-correcting data, but data publishing still relies on scheduled timeouts.

[0004] The scheduled release of fare data results in a delay in data delivery from airlines to users. For example, a 30-minute schedule could take nearly an hour in the worst-case scenario to complete the data release after upload. With the rapid development of the civil aviation industry and increasingly fierce market competition, airlines urgently need fare data to be released and effective as quickly as possible. Furthermore, when releasing domestic fare data, airline fare management personnel typically utilize the characteristics of domestic fare data's sales and release times to perform data verification after release but before it becomes available for sale, in order to check for any abnormal pricing or other errors. The longer release delay compresses this buffer time for quality verification.

[0005] The traditional timed scheduling and release mode suffers from technical problems such as high delays and slow effectiveness of fare data release. There is an urgent need in this field to propose a new fare data release mode that can greatly improve the effective time of airline fares and realize a technical solution for real-time release of fare data. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and apparatus for real-time release of freight rate data based on a streaming mode. This aims to solve the problems of high latency and slow effectiveness of freight rate data caused by the traditional timed scheduling release mode, and to realize near real-time flow of freight rate data from production to the user end.

[0007] To solve the above problems, the present invention adopts the following technical solution: Firstly, this invention provides a method for real-time publishing of freight rate data based on a streaming model, the process of which is as follows: Figure 2 As shown, applied to a system including a fare collection component, a fare storage component, a fare query component, and a fare publishing component, the method includes the following steps: S1, the fare collection component polls the data source containing fare data at a preset period, captures incremental fare data, and stores the captured incremental fare data in the fare storage component; wherein, the incremental fare data includes newly added, modified, and deleted fare data; S2, the fare query component continuously queries the fare storage component in streaming mode to obtain the incremental fare data; wherein, the streaming mode means that the next query is initiated immediately after the completion of one query; S3, the fare publishing component encapsulates the incremental fare data obtained by the fare query component into a message and publishes it to the message middleware so that downstream systems that have subscribed to the message can receive it in real time.

[0008] Furthermore, in step S1, the preset polling period is ≤60s, for example, it can be 15s, 30s, etc. By setting an extremely short polling interval, near real-time data acquisition is achieved.

[0009] Furthermore, the fare collection component polls the data source containing fare data at a preset period to capture incremental fare data, including: The freight rate collection component is deployed in a distributed cluster with multiple collection service points, which concurrently poll the data source containing freight rate data to check for any incremental freight rate data. When incremental fare data is detected, an exclusive lock is set in the distributed cache to ensure that the incremental fare data is processed by only one thread at a time.

[0010] Furthermore, the freight rate storage component adopts a relational database, including a business table, a high-water mark table for recording data batches, and a deletion record table for recording deletion operations; the business table includes a data creation timestamp field, a data last update timestamp field, and a high-water mark identifier field associated with the high-water mark table.

[0011] Furthermore, the fare query component continuously queries the fare storage component in streaming mode to obtain the incremental fare data, including: Based on the batch creation timestamp in the high water level table, delete the high water level identifier in the record table, as well as the high water level identifier, creation timestamp, and last update timestamp in the business table, and construct query conditions to distinguish and obtain incremental fare data from batch file uploads and front-end operations that include additions, modifications, and deletions.

[0012] Furthermore, the query conditions include: for new and modified data in batch file uploads, the query conditions are: the high watermark of the business table is equal to the high watermark of the table, and the creation timestamp of the high watermark is within the time range of this query. For deleted data from batch file uploads, the query conditions are: the high-water mark of the deleted record table is equal to the high-water mark of the high-water mark table, and the creation timestamp of the high-water mark table is within the time range of this query. For adding and modifying data in front-end operations, the query conditions are: the high-water mark of the business table is empty, and the creation timestamp or last update timestamp of the business table is within the time range of this query. For data deletion operations on the front end, the query conditions are: the high-water mark of the deleted record table is empty, and the creation timestamp of the deleted record table is within the time range of this query.

[0013] Furthermore, the fare query component continuously queries the fare storage component in streaming mode, and also includes: sharding the target business table whose data volume exceeds a threshold according to the hash value of the primary identifier of its data records; Start multiple threads to query different data shards concurrently, and encapsulate the query results of each shard into an independent message.

[0014] Furthermore, the fare publishing component encapsulates the incremental fare data obtained by the fare query component into a message and publishes it to the message middleware, including: For each streaming mode release process, create a manifest message to record batch metadata and a data message to record incremental fare data; The message body of the data message contains incremental fare data and is set with attributes, including: the release timestamp of all incremental fare data in this batch, the full name of the source business table, the total number of messages corresponding to the business table, the sequence number of the current message, and the number of data records in the current message body; The message body of the list message is empty, and its attributes include: the timestamp of this data release, the timestamp of the last data release, and the total number of all data messages released this time; List messages and data messages belonging to the same release batch will be published as sequential messages to the same message topic in the message middleware.

[0015] Furthermore, when the downstream system subscribes to and processes the message, it performs the following steps: All received messages with the same publication timestamp are checked to see if their total number is equal to the total number of messages recorded in the corresponding list message plus one. Data messages originating from the same business table are sorted according to their message sequence number, and the number of messages in the table is verified to be equal to the corresponding attribute value. Verify that the number of data records in each data message body is equal to the total number of messages corresponding to the business table; Based on the timestamp of the last publication recorded in the list message, the current batch of messages will only begin to be processed after confirming that the previous batch of messages has been processed.

[0016] Furthermore, the message middleware adopts a message queue based on the publish-subscribe model, and the message middleware adopts a multi-partition redundant storage design. The message type published by the fare publishing component is a sequential message.

[0017] Secondly, the present invention provides a real-time fare data publishing device based on a streaming mode, comprising: The fare collection module is used to poll data sources containing fare data at preset intervals to capture incremental fare data; wherein, the incremental fare data includes newly added, modified, and deleted fare data; A freight rate storage module is used to persistently store the incremental freight rate data captured by the freight rate acquisition module; The fare query module is used to continuously query the fare storage component in streaming mode to obtain the incremental fare data; wherein, the streaming mode means that the next query is initiated immediately after the first query is completed; The fare publishing module is used to encapsulate the incremental fare data obtained by the fare query module into a message and publish it to the message middleware so that downstream systems that have subscribed to the message can receive it in real time.

[0018] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned method for real-time publication of freight rate data based on a streaming mode.

[0019] Fourthly, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores computer-readable instructions, and the computer-readable instructions, when executed by the processor, implement the aforementioned method for real-time publishing of freight rate data based on streaming mode.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. Significantly Reduced Release Delay, Achieving Near Real-Time Effectiveness: This invention utilizes a messaging mechanism in a "streaming mode" to achieve incremental data querying and release. Data is continuously released to downstream fare calculation or fare search systems without any redundancy or waiting, greatly improving the effective time of airline fares. Whether inputting a small amount of fare data through front-end products or uploading large amounts of fare data as files, the end-to-end delay from data upload to downstream system effectiveness is reduced from minutes to seconds or near real-time, meeting the urgent market demand for rapid fare changes.

[0021] 2. Ensuring strong consistency of business data in streaming scenarios: Addressing the complex relationships and order-sensitive nature of freight rate data, an innovative publishing protocol combining "list messages + data messages" and corresponding attribute validation rules was designed. This not only guarantees the reliability of message transmission but also ensures that subscribers can correctly, completely, and orderly reassemble and process data, solving the critical challenge of data consistency in streaming publishing.

[0022] 3. High throughput and high availability: The solution adopts a distributed, clustered, and parallel design in all stages of data collection, querying, and publishing (such as distributed locks, hash sharding concurrent queries, and multi-partition redundant message queues), which enables the system to support high concurrency and real-time processing of massive freight rate data, ensuring the overall high performance and stability of the system.

[0023] Other features and effects of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This diagram illustrates the traditional freight rate data ecosystem mentioned in the background section of this invention. Figure 2 A flowchart illustrating a real-time fare data publishing method based on a streaming mode according to an embodiment of the present invention is shown. Figure 3 A schematic diagram of the implementation process of freight rate collection in one embodiment of the present invention is shown; Figure 4 This diagram illustrates an example of the data table structure for freight rate storage in one embodiment of the present invention. Figure 5This diagram illustrates the message structure used for fare publication in one embodiment of the present invention. Figure 6 A code example snippet of a message subscriber according to an embodiment of the present invention is shown; Figure 7 A schematic diagram of a real-time fare data publishing device based on a streaming mode is shown in one embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1 The core of this invention lies in constructing a "continuously flowing" data processing pipeline to replace the traditional "collection-storage-timed triggering-batch release" pulse-style processing mode. This real-time fare data release method based on the streaming mode involves four logical components: a fare collection component, a fare storage component, a fare query component, and a fare release component.

[0028] This embodiment provides a method for real-time publishing of freight rate data based on a streaming model, which specifically includes the following steps: Step S101, freight rate collection.

[0029] like Figure 3 As shown, the fare collection component is responsible for capturing fare data from the data source (usually the airline) in near real-time.

[0030] 1. Data Upload: The airline packages a batch of relevant fare data files and uploads them to the designated FTP server directory.

[0031] 2. Distributed Polling Data Collection: The fare collection service is deployed using a distributed multi-machine cluster with multiple collection service points. Each collection service point in the cluster is assigned an independent processing thread for each cooperating airline. Each thread polls and scans the corresponding airline path on the FTP server at very short intervals (e.g., 10 seconds) to check for any new file uploads.

[0032] 3. Deduplication Prevention: When threads on different servers in the cluster simultaneously detect a new file from the same airline, mutual exclusion is achieved by writing an exclusive lock to a distributed cache (such as Redis). The thread that successfully acquires the lock gains the right to process that batch of data, while other threads relinquish their rights, thus ensuring that the same batch of data is not processed repeatedly.

[0033] 4. Processing and Storage: The processing thread downloads and parses the data file, performs business verification and transformation, and finally submits all the incremental freight rate data of the batch of new, modified and deleted operations to the freight rate storage component in a database transaction to ensure the atomicity of the data within the batch.

[0034] Step S102: Freight rate storage.

[0035] The freight rate storage component uses a relational database (such as Oracle). Figure 4 As shown, the core of its database design lies in supporting efficient incremental data identification.

[0036] 1. Table structure design: Business tables, such as the fare table (FARE) and rule table (RULE), store core business data. In addition to the business fields, each business table includes the following fields: create_time (data creation timestamp field), last_update_time (data last update timestamp field), and high_water_id (a high-water level identifier field associated with the high-water level table, where the high-water level identifier field is used to associate the ID field of the data batch).

[0037] The high watermark (HIGH_WATER) is used to record data uploaded and submitted in the same batch. Using the HIGH_WATER mechanism can minimize the interval between data processing and actual data entry into the database. It includes the create_time field (data creation timestamp field).

[0038] The deletion record table (5, DELETE_DATA) is specifically used to record deleted data, including the table_name (table name field), create_time (data creation timestamp field), data (data field), and high_water_id (high water level identifier field associated with the high water level table) of the deleted records.

[0039] Incremental identification principle: The `high_water_id` field can be used to distinguish between two categories: "batch file uploads" (`high_water_id` is not null) and "front-end operation input" (`high_water_id` is null). Combined with the timestamp fields of each table, the time and source of any data change can be accurately traced.

[0040] Deduplication prevention: When threads on different servers in the cluster simultaneously detect a new file from the same airline, mutual exclusion is achieved by writing an exclusive lock with a timeout to a distributed cache (such as Redis). The thread that successfully acquires the lock gains the right to process that batch of data, while other threads relinquish their rights, thus ensuring that the same batch of data is not processed repeatedly.

[0041] S103: Flow mode fare inquiry.

[0042] 1. The fare query component operates in "stream mode," meaning that after a query-publish cycle ends, the next query begins immediately without any hibernation or waiting in between.

[0043] 2. In addition to the data uploaded in batches by airlines as files as mentioned above, the database also contains a small amount of data entered through the front-end product. Each query records the start and end times, and constructs query conditions by deleting the high-water mark identifier from the record table, the high-water mark identifier, creation timestamp, and last update timestamp from the business table, based on the batch creation timestamp in the high-water mark table designed in S102 above, and the high-water mark identifier, creation timestamp, and last update timestamp from the business table. The specific query conditions are as follows: 2.1) New and modified data for batch file uploads: The high-water mark of the business table is equal to the high-water mark of the high-water table, and the creation timestamp of the high-water table is within the time range of the current query; that is: business table (e.g., FARE).high_water_id=HIGH_WATER.ID and (HIGH_WATER.create_time>= last query time and HIGH_WATER.create_time< current query time).

[0044] 2.2) Deleting data in batch file uploads: The high watermark of the deleted record table is equal to the high watermark of the high watermark table, and the creation timestamp of the high watermark table is within the time range of the current query; that is: DELETE_DATA.high_water_id=HIGH_WATER.ID and (HIGH_WATER.create_time>= last query time and HIGH_WATER.create_time< current query time).

[0045] 2.3) For data added and modified in front-end operations: the high-water mark of the business table is empty, and the creation timestamp or last update timestamp of the business table is within the range of the current query time; that is: business table (e.g., FARE).high_water_id is NULL and ((business table.create_time>= last query time and business table.create_time< current query time) or (business table.last_update_time>= last query time and business table.last_update_time< current query time)).

[0046] 2.4) For data deletion operations in the front end: the high watermark of the deleted record table is empty, and the creation timestamp of the deleted record table is within the current query time range; that is: DELETE_DATA.high_water_id is NULL and(DELETE_DATA.create_time>= last query time and DELETE_DATA.create_time< current query time).

[0047] 3. High-performance parallel query: The fare query is implemented using a multi-machine cluster, and multi-threaded parallel queries are performed on different servers according to different airline schemas (data interfaces).

[0048] 4. For target business tables with data volumes exceeding the threshold (such as rule-based fare association tables), a performance optimization strategy is adopted: the data in the table is sharded by taking the modulo of the hash value of the primary key ID (e.g., divided into 8 shards), and then multiple threads are started to concurrently query different shards, greatly improving throughput. The query results are no longer combined, but the result set of each shard is directly sent to the publishing stage.

[0049] S104: Freight Rate Publication. The freight rate publication component transforms query results into a reliable data stream.

[0050] 1. Message encapsulation and attribute settings: Message structure: such as Figure 5 As shown, each message consists of a Topic, a Body, and several ATTRs, with each ATTR being a key-value pair.

[0051] Data Messages: The results of parallel queries in the fare query component are no longer combined, but are directly encapsulated into different messages. All the increased fare data retrieved are used as the message body. Key attributes are set for the data messages: vintage (timestamp of this data release), table_name (full name of the source business table), message_count (total number of messages corresponding to the business table), message_seq (sequence number of the current message), and row_count (number of incremental fare data records in the current message body).

[0052] Inventory Messages: In addition to the data messages mentioned above, an inventory message is created for each complete streaming release. The message body is empty. Attributes include: vintage (consistent with the data messages in this batch), last_vintage (the timestamp of the previous data release), and total_message_count (the total number of all data messages in this release). The specific attributes set in the inventory message and data message are shown in Table 1 below.

[0053] Table 1

[0054] 2. Sequential Publishing: The list messages and all data messages of this batch are published as sequential messages to the same topic (e.g., FARE_DIST) in a message middleware based on a publish-subscribe model (such as a modified Kafka with multi-partition redundant storage). In short, the freight rate data containing all increments is encapsulated into a message and sent to the message middleware's interface. The message middleware then passes the message to a message queue, publishing it to downstream subscribers in a broadcast-subscribe manner.

[0055] 3. Subscription-side processing: Downstream systems subscribe to this topic. For example... Figure 6 As shown, after receiving a message, the subscriber client needs to perform the following steps to ensure data integrity, consistency, and order: 3.1) Collect all messages received with the same publication timestamp and verify that their total number is equal to the total number of messages recorded in the corresponding list message plus one (the total number of messages in each batch == the value of total_message_count in the list message + 1). 3.2) Sort data messages from the same business table according to their message sequence number, and verify whether the number of messages in the table is equal to the corresponding attribute value (group the data messages by table_name, sort the messages in each group according to the value of message_seq in the data message, and the number of messages in each group is equal to the value of message_count in the data message). 3.3) Verify whether the number of data records in each data message body is equal to the number of records recorded in its attributes (the number of rows in the body of each data message == the value of row_count in the data message); 3.4) Find the vintage of other batches of messages, i.e. the previous batch of messages, according to the value of last_vintage in the manifest message. If the previous batch of messages has not been processed, the current batch of messages will wait until the previous batch of messages has been processed before the current batch of messages will begin to be processed.

[0056] Through the above steps, airline fare data changes can be transmitted through the entire system in near real-time in the form of a "stream" and reliably consumed by downstream systems, achieving the minimization of release delays and the maximization of data consistency.

[0057] Example 2 A real-time fare data publishing device based on a streaming model is used to implement the aforementioned real-time fare data publishing method based on a streaming model, such as... Figure 7 As shown, it includes: (1) A fare collection module is used to poll a data source containing fare data at a preset period to capture incremental fare data; wherein the incremental fare data includes newly added, modified and deleted fare data.

[0058] The fare collection module is the data source for the publishing methods and devices, and its reliability and efficiency directly determine the data quality of all subsequent processes. It acts as a "data tentacle," responsible for capturing and initially processing fare data in real time from the external environment (mainly data sources containing fare data provided by airlines).

[0059] In terms of data acquisition mechanisms, the fare collection module employs high-frequency polling to achieve "near real-time" data collection. The collection module scans the target directory on the FTP server at extremely short intervals (e.g., every 15 or 30 seconds) to check for new files being created or old files being updated. This design sacrifices absolute real-time performance (sub-second level) but gains extremely high stability and good compatibility with existing infrastructure, avoiding the complexity of maintaining long connections.

[0060] The data capture mechanism includes comprehensive anomaly detection and handling logic, such as FTP connection anomaly detection and reconnection mechanisms, and retry mechanisms for file transfer interruptions or failures. For files that fail after multiple retries, an alarm needs to be triggered to notify maintenance personnel for manual intervention.

[0061] After the data file is successfully captured, it undergoes format parsing and business logic processing. The original freight rate file is then transformed into a collection of data objects with a clear structure that conforms to business specifications, and finally stored in the freight rate storage module.

[0062] (2) A fare storage module, used to persistently store the incremental fare data captured by the fare acquisition module.

[0063] The freight rate storage module bears the important responsibility of persistent data. Its design is directly related to data integrity, query performance and system scalability. It usually adopts a relational database, and freight rate data is stored in the form of database tables according to different types such as freight rate, rules, routes, etc.

[0064] (3) A fare query module, used to continuously query the fare storage module in streaming mode to obtain the incremental fare data; wherein, the streaming mode means that the next query is initiated immediately after the first query is completed; As the data flow hub of the facility, the fare query module's core responsibility is to efficiently and accurately retrieve incrementally updated fare data from the fare storage module. This module features a deeply optimized query mechanism, employing streaming query technology to nearly instantly "follow" database changes, thereby continuously capturing all added, modified, or deleted data events and minimizing data retrieval latency.

[0065] At the query execution level, the module employs a highly concurrent query strategy to maximize throughput. This means that complex query tasks are broken down according to business logic, and concurrent data retrieval operations are initiated on multiple related tables or even different data partitions of a single large table. This significantly shortens the overall data query time and provides a stable and fast data flow for downstream real-time processing and publishing.

[0066] (4) The fare publishing module is used to encapsulate the incremental fare data obtained by the fare query module into a message and publish it to the message middleware so that downstream systems that have subscribed to the message can receive it in real time.

[0067] The fare publishing module breaks away from the inefficient traditional model of timed batch synchronization or direct database access, adopting a publish-subscribe model based on a message mechanism to achieve real-time, reliable, and wide-area distribution of fare query results. This fare publishing module comprises two core modules: Message Broadcast Module: Acts as a message producer. The fare publishing module encapsulates the data obtained from the fare query module into a standard-format message and publishes it to the message middleware with a specific topic. This process is "broadcast," and the publisher does not care who will receive it.

[0068] Message Subscription Module: Acts as a message consumer. All downstream systems that require the latest fare data (such as fare calculation systems, fare search systems, etc.) will deploy a client that subscribes to this message topic. Once a new fare change message is published, all downstream systems that have subscribed to this topic will receive a notification almost in real time.

[0069] Freight rate data differs from ordinary data. Its various data types, such as agreements, freight rates, and rules, have complex relationships, and the data's effective deadlines follow a strict order. Therefore, when the freight rate publishing module encapsulates "streaming" data into messages, it must ensure the integrity, consistency, and order of the data.

[0070] Example 3 This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the program, it implements the method described in Embodiment 1. Additionally, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the method described in Embodiment 1.

[0071] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time publishing of freight rate data based on a streaming model, characterized in that, When applied to a system that includes a fare collection component, a fare storage component, a fare query component, and a fare publishing component, the following steps are included: The fare collection component polls the data source containing fare data at a preset period, captures incremental fare data, and stores the captured incremental fare data in the fare storage component; wherein, the incremental fare data includes newly added, modified, and deleted fare data; The fare query component continuously queries the fare storage component in streaming mode to obtain the incremental fare data; wherein, the streaming mode means that the next query is initiated immediately after the completion of one query; The fare publishing component encapsulates the incremental fare data obtained by the fare query component into a message and publishes it to the message middleware for downstream systems that have subscribed to the message to receive it in real time.

2. The method according to claim 1, characterized in that, The fare collection component polls the data source containing fare data at a preset period to capture incremental fare data, including: The fare collection component is deployed in a distributed cluster with multiple collection service points, which concurrently poll the data source containing fare data to check for incremental fare data; wherein the preset period of the polling is ≤60s; When incremental fare data is detected, an exclusive lock is set in the distributed cache to ensure that the incremental fare data is processed by only one thread at a time.

3. The method according to claim 1, characterized in that, The freight rate storage component uses a relational database, including a business table, a high-water mark table that records data batches, and a deletion record table that records deletion operations. The business table includes a data creation timestamp field, a data last update timestamp field, and a high-water mark identifier field associated with the high-water mark table.

4. The method according to claim 3, characterized in that, The fare query component continuously queries the fare storage component in streaming mode to obtain the incremental fare data, including: Based on the batch creation timestamp in the high water level table, delete the high water level identifier in the record table, as well as the high water level identifier, creation timestamp, and last update timestamp in the business table, and construct query conditions to distinguish and obtain incremental fare data from batch file uploads and front-end operations that include additions, modifications, and deletions.

5. The method according to claim 4, characterized in that, The query conditions include: For new and modified data in batch file uploads, the query conditions are: the high-water mark of the business table is equal to the high-water mark of the high-water mark table, and the creation timestamp of the high-water mark table is within the time range of this query. For deleted data from batch file uploads, the query conditions are: the high-water mark of the deleted record table is equal to the high-water mark of the high-water mark table, and the creation timestamp of the high-water mark table is within the time range of this query. For adding and modifying data in front-end operations, the query conditions are: the high-water mark of the business table is empty, and the creation timestamp or last update timestamp of the business table is within the time range of this query. For data deletion operations on the front end, the query conditions are: the high-water mark of the deleted record table is empty, and the creation timestamp of the deleted record table is within the time range of this query.

6. The method according to claim 1, characterized in that, The fare query component continuously queries the fare storage component in streaming mode, and also includes: For target business tables whose data volume exceeds the threshold, shard them according to the hash value of the primary identifier of their data records; Start multiple threads to query different data shards concurrently, and encapsulate the query results of each shard into an independent message.

7. The method according to claim 1, characterized in that, The fare publishing component encapsulates the incremental fare data obtained by the fare query component into a message and publishes it to the message middleware, including: For each streaming mode release process, create a manifest message to record batch metadata and a data message to record incremental fare data; The message body of the data message contains incremental fare data and is set with attributes, including: the release timestamp of all incremental fare data in this batch, the full name of the source business table, the total number of messages corresponding to the business table, the sequence number of the current message, and the number of incremental fare data records in the current message body; The message body of the list message is empty, and its attributes include: the timestamp of this data release, the timestamp of the last data release, and the total number of all data messages released this time; List messages and data messages belonging to the same release batch will be published as sequential messages to the same message topic in the message middleware.

8. The method according to claim 1, characterized in that, When a downstream system subscribes to and processes the message, it performs the following steps: All received messages with the same publication timestamp are checked to see if their total number is equal to the total number of messages recorded in the corresponding list message plus one. Data messages originating from the same business table are sorted according to their message sequence number, and the number of messages in that table is verified to be equal to the total number of messages in the corresponding business table. Verify that the number of data records in each data message body is equal to the number of records recorded in its attributes; Based on the timestamp of the last publication recorded in the list message, the current batch of messages will only begin to be processed after confirming that the previous batch of messages has been processed.

9. The method according to claim 1, characterized in that, The message middleware adopts a message queue based on the publish-subscribe model, and the message middleware adopts a multi-partition redundant storage design. The message type published by the fare publishing component is sequential message.

10. A real-time fare data publishing device based on a streaming model, characterized in that, include: The fare collection module is used to poll data sources containing fare data at preset intervals to capture incremental fare data; wherein, the incremental fare data includes newly added, modified, and deleted fare data; A freight rate storage module is used to persistently store the incremental freight rate data captured by the freight rate acquisition module; The fare query module is used to continuously query the fare storage component in streaming mode to obtain the incremental fare data; wherein, the streaming mode means that the next query is initiated immediately after the first query is completed; The fare publishing module is used to encapsulate the incremental fare data obtained by the fare query module into a message and publish it to the message middleware so that downstream systems that have subscribed to the message can receive it in real time.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time fare data publishing method based on the streaming mode as described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: A processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, implement the real-time publishing method for freight rate data based on a streaming mode as described in any one of claims 1 to 9.

Images