Taxi operation data forwarding method and device, medium and program product

Abstract

The invention provides a taxi operation data forwarding method and device, a medium and a program product, and the method comprises the steps: monitoring a redis real-time message sent to a first receiver according to a subscribed redis message channel, and obtaining taxi operation data corresponding to a pre-subscribed redis message channel from the monitored redis real-time message; converting the acquired taxi operation data into a kafka message, and writing the kafka message into a kafka message queue for caching; according to a taxi operation data forwarding requirement received from a second receiver, obtaining historical kafka messages meeting the taxi operation data forwarding requirement from the kafka message queue; and converting the historical kafka message meeting the taxi operation data forwarding requirement into a redis message, and forwarding the redis message to the second receiver through the redis connection. By means of the technical scheme, through the message conversion design of redis and kafka, on the basis of not changing an existing platform software framework, compensation type forwarding of historical operation data is achieved, and persistence of the operation data is achieved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium, and program product for forwarding taxi operation data. Background Technology

[0002] In the taxi industry, the analysis of meter operation data is an important tool for government departments to assess companies and drivers, and the completeness and accuracy of the operational data forwarded by company platforms are also valued by functional departments at all levels. With business development, taxi company platforms are increasingly connecting with third-party organizations, requiring a single business platform to forward meter operation data to multiple regulatory agencies.

[0003] For traditional platform data integration, there are currently two main solutions in the industry:

[0004] 1. Real-time forwarding: Establish communication connections with third-party platforms, such as Redis, and use the business platform's message flow for rapid forwarding;

[0005] 2. Scheduled batch forwarding: Operational data within a certain time range is forwarded at regular intervals through job task control.

[0006] In comparison, Option 1 is more timely and simple to implement, but it cannot solve problems such as missed reports in abnormal situations, the data retransmission mechanism is weak, and the transformation cost is also high. Option 2 has stronger data integrity, but the timeliness of forwarding is insufficient. In addition, operational data will be lost in the event of communication abnormalities, and the forwarding efficiency is also low due to the influence of the network and server.

[0007] Both of the above solutions carry the risk of operational data loss, resulting in the data recipient receiving operational data from a third-party platform. Currently, resolving this issue requires modifying the existing operational data forwarding mechanism. This involves transforming the existing platform, significantly intruding on it, disrupting its code logic, and incurring high costs. Summary of the Invention

[0008] The embodiments of the present invention provide a method, apparatus, medium and program product for forwarding taxi operation data, which achieves the permanence of operation data by using message conversion design of Redis and Kafka without changing the existing platform software framework.

[0009] To achieve the above objectives, on the one hand, a method for forwarding taxi operation data is provided, including:

[0010] Based on the pre-subscribed Redis message channel, listen for Redis real-time messages sent to the first receiver through a pre-established Redis connection, and obtain taxi operation data corresponding to the pre-subscribed Redis message channel from the listened Redis real-time messages.

[0011] The acquired taxi operation data is converted into Kafka messages and written to the Kafka message queue cache through a pre-established Kafka connection;

[0012] Based on the taxi operation data forwarding requirements received from the second recipient, historical Kafka messages that meet the taxi operation data forwarding requirements are obtained from the Kafka message queue, wherein the taxi operation data forwarding requirements include: the time range for uploading taxi operation data;

[0013] Historical Kafka messages that meet the requirements for forwarding taxi operation data are converted into Redis messages and forwarded to the second receiver through the Redis connection.

[0014] Preferably, in the forwarding method, the first receiver and the second receiver may be the same or different.

[0015] Preferably, the forwarding method further includes, after obtaining the taxi operation data corresponding to the pre-subscribed Redis message channel and before writing the obtained taxi operation data into the Kafka message queue cache:

[0016] The acquired taxi operation data is cleaned.

[0017] The data cleaning includes: data deduplication, used to deduplicat taxi operation data with the same vehicle identification, pick-up time and drop-off time; and / or data correction, used to modify taxi operation data where the drop-off time is earlier than the pick-up time, by changing the date of the drop-off time to one day later than the pick-up time.

[0018] Preferably, in the forwarding method, when subscribing to the Redis message channel, the Redis message channel uses a regular expression, and the step of obtaining taxi operation data corresponding to the pre-subscribed Redis message channel from the monitored Redis real-time messages includes:

[0019] Determine if the message object in the monitored Redis real-time message is of type pmessage. If so, retrieve the data object from that message.

[0020] Preferably, in the forwarding method, when the acquired taxi operation data is converted into a Kafka message, the topic of the Kafka message corresponds to the message channel of the corresponding Redis real-time message, and the message group ID of the Kafka message is unique.

[0021] Preferably, the forwarding method, wherein converting historical Kafka messages that meet the taxi operation data forwarding requirements into Redis messages includes:

[0022] The vehicle ID of the taxi is obtained by parsing the historical Kafka messages.

[0023] Generate a Redis key based on the vehicle identity ID and the subscribed Redis message channel.

[0024] Preferably, in the forwarding method, when forwarding Redis messages converted from historical Kafka messages to the second receiver, a predetermined time interval is set between each predetermined number of Redis messages sent.

[0025] On the other hand, a device for forwarding taxi operation data is provided, including a memory and a processor, wherein the memory stores at least one program, which is executed by the processor to implement the forwarding method as described above.

[0026] In another aspect, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement the forwarding method as described in any of the above descriptions.

[0027] In another aspect, a computer program product is provided, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the forwarding methods described above.

[0028] The above technical solution has the following technical effects:

[0029] The technical solution of this invention, based on a deep understanding of the usage patterns of Redis messages on the business platform, creatively introduces Kafka message stream caching. Utilizing the persistence feature of Kafka messages, which can be stored locally in sequence, the real-time Redis message stream is converted into a Kafka message stream, and when needed, the Kafka messages are converted back into Redis messages for forwarding. Without adjusting the existing platform business logic, historical operational data can be saved, thereby realizing the storage and persistence of taxi operational data and solving the problem of missed Redis operational data reports at the lowest cost.

[0030] Furthermore, the time range for data retransmission can be flexibly customized, which facilitates enterprises to synchronize operational data multiple times;

[0031] Furthermore, in addition to compensatory forwarding, the technical solution using the embodiments of the present invention also facilitates the forwarding of complete and accurate taxi operation data to third-party platforms; moreover, the method of the embodiments of the present invention is also applicable to the forwarding of similar business data other than taxi operation data. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a method for forwarding taxi operation data according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram illustrating the principle of a method for forwarding taxi operation data according to another embodiment of the present invention;

[0034] Figure 3 This is a flowchart illustrating a method for forwarding taxi operation data according to another embodiment of the present invention. Detailed Implementation

[0035] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0036] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0037] Example 1:

[0038] Figure 1 This is a flowchart illustrating a method for forwarding taxi operation data according to an embodiment of the present invention. Figure 1 The method for forwarding taxi operation data in this embodiment includes the following steps:

[0039] Based on the pre-subscribed Redis message channel, listen for Redis real-time messages sent to the first recipient through a pre-established Redis connection, and retrieve taxi operation data corresponding to the pre-subscribed Redis message channel from the listened Redis real-time messages.

[0040] The acquired taxi operation data is converted into Kafka messages and written to the Kafka message queue cache through a pre-established Kafka connection;

[0041] Based on the taxi operation data forwarding requirements received from the second recipient, historical Kafka messages that meet the taxi operation data forwarding requirements are retrieved from the Kafka message queue. The taxi operation data forwarding requirements include the time range for uploading taxi operation data.

[0042] Historical Kafka messages that meet the requirements for forwarding taxi operation data are converted into Redis messages and forwarded to the second recipient via a Redis connection.

[0043] In one specific implementation, the first recipient and the second recipient may be the same or different. If they are the same, it is a compensatory forwarding; if they are different, it is providing historical operational data to a third party.

[0044] Example 2:

[0045] Figure 2 This is a schematic diagram of the forwarding model for the taxi operation data forwarding method of this embodiment of the present invention. Figure 2 The system converts operational data received from taxis into real-time Redis messages and transmits them in real-time to the corresponding recipients, such as monitoring platforms A and B, via the Redis message bus. Simultaneously, it converts the data into Kafka messages and sends these Kafka messages to a server capable of providing Kafka messaging services for chronological storage and persistence. When needed, Kafka messages within a selected time range are converted back into Redis messages and sent to the appropriate recipients. In practical implementation, the conversion between Redis and Kafka messages for taxi operational data can be achieved by deploying a software module, such as a script, that performs incremental deployment and message synchronization. This approach does not require disrupting the existing platform's code logic, offering a degree of versatility and representing a low-intrusion operational data forwarding solution.

[0046] Figure 3 This is a flowchart illustrating the method for forwarding taxi operation data according to an embodiment of the present invention. Figure 3 The method for forwarding taxi operation data in this embodiment includes the following seven parts: 1. Redis operation data message processing

[0047] (1) Redis connection

[0048] Configure the parameters related to the Redis service, such as the host (redis_host), port (redis_port), and password (redis_password) to establish a Redis connection and obtain the Redis service.

[0049] Use the Redis service to send real-time Redis messages containing taxi operation data to the corresponding recipients.

[0050] (2) Redis message processing, including: subscription and processing of operational messages.

[0051] Based on the Redis message channel corresponding to the taxi operation data to be acquired, the system pre-subscribes to the corresponding Redis channel and then listens for and retrieves the required real-time Redis messages containing the operation data. In the specific implementation, the channel uses a regular expression, and multiple channels are subscribed to. For example, the channel being listened to might be the string: `im:t8:0B05:*`. Here, `0B05` corresponds to the business type of the operation data to be listened to, `*` means to listen to the operation data of all related vehicles, and `im:t8` is a custom string. In other implementations, the channel is adjusted accordingly based on the business type of the taxi operation data to be listened to and the actual application scenario.

[0052] After receiving the corresponding taxi operation data Redis real-time message, it determines whether the type of the message object is pmessage. If so, it retrieves the data object from the message for subsequent data cleaning services.

[0053] 2. Data Cleaning

[0054] (1) Data deduplication

[0055] From the Redis message data monitored above, a predetermined number of data records are read. One implementation reads a certain number of data records at once. Operational data is then analyzed to remove duplicate data. In another specific implementation, the criteria for determining duplicate data are: the vehicle identifier, pick-up time, and drop-off time are all the same. This is because, during network fluctuations, taxi metering devices may upload operational data multiple times. This duplication occurs within a short period, and deduplication reduces interference and improves the accuracy of total data statistics.

[0056] (2) Time correction

[0057] Some fare meters report incorrect operating data in the early morning hours, with drop-off times appearing earlier than pick-up times across different days. This inaccurate data, known as "dirty data," needs correction. One implementation corrects this by modifying the date in the dirty data, ensuring the drop-off date is one day later than the pick-up date. The purpose of this correction is to improve the accuracy of data forwarding.

[0058] 3. Write Kafka messages and obtain the Kafka message stream.

[0059] Configure the Kafka parameters corresponding to the Kafka service. In this example, the Kafka topic parameter is set to topic_0B05, where the content "0B05" is consistent with the business data type in the subscribed Redis message channel. The Kafka topic and the subscribed Redis message channel must correspond to each other. In other implementations, the two do not necessarily have to be consistent, but there must be a definite correspondence.

[0060] The cleaned data is written to a Kafka message queue. In one specific implementation, the structure of the object written to Kafka is kept consistent with the structure of the object read from Redis. This facilitates the subsequent reverse conversion of the Kafka object into an Redis object.

[0061] 4. Kafka message processing: Transform Kafka messages into Redis messages to obtain a real-time Redis message stream.

[0062] (1) Kafka connection

[0063] By configuring parameters related to the Kafka service, such as IP address, port por, topic, and auto_offset_reset, a Kafka connection service is established with the server providing the Kafka service.

[0064] In one example, the key parameters of the kfaka service are configured as follows:

[0065] Topic (subscription message ID): topic_0B05; where this topic is consistent with the topic written from Redis to Kafka, in this example, it is consistent with 0B05;

[0066] `group_id` (message group ID): `my-group` + timestamp; This content uses `my-group` to identify the message group to be read, and the timestamp ensures the uniqueness of `group_id`. In other implementations, other methods can be used to ensure the uniqueness of the corresponding `group_id`. This is because in Kafka's read mechanism, the same message group ID will not be read repeatedly. This would prevent repeated resending of the same data. Therefore, by using the unique `group_id`, repeated resending can be implemented as needed for operational data containing the same time range. Setting `group_id` with a timestamp allows for filtering all Kafka messages each time they are read.

[0067] auto_offset_reset: earliest; indicates that reading starts from the earliest message, i.e., from when Kafka caching began.

[0068] redis_channel = 'im:t8:0B05:'; represents the content of the Redis channel when converted into a Kafka message;

[0069] start_time: Start time

[0070] end_time: End time

[0071] (2) Redis write

[0072] This step is used to reverse-engineer Kafka messages that meet predetermined requirements and are retrieved from the Kafka message queue into Redis messages.

[0073] Specifically, the process reads the Kafka message of the specified Topic from the Kafka message queue, parses it to obtain the vehicle identification ID (vid, Vehicle Identification) contained in the message, and combines it with the configuration content of the redis_channel parameter to generate the corresponding Redis key, i.e., redis_key; in this example, it is im:t8:0B05:vid. Then, it is written to the Redis service module, that is, the corresponding Redis message is forwarded to the corresponding data receiver using the corresponding Redis connection.

[0074] In one specific implementation, to reduce data communication congestion, when forwarding Redis messages converted from historical Kafka messages to the corresponding receiver, a predetermined interval is set between sending a predetermined number of Redis messages. In one specific example, the interval is 10 milliseconds after each message is written. In other implementations, different numbers of messages and intervals can be used as needed.

[0075] 5. Time Filtering

[0076] In one specific implementation, it is necessary to filter the time range specified, defined, or input by the user and write the required Kafka messages to the Redis service.

[0077] Filter historical Kafka messages, parse each Kafka message object one by one, obtain the initial upload time (up_time) of the taxi operation data, and compare it with the time range specified by the user. If it is within the time range, write the Kafka message to the Redis service; otherwise, skip it and continue iterating until the end.

[0078] 6. Use Kafka messages converted into Redis message streams for log output.

[0079] Successfully filtered Redis messages are output to the log file for debugging and data verification to confirm the effectiveness of data resending or retransmission.

[0080] 7. Message Statistics

[0081] The log output file displays the number of data forwarding compensations for this round, allowing users to query individual bike details and calculate the total number for data verification by the monitoring platform. This statistical analysis helps identify the causes of data loss and assess whether complete retransmission has been achieved or if further retransmission is necessary.

[0082] In one implementation of this invention, Redis message forwarding to Kafka messages is automatic and requires no human intervention; Kafka message conversion to Redis messages requires setting a time period and starting the process. After successful startup, Redis will generate a historical operational data message stream after Kafka conversion, thereby triggering the data forwarding function.

[0083] Example 3:

[0084] The present invention also provides a device for forwarding taxi operation data, including a memory and a processor, wherein the memory stores at least one program, which is executed by the processor to implement the forwarding method as described above.

[0085] Furthermore, as an executable solution, the forwarding device can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0086] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0087] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0088] Example 4:

[0089] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the forwarding method described above in the embodiments of the present invention.

[0090] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0091] Example 5:

[0092] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the forwarding method described above.

[0093] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for forwarding taxi operation data, characterized in that, include: Based on the pre-subscribed Redis message channel, listen for Redis real-time messages sent to the first receiver through a pre-established Redis connection, and obtain taxi operation data corresponding to the pre-subscribed Redis message channel from the listened Redis real-time messages. The acquired taxi operation data is converted into Kafka messages and written to the Kafka message queue cache through a pre-established Kafka connection; Based on the taxi operation data forwarding requirements received from the second recipient, historical Kafka messages that meet the taxi operation data forwarding requirements are obtained from the Kafka message queue, wherein the taxi operation data forwarding requirements include: the time range for uploading taxi operation data; Historical Kafka messages that meet the requirements for forwarding taxi operation data are converted into Redis messages and forwarded to the second receiver through the Redis connection.

2. The forwarding method according to claim 1, characterized in that, The first recipient may be the same as or different from the second recipient.

3. The forwarding method according to claim 1, characterized in that, After obtaining the taxi operation data corresponding to the pre-subscribed Redis message channel and before writing the obtained taxi operation data into the Kafka message queue cache, the process also includes: The acquired taxi operation data is cleaned. The data cleaning includes: data deduplication, used to deduplicat taxi operation data with the same vehicle identification, pick-up time and drop-off time; and / or data correction, used to modify taxi operation data where the drop-off time is earlier than the pick-up time, by changing the date of the drop-off time to one day later than the pick-up time.

4. The forwarding method according to claim 1, characterized in that, When subscribing to the Redis message channel, the Redis message channel uses regular expressions, and the steps for obtaining taxi operation data corresponding to the pre-subscribed Redis message channel from the monitored Redis real-time messages include: Determine if the message object in the monitored Redis real-time message is of type pmessage. If so, retrieve the data object from that message.

5. The forwarding method according to claim 3, characterized in that, When the acquired taxi operation data is converted into Kafka messages, the topic of the Kafka message corresponds to the message channel of the corresponding Redis real-time message, and the message group ID of the Kafka message is unique.

6. The forwarding method according to claim 1, characterized in that, Converting historical Kafka messages that meet the taxi operation data forwarding requirements into Redis messages includes: The vehicle ID of the taxi is obtained by parsing the historical Kafka messages. Generate a Redis key based on the vehicle identity ID and the subscribed Redis message channel.

7. The forwarding method according to claim 1, characterized in that, When forwarding Redis messages converted from historical Kafka messages to the second receiver, a predetermined interval is set between each predetermined number of Redis messages sent.

8. A device for forwarding taxi operation data, characterized in that, It includes a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the forwarding method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is executed by a processor to implement the forwarding method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the forwarding method as described in any one of claims 1 to 7.

Images