A data processing method, device and equipment for realizing low delay

Abstract

The present application relates to a kind of data processing method, device and equipment for realizing low delay, belong to data processing technical field.The present application is by obtaining waybill message, waybill message includes waybill number, waybill route section and waybill scanning date;Multiple waybill messages are aggregated, and waybill message is marked message type and version, obtain marked waybill message, and send marked waybill message to Kafka message queue;Marked waybill message is grouped and calculated, corresponding summary result data is generated, and a corresponding timer is registered for summary result data;When the timer triggers, summary result data is sent to task downstream operator for storage.By message type and version marking to waybill message, grouping calculation and setting timer, real-time calculation is carried out on data, so as to solve the problem that existing logistics business data exists and passive data query cannot meet the real-time requirement of data calculation.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and specifically relates to a method, apparatus and device for achieving low-latency data processing. Background Technology

[0002] With the continuous development and improvement of big data technology, the timeliness requirements for enterprise-level data indicator calculations are becoming increasingly stringent. The more real-time the data calculation, the better it can display the current status of the system, enabling agile adjustments to system-level strategies based on the current real-time indicators.

[0003] Currently, in practical applications within the express delivery industry, there is an increasing reliance on the real-time performance of business data calculations, particularly logistics routing data. Certain business data requires minute-level data calculation metrics to promptly address package backlogs at centers and branches through vehicle capacity scheduling. However, existing technologies generally perform calculations on offline logistics business data, and passive data queries cannot meet the real-time requirements for data computation. Summary of the Invention

[0004] Therefore, this invention provides a method, apparatus, and device for low-latency data processing, which helps to solve the problem that passive data queries cannot meet the real-time requirements of data calculation in existing logistics business data. To achieve the above objectives, this invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a method for achieving low-latency data processing, comprising:

[0006] Obtain waybill information; wherein, the waybill information includes waybill number, waybill route segment, and waybill scan date;

[0007] Multiple waybill messages are aggregated, and each waybill message is labeled with a message type and version to obtain a labeled waybill message. The labeled waybill message is then sent to a Kafka message queue.

[0008] The marked waybill messages in the Kafka message queue are processed by data grouping and calculation to generate corresponding summary result data, and a corresponding timer is registered for the summary result data;

[0009] When the timer is triggered, the summarized result data is sent to the downstream operator of the task for storage.

[0010] Furthermore, obtaining the waybill message includes:

[0011] The Binlog files in the business database are collected into the RocketMQ message queue; wherein, the Binlog files record the SQL statement information of the user updating the database;

[0012] The waybill message is obtained by consuming messages in the RocketMQ message queue using the Flink component.

[0013] Furthermore, the aggregation of multiple waybill messages includes:

[0014] Multiple waybill messages with the same waybill number but different waybill routing segments are aggregated.

[0015] Further, the waybill message is labeled with message type and version to obtain a labeled waybill message, including:

[0016] If the waybill message is received for the Nth time, and the message type of this waybill message is set to CALCULATE and the version is set to N, then the current waybill message is determined to be the Nth calculated mark waybill message; and

[0017] Set the message type of the Nth calculated marked waybill message to ROLLBACK to obtain the Nth rollback marked waybill message, and store the Nth rollback marked waybill message;

[0018] If a waybill message is received for the (N+1)th time, the Nth rollback-marked waybill message is sent to the Kafka message queue. A calculation is performed between this message and the (N+1)th received waybill message to obtain the (N+1)th calculated marked waybill message. The message type of the (N+1)th calculated marked waybill message is set to CALCULATE, and the version is set to N+1.

[0019] Set the message type of the (N+1)th calculated marked waybill message to ROLLBACK to obtain the (N+1)th rollback marked waybill message, and store the (N+1)th rollback marked waybill message;

[0020] The marked waybill message includes a waybill number, waybill routing segment, waybill scan date, message type, and version; the message type of the Nth calculated marked waybill message is different from that of the Nth rollback marked waybill message, but other data content is exactly the same; the waybill number of the Nth calculated marked waybill message is the same as that of the N+1th calculated marked waybill message, and the waybill routing segment ID of the Nth calculated marked waybill message is N, while the waybill routing segment ID of the N+1th calculated marked waybill message is N+1.

[0021] Furthermore, after determining that the current waybill message is the Nth calculated marked waybill message, the method further includes:

[0022] The Nth computed mark waybill message is serialized according to a preset format and sent to the Kafka message queue via a transaction.

[0023] Further, the step of performing data grouping calculations on the marked waybill messages in the Kafka message queue to generate corresponding summary result data includes:

[0024] The marked waybill messages are grouped according to data dimensions; wherein, the data dimensions include waybill routing segment and waybill scanning date;

[0025] The marked waybill messages with the same waybill route segment and waybill scan date are aggregated to obtain the aggregated result data.

[0026] Further, the process of summarizing the marked waybill messages with the same waybill route segment and waybill scanning date to obtain summary result data includes:

[0027] If the message type of the marked waybill message is CALCULATE, then an addition operation is performed to add the marked waybill message to the summary result data; and the detailed information extracted from the marked waybill message is added to the detailed data of the summary result data.

[0028] If the message type of the marked waybill message is ROLLBACK, then a subtraction operation is performed to remove the marked waybill message from the summary result data; and the detailed information previously extracted from the marked waybill message is removed from the detailed data of the summary result data.

[0029] Furthermore, the step of sending the summarized result data to the downstream operator of the task for storage includes:

[0030] Store the summarized results in MySQL or Elasticsearch;

[0031] The detailed data of the summarized results are concatenated and stored in HBase.

[0032] In a second aspect, the present invention provides a data processing apparatus for achieving low latency, comprising:

[0033] The message acquisition module is used to acquire waybill messages, wherein the waybill messages include the waybill number, waybill routing segment, and waybill scan date.

[0034] The message tagging module is used to aggregate multiple waybill messages, tag the waybill messages with message type and version, obtain tagged waybill messages, and send the tagged waybill messages to the Kafka message queue.

[0035] The result data generation module is used to perform data grouping calculations on the marked waybill messages in the Kafka message queue, generate corresponding summary result data, and register a corresponding timer for the summary result data;

[0036] The data persistence module is used to send the summarized result data to the downstream operator of the task for storage when the timer is triggered.

[0037] Thirdly, the present invention provides a data processing apparatus for achieving low latency, comprising:

[0038] One or more memories on which executable programs are stored;

[0039] One or more processors are configured to execute the executable program in the memory to implement the steps of any of the methods described above.

[0040] The present invention, by adopting the above technical solution, has at least the following beneficial effects:

[0041] This invention acquires waybill messages, including waybill number, waybill route segment, and waybill scan date. Multiple waybill messages are aggregated, and each message is labeled with a message type and version to obtain labeled waybill messages. These labeled waybill messages are then sent to a Kafka message queue. Data grouping and calculation are performed on the labeled waybill messages in the Kafka message queue to generate corresponding summary result data. A corresponding timer is registered for the summary result data. When the timer is triggered, the summary result data is sent to downstream operators for storage. By labeling waybill messages with message type and version and performing grouping and calculation, summary result data is obtained, enabling real-time data computation. Due to the massive amount of logistics data, by setting a timer, the summary result data is only sent to downstream tasks when the timer is triggered, allowing for a smoother receipt of the required real-time computation result data. This solves the problem that passive data queries in existing logistics business data cannot meet the real-time requirements for data computation.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating a low-latency data processing method according to an embodiment of the present invention;

[0045] Figure 2 This is a block diagram illustrating a low-latency data processing device according to an embodiment of the present invention;

[0046] Figure 3 This is a block diagram illustrating a low-latency data processing device according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating a low-latency data processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0049] Step S11: Obtain waybill message; wherein, the waybill message includes waybill number, waybill route segment and waybill scan date;

[0050] Step S12: Aggregate the data of multiple waybill messages, and mark the waybill messages with message type and version to obtain marked waybill messages, and send the marked waybill messages to the Kafka message queue;

[0051] Step S13: Perform data grouping calculation on the marked waybill messages in the Kafka message queue to generate corresponding summary result data, and register a corresponding timer for the summary result data;

[0052] Step S14: When the timer is triggered, the summarized result data is sent to the downstream operator of the task for storage.

[0053] It should be noted that in this embodiment, the executing entity is an electronic device such as a server.

[0054] It should be noted that a logistics waybill includes a waybill number and at least one waybill routing segment. For example, if a logistics waybill originates in Shanghai and terminates in Beijing, then Shanghai-Nanjing is the first routing segment, Nanjing-Shandong is the second routing segment, and Shandong-Beijing is the third routing segment. These three routing segments all share the same waybill number, and each routing segment has a unique, auto-incrementing `customize_id` field to identify the order of the waybill routing segments for the same waybill. For example, if a logistics waybill has a waybill number of 10000, Shanghai-Nanjing is the first routing segment with a `customize_id` of 1; Nanjing-Shandong is the second routing segment with a `customize_id` of 2; and Shandong-Beijing is the third routing segment with a `customize_id` of 3.

[0055] It should be noted that the waybill scanning date is the time when the latest information on the logistics waybill's operation is received, which is also the time when the waybill information is marked.

[0056] It should be noted that aggregating waybill messages and marking them with message type and version to obtain marked waybill messages is to roll back the already calculated messages and ensure that downstream processes use the latest messages for calculation.

[0057] It should be noted that Kafka message queue is a distributed, high-throughput, and highly scalable message queue system that acts as an intermediate message queue for consuming waybill messages.

[0058] It's important to note that the process involves grouping and calculating data for marked waybill messages to generate corresponding summary results. A timer is registered for each summary result; when the timer triggers, the summary result data is sent to the downstream operator for data persistence. In this example, when performing real-time calculations on logistics data, considering the rapid changes in the upstream database (tens of thousands of messages per second), each message could potentially alter the downstream summary and details, persisting these real-time changes to external storage would be extremely burdensome. Therefore, the design only writes the results stored within the job to the downstream operator when the timer triggers, achieving low latency in data result storage.

[0059] It is understood that this embodiment obtains waybill messages, including waybill numbers and waybill routing segments; aggregates multiple waybill messages, and marks each waybill message with a message type and version to obtain marked waybill messages, which are then sent to a Kafka message queue; performs data grouping calculations on the marked waybill messages in the Kafka message queue to generate corresponding summary result data, and registers a corresponding timer for the summary result data; when the timer is triggered, the summary result data is sent to the downstream operator of the task for storage. By marking waybill messages with message types and versions and performing grouping calculations, summary result data is obtained, enabling real-time calculation of the data; due to the massive amount of logistics data, by setting a timer so that the summary result data is only sent to the downstream of the task when the timer is triggered, the required real-time calculation result data can be received more smoothly, thereby solving the problem that passive data querying in existing logistics business data cannot meet the real-time requirements of data calculation.

[0060] Furthermore, obtaining the waybill message includes:

[0061] The Binlog files in the business database are collected into the RocketMQ message queue; wherein, the Binlog files record the SQL statement information of the user updating the database;

[0062] The waybill message is obtained by consuming messages in the RocketMQ message queue using the Flink component.

[0063] It's important to note that Binlog is a binary file used to record SQL statements used by users to update the database. For example, SQL statements that modify database tables and their contents are recorded in the Binlog, but queries to the database tables are not. RocketMQ is a pure Java, distributed, queue-based open-source message middleware that supports transactional messages, ordered messages, batch messages, scheduled messages, and message backtracking. The Binlog file from the database is collected into the RocketMQ message queue, and messages are consumed from the RocketMQ message queue using the Flink component to obtain waybill messages.

[0064] It should be noted that the RocketMQ message queue used to transmit Binlog files can also be replaced with any other message queue.

[0065] Furthermore, the aggregation of multiple waybill messages includes:

[0066] Multiple waybill messages with the same waybill number but different waybill routing segments are aggregated.

[0067] It should be noted that when aggregating waybill messages, the aggregation is based on the waybill number. Waybill messages with the same waybill number but different waybill routing segments are aggregated and merged as a whole for subsequent operations.

[0068] Further, the waybill message is labeled with message type and version to obtain a labeled waybill message, including:

[0069] If the waybill message is received for the Nth time, and the message type of this waybill message is set to CALCULATE and the version is set to N, then the current waybill message is determined to be the Nth calculated mark waybill message; and

[0070] Set the message type of the Nth calculated marked waybill message to ROLLBACK to obtain the Nth rollback marked waybill message, and store the Nth rollback marked waybill message;

[0071] If a waybill message is received for the (N+1)th time, the Nth rollback-marked waybill message is sent to the Kafka message queue. A calculation is performed between this message and the (N+1)th received waybill message to obtain the (N+1)th calculated marked waybill message. The message type of the (N+1)th calculated marked waybill message is set to CALCULATE, and the version is set to N+1.

[0072] Set the message type of the (N+1)th calculated marked waybill message to ROLLBACK to obtain the (N+1)th rollback marked waybill message, and store the (N+1)th rollback marked waybill message;

[0073] The marked waybill message includes a waybill number, waybill routing segment, waybill scan date, message type, and version; the message type of the Nth calculated marked waybill message is different from that of the Nth rollback marked waybill message, but other data content is exactly the same; the waybill number of the Nth calculated marked waybill message is the same as that of the N+1th calculated marked waybill message, and the waybill routing segment ID of the Nth calculated marked waybill message is N, while the waybill routing segment ID of the N+1th calculated marked waybill message is N+1.

[0074] It should be noted that the specific operations for marking each waybill message with message type and version to obtain marked waybill messages are as follows: When a waybill message is received for the first time, the message type is set to CALCULATE and the version is set to 1, resulting in the first calculated marked waybill message, which is then written to the Kafka message queue. Next, the message type of this message is set to ROLLBACK and stored, resulting in the first rollback marked waybill message. When a message with the same waybill number is received for the second time, the stored first rollback marked waybill message with message type ROLLBACK is sent to the Kafka message queue. A new result is calculated based on the new message, resulting in the second calculated marked waybill message, whose message type is set to CALCULATE and the version is set to 2. The latest result is written to the Kafka message queue, and then the message type of this message is set to ROLLBACK and stored, resulting in the second rollback marked waybill message. Furthermore, this process will be repeated upon receiving a new message for this waybill. When N messages for this waybill are received, a total of 2N-1 messages will be generated. Messages [1, 2], [3, 4]…[2N-3, 2N-2] appear in pairs. These paired messages differ only in message type; all other data content is identical. This ensures that downstream systems can roll back calculated messages based on message type when receiving data. Simultaneously, the 2N-1th message represents the latest status of this waybill number; its message type is CALCULATE, and downstream systems use this message for calculations.

[0075] It should be noted that each routing segment has a unique and auto-incrementing customization_id field for the same waybill to identify the order of the waybill routing segments for this logistics waybill, i.e., the waybill routing segment ID.

[0076] For example, consider a logistics waybill with waybill number 10000. Shanghai-Nanjing is the first routing segment with a customization_id of 1; Nanjing-Shandong is the second routing segment with a customization_id of 2; and Shandong-Beijing is the third routing segment with a customization_id of 3. For waybill number 10000, upon receiving the first routing segment message with customization_id 1 belonging to this waybill number, a message of type CALCULATE is sent to the intermediate layer, set to version 1. This is the first calculated waybill tag message, where the routing segment information represents the routing segment of this waybill message. Then, the message type of the message sent downstream is set to ROLLBACK, and this message, identical except for the message type, is stored on disk—this is the first rollback tag waybill message. When the second message with a customization_id of 2 is received for this waybill, the first rollback marker waybill message preceding this waybill is read from the disk and sent directly to the middle layer as a rollback of the first calculated marker waybill message. Then, the customization_id of the current message is compared with the disk data. If the customization_ids are different, the two customization_ids are combined, the message type is set to CALCULATE, and sent to the middle layer to obtain the second calculated marker waybill message. Then, its message type is set to ROLLBACK and stored on disk to obtain the second rollback marker waybill message. This process is repeated sequentially.

[0077] Furthermore, after determining that the current waybill message is the Nth calculated marked waybill message, the method further includes:

[0078] The Nth computed mark waybill message is serialized according to a preset format and sent to the Kafka message queue via a transaction.

[0079] It should be noted that after marking the waybill message with message type and version, messages marked as CALCULATE are serialized according to a preset format so that they can be used for subsequent related data calculations and sent to the Kafka message queue via a transaction. In this embodiment, the preset format can be JSON or other formats, and no specific limitation is made here.

[0080] It should be noted that Kafka message queues can also be written without using transactions. However, when downstream processes data sent from Kafka message queues, a deduplication state based on each waybill number version needs to be maintained internally. This is to ensure that when the upstream aggregation task fails and restarts, each message will not be consumed repeatedly, which could lead to over- or under-calculation of the data.

[0081] Further, the step of performing data grouping calculations on the marked waybill messages in the Kafka message queue to generate corresponding summary result data includes:

[0082] The marked waybill messages are grouped according to data dimensions; wherein, the data dimensions include waybill routing segment and waybill scanning date;

[0083] The marked waybill messages with the same waybill route segment and waybill scan date are aggregated to obtain the aggregated result data.

[0084] It should be noted that when receiving marked waybill messages from the Kafka message queue for real-time data computation, the data needs to be grouped according to data dimensions. Data dimensions can be waybill route segments, waybill scan dates, or others. In this embodiment, no specific limitations are placed on the data dimensions.

[0085] When summarizing the result data, waybill messages with the same scan date but different waybill numbers or the same waybill number are aggregated from the waybill route segment to obtain the latest summary result data. The summary result data shows the total number of waybill messages for a specific waybill route segment at a specific time. Real-time calculation results can be obtained directly from the summary result data.

[0086] Further, the process of summarizing the marked waybill messages with the same waybill route segment and waybill scanning date to obtain summary result data includes:

[0087] If the message type of the marked waybill message is CALCULATE, then an addition operation is performed to add the marked waybill message to the summary result data; and the detailed information extracted from the marked waybill message is added to the detailed data of the summary result data.

[0088] If the message type of the marked waybill message is ROLLBACK, then a subtraction operation is performed to remove the marked waybill message from the summary result data; and the detailed information previously extracted from the marked waybill message is removed from the detailed data of the summary result data.

[0089] It should be noted that the specific operations performed when aggregating marked waybill messages with the same waybill route segment and waybill scan date to obtain the aggregated result data are as follows:

[0090] When the message type of the marked waybill message is CALCULATE, an addition operation is performed, adding the marked waybill message to the summary result data, and simultaneously adding the detailed information extracted from the marked waybill message to the detailed data of the summary result data. When the message type of the marked waybill message is ROLLBACK, a subtraction operation is performed, removing the marked waybill message from the summary result data, and simultaneously removing the detailed information previously extracted from the marked waybill message from the detailed data of the summary result data.

[0091] It should be noted that the summary result data is generally the number of summarized messages; the detailed data is the detailed information of each waybill route segment, waybill scanning date, and waybill weight in the corresponding summary result data.

[0092] Suppose there exists a waybill with a route segment of Shanghai-Nanjing-Shandong and a scan date of March 16, 2023, whose current summary value is 100. When it encounters another message of the same dimension with the type CALCULATE, the summary value of this waybill message will become 101; when it encounters another message of the same dimension with the type ROLLBACK, the summary value will become 99. Simultaneously, the corresponding detailed data will be added or removed accordingly.

[0093] Furthermore, the step of sending the summarized result data to the downstream operator of the task for storage includes:

[0094] Store the summarized results in MySQL or Elasticsearch;

[0095] The detailed data of the summarized results are concatenated and stored in HBase.

[0096] It should be noted that the summary result data is sent to the downstream operator of the task for storage. That is, the downstream operator defines the persistence operation on the summary result data and the detailed data of the summary result data. Generally, the summary result data is stored in MySQL or Elasticsearch, and the detailed data of the summary result data is concatenated and written to HBase.

[0097] Please see Figure 2 , Figure 2 This is a block diagram illustrating a low-latency data processing device according to an embodiment of the present invention. The low-latency data processing device 2 includes:

[0098] Message acquisition module 21 is used to acquire waybill messages; wherein the waybill messages include waybill number, waybill routing segment and waybill scan date;

[0099] The message tagging module 22 is used to aggregate multiple waybill messages, tag the waybill messages with message type and version, obtain tagged waybill messages, and send the tagged waybill messages to the Kafka message queue.

[0100] The result data generation module 23 is used to perform data grouping calculation on the marked waybill messages in the Kafka message queue, generate corresponding summary result data, and register a corresponding timer for the summary result data;

[0101] The data persistence operation module 24 is used to send the summarized result data to the downstream operator of the task for storage when the timer is triggered.

[0102] Regarding the control device 2 for the refrigerator compressor in the above embodiments, the specific methods by which each module performs its operation have been described in detail in the embodiments of the above related methods, and will not be elaborated here.

[0103] Please see Figure 3 , Figure 3 This is a block diagram illustrating a low-latency data processing device according to an embodiment of the present invention. The low-latency data processing device 3 includes:

[0104] One or more memories 31, on which executable programs are stored;

[0105] One or more processors 32 are configured to execute the executable program in the memory 31 to implement the steps of any of the methods described above.

[0106] Regarding the low-latency data processing device 3 in the above embodiments, the specific manner in which its processor 32 executes the program in the memory 31 has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0107] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0108] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" or "more" means at least two.

[0109] It should be understood that when an element is referred to as "fixed to" or "set on" another element, it may be directly on the other element or may have an intervening element present at the same time; when an element is referred to as "connected to" another element, it may be directly connected to the other element or may have an intervening element present at the same time. In addition, the term "connected" as used herein may include wireless connections; the word "and / or" as used includes any unit and all combinations of one or more of the associated listed items.

[0110] Any process or method description in the flowchart or otherwise herein can be understood as: representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0111] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0112] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0113] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0114] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0115] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0116] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for achieving low-latency data processing, characterized in that, include: Obtain waybill information; wherein, the waybill information includes waybill number, waybill route segment, and waybill scan date; Multiple waybill messages are aggregated, and each waybill message is labeled with a message type and version to obtain a labeled waybill message. The labeled waybill message is then sent to a Kafka message queue. The process of labeling the waybill message with a message type and version to obtain a labeled waybill message includes: If the waybill message is received for the Nth time, and the message type of this waybill message is set to CALCULATE and the version is set to N, then the current waybill message is determined to be the Nth calculated mark waybill message; and Set the message type of the Nth calculated marked waybill message to ROLLBACK to obtain the Nth rollback marked waybill message, and store the Nth rollback marked waybill message; If a waybill message is received for the (N+1)th time, the Nth rollback-marked waybill message is sent to the Kafka message queue. A calculation is performed between this message and the (N+1)th received waybill message to obtain the (N+1)th calculated marked waybill message. The message type of the (N+1)th calculated marked waybill message is set to CALCULATE, and the version is set to N+1. Set the message type of the (N+1)th calculated marked waybill message to ROLLBACK to obtain the (N+1)th rollback marked waybill message, and store the (N+1)th rollback marked waybill message; The marked waybill message includes a waybill number, waybill routing segment, waybill scan date, message type, and version; the message type of the Nth calculated marked waybill message is different from that of the Nth rollback marked waybill message, but other data content is exactly the same; the waybill number of the Nth calculated marked waybill message is the same as that of the N+1th calculated marked waybill message, and the waybill routing segment ID of the Nth calculated marked waybill message is N, while the waybill routing segment ID of the N+1th calculated marked waybill message is N+1; The marked waybill messages in the Kafka message queue are processed by data grouping and calculation to generate corresponding summary result data, and a corresponding timer is registered for the summary result data; When the timer is triggered, the summarized result data is sent to the downstream operator of the task for storage.

2. The method according to claim 1, characterized in that, The process of obtaining the waybill message includes: The Binlog files in the business database are collected into the RocketMQ message queue; wherein, the Binlog files record the SQL statement information of the user updating the database; The waybill message is obtained by consuming messages in the RocketMQ message queue using the Flink component.

3. The method according to claim 2, characterized in that, The process of aggregating multiple waybill messages includes: Multiple waybill messages with the same waybill number but different waybill routing segments are aggregated.

4. The method according to claim 1, characterized in that, After determining that the current waybill message is the Nth calculated marked waybill message, the method further includes: The Nth computed mark waybill message is serialized according to a preset format and sent to the Kafka message queue via a transaction.

5. The method according to claim 1, characterized in that, The step of performing data grouping and calculation on the marked waybill messages in the Kafka message queue to generate corresponding summary result data includes: The marked waybill messages are grouped according to data dimensions; wherein, the data dimensions include waybill routing segment and waybill scanning date; The marked waybill messages with the same waybill route segment and waybill scan date are aggregated to obtain the aggregated result data.

6. The method according to claim 5, characterized in that, The process of summarizing the marked waybill messages with the same waybill route segment and waybill scanning date to obtain the summarization result data includes: If the message type of the marked waybill message is CALCULATE, then an addition operation is performed to add the marked waybill message to the summary result data; and the detailed information extracted from the marked waybill message is added to the detailed data of the summary result data. If the message type of the marked waybill message is ROLLBACK, then a subtraction operation is performed to remove the marked waybill message from the summary result data; and the detailed information previously extracted from the marked waybill message is removed from the detailed data of the summary result data.

7. The method according to claim 6, characterized in that, The step of sending the summarized result data to the downstream operator of the task for storage includes: Store the summarized results in MySQL or Elasticsearch; The detailed data of the summarized results are concatenated and stored in HBase.

8. A data processing apparatus for achieving low latency, characterized in that, include: The message retrieval module is used to retrieve waybill messages; The waybill message includes the waybill number, waybill routing segment, and waybill scan date; The message tagging module is used to aggregate multiple waybill messages, tag the waybill messages with message type and version to obtain tagged waybill messages, and send the tagged waybill messages to the Kafka message queue; specifically, if a waybill message is received for the Nth time, the message type of this waybill message is set to CALCULATE and the version is set to N, then the current waybill message is determined to be the Nth tagged waybill message. Additionally, the message type of the Nth calculated marked waybill message is set to ROLLBACK to obtain the Nth rollback marked waybill message, and the Nth rollback marked waybill message is stored. If a waybill message is received for the (N+1)th time, the Nth rollback marked waybill message is sent to the Kafka message queue. The (N+1)th calculated marked waybill message is obtained by calculating with the waybill message received for the (N+1)th time, and the message type of the (N+1)th calculated marked waybill message is set to CALCULATE and the version is set to N+1. Furthermore, the message type of the (N+1)th calculated marked waybill message is set to ROLLBACK to obtain the (N+1)th rollback marked waybill message, and the (N+1)th rollback marked waybill message is stored; wherein, the marked waybill message includes waybill number, waybill routing segment, waybill scan date, message type, and version; the message type of the Nth calculated marked waybill message and the Nth rollback marked waybill message are different, but other data content is exactly the same; the waybill number of the Nth calculated marked waybill message and the (N+1)th calculated marked waybill message is the same, and the waybill routing segment ID of the Nth calculated marked waybill message is N, and the waybill routing segment ID of the (N+1)th calculated marked waybill message is N+1; The result data generation module is used to perform data grouping calculations on the marked waybill messages in the Kafka message queue, generate corresponding summary result data, and register a corresponding timer for the summary result data; The data persistence module is used to send the summarized result data to the downstream operator of the task for storage when the timer is triggered.

9. A data processing device for achieving low latency, characterized in that, include: One or more memories on which executable programs are stored; One or more processors are configured to execute the executable program in the memory to implement the steps of the method according to any one of claims 1-7.

Images