Data processing methods and apparatus

Abstract

This invention discloses a data processing method and apparatus. The invention includes: filtering target orders from a database, wherein the target orders are incomplete and their stream processing status data is about to expire; generating a simulated order update message corresponding to the target order based on the information of the target order and according to the message format of a first message queue; writing the simulated order update message into the first message queue, causing stream processing operators in the stream processing framework to consume the simulated order update message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database. This invention can accurately and flexibly obtain status maintenance objects, improve status maintenance efficiency, fully utilize the original system performance to update status expiration times, and accurately monitor orders with long completion cycles without increasing storage capacity. It provides a solid foundation for the full utilization and rational allocation of data resources, greatly improving the flexibility of business status monitoring and stream processing.

Description

Technical Field

[0001] This invention relates to the field of automated data processing technology, and more particularly to data processing methods and apparatus. Background Technology

[0002] Order data is often stored in a database sharding and table partitioning manner due to its large volume. Data statistics cannot be easily completed. A stream processing framework is usually used to access the database log file, process the multiple data tables into wide tables, and store the wide tables in an online analysis database. The data required for statistics can be obtained by querying the wide tables in the online analysis database.

[0003] When processing wide tables, stream processing status data needs to be temporarily stored in the key-value database of the stream processing framework according to a preset lifespan. During temporary storage, a shorter lifespan results in less temporary data and less storage capacity, while a longer lifespan requires more. To ensure that most orders are monitored, the lifespan must be long enough, inevitably leading to a large amount of temporary data. However, this slows down the startup or shutdown of the stream processing framework, affecting the speed of consuming online messages. Furthermore, order completion periods vary, and stream processing status data after order completion does not need to be temporarily stored. A long preset lifespan results in a large amount of un-temporarily stored data in the key-value database. In addition, some orders will have completion periods exceeding the lifespan, causing their stream processing status data to expire and become invalid, resulting in unmonitored order statuses due to lack of updates. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data processing method and apparatus, the purpose of which is to achieve accurate monitoring of orders with any completion cycle without increasing storage capacity, so as to solve the problems existing in the background art.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a data processing method is provided, comprising:

[0006] Filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework.

[0007] Based on the information of the target order, an order update simulation message corresponding to the target order is generated according to the message format of the first message queue;

[0008] The simulated order update message is written to the first message queue, so that the stream processing operator in the stream processing framework consumes the simulated order update message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

[0009] Furthermore, filtering target orders from the database, wherein the order status of the target orders is incomplete and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework, includes: obtaining a stream processing result wide table from the database; filtering the target orders from the stream processing result wide table, wherein the order status of the target orders is incomplete and the difference between the expiration time of the stream processing status data corresponding to the target orders in the key-value database of the stream processing framework and the current time is less than or equal to a preset time threshold.

[0010] Furthermore, based on the information of the target order and in accordance with the message format of the first message queue, an order update simulation message corresponding to the target order is generated, including: selecting one or more sets of fields and corresponding field contents of the target order from the first message queue from the wide table of the stream processing results;

[0011] Based on one or more sets of fields of the target order and their corresponding field contents, an order update simulation message corresponding to the target order is generated in accordance with the message format of the first message queue.

[0012] Furthermore, the one or more sets of fields and their corresponding field contents are order information unrelated to the order status.

[0013] Furthermore, the lifespan of the stream processing status data corresponding to the target order in the key-value database is longer than the preset lifespan period of the stream processing status data in the key-value database.

[0014] Furthermore, the stream processing operator in the stream processing framework consumes the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order stored in the key-value database. This includes: the stream processing operator in the stream processing framework processes the first message queue and the second message queue flowing through the stream processing operator, and during the processing, by querying and / or updating the stream processing status data in the key-value database, updates the expiration time of the stream processing status data corresponding to the target order in the key-value database. After the update, the expiration time is the update time plus the lifespan of the key-value database. The first message queue is used to store update messages for order information or update messages for order status, and the second message queue is used to store update messages for waybill check-in information.

[0015] Furthermore, it also includes: filtering out stream processing status data with an expiration time less than the current time from the key-value database, and deleting the stream processing status data.

[0016] According to a second aspect of the present invention, a data processing apparatus is provided, comprising:

[0017] The order filtering module is used to filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework.

[0018] The message generation module is used to generate a simulated order update message corresponding to the target order based on the information of the target order and in accordance with the message format of the first message queue.

[0019] The message writing module is used to write the order update simulation message into the first message queue, so that the stream processing operator in the stream processing framework consumes the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

[0020] According to a third aspect of the present invention, an electronic processing apparatus is provided, comprising:

[0021] One or more processors; and,

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, cause the one or more processors to implement the method provided in the first aspect of the present invention.

[0024] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0025] According to a fifth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0026] One embodiment of the invention has the following advantages or beneficial effects:

[0027] By filtering orders whose stream processing status data is about to expire but not yet completed, the system can accurately and flexibly obtain status maintenance objects, improving status maintenance efficiency. By generating order update simulation messages and adding them to the first message queue, the order update simulation messages can update the expiration time of the stream processing status data corresponding to the orders, making full use of the original system performance. Without increasing storage capacity, the system can accurately monitor orders with long completion cycles, providing a solid foundation for the full utilization and rational allocation of data resources, and greatly improving the flexibility of business status monitoring and stream processing. Attached Figure Description

[0028] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0029] Figure 1 This is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the principle of the data processing method applied in this embodiment of the invention;

[0031] Figure 3 This is a schematic diagram illustrating the principle of a data processing method applying another embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of a data processing system according to an embodiment of the present invention;

[0034] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0035] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0036] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0038] To address the problems in the background technology, this invention proposes a data processing method, including filtering orders whose stream processing status data is about to expire but has not been completed, generating an order update simulation message and adding it to a first message queue, so that the order update simulation message is consumed by various stream processing operators of the stream processing framework, updating the expiration time of the stream processing status data corresponding to the order. This invention can accurately and flexibly obtain status maintenance objects, improve status maintenance efficiency, fully utilize the original system performance to update status expiration times, and accurately monitor orders with long completion cycles without increasing storage capacity. This provides a solid foundation for the full utilization and rational allocation of data resources, and greatly improves the flexibility of business status monitoring and stream processing.

[0039] Figure 1 This is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the principle of the data processing method applied in this embodiment of the invention.

[0040] like Figure 1 and Figure 2 As shown, the data processing method of the present invention in one embodiment includes the following steps S100 to S300.

[0041] S100, filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework.

[0042] Specifically, such as Figure 2 As shown, the order system uses a database sharding and table partitioning approach for data storage. The data tables include a master table and slave tables, with the master table typically corresponding to one or more slave tables. The master table stores the primary data and usually contains a unique identifier for each record, such as a primary key. Slave tables store detailed data related to the master table and typically contain a foreign key referencing the master table's primary key. This foreign key establishes the relationship between the master and slave tables, ensuring data consistency. The stream processing framework accesses the binary log files of the master table and one or more slave tables. These binary log files (e.g., the binlog of a MySQL database) record database operations for data replication and recovery. The stream processing framework generates message queues for the corresponding data tables based on their binary logs. Figure 2 The diagram shows the message flow of the main table and the message flow of the secondary table. Specifically, Figure 2 The diagram illustrates a master data table and a corresponding slave data table. Correspondingly, the message queue includes message streams from the master table and message streams from the slave table. The stream processing framework includes a series of stream processing operators (…). Figure 2(Only one example is shown for illustrative purposes, not as a limitation.) The main table message stream and the slave table message stream are processed to obtain a processing result message stream. Based on the processing result message stream, a wide table containing all statistical data is obtained. When the stream processing operator consumes the data queue of the data table, it needs to temporarily store the stream processing status data in the key-value database. The lifespan of the stream processing status data, or the duration for which the status is valid, is limited by the preset lifespan period in the key-value database. The expiration time of the stream processing status data is its creation time plus the lifespan period. When the expiration time is greater than the current time, the stream processing status data is valid; when the expiration time is less than or equal to the current time, the stream processing status data is invalid. Invalid or expired stream processing status data can be deleted to save storage space and improve access efficiency.

[0043] In the existing order system, the key-value database's lifespan is initially set to 7 days. However, due to factors such as traffic control, road damage, severe weather, and the pandemic, some orders cannot be delivered within 7 days and their data remains unchanged. The stream processing status data temporarily stored in the key-value database expires after the storage period exceeds the lifespan, leading to inaccurate order monitoring. Therefore, the initial setting cannot support the business in this scenario. Extending the lifespan to 30 days to support this business would quadruple the data storage volume for stream processing status data, which would negatively impact the storage and recovery of this data. Furthermore, regardless of the lifespan setting, some orders will always exceed the lifespan, such as orders that haven't been completed within 30 days or orders created 30 days in advance. These orders will be unable to be monitored properly after 30 days, impacting business operations. Therefore, in this embodiment of the invention, to ensure effective monitoring of these orders, orders that may not be properly monitored are first filtered out.

[0044] Specifically, in some embodiments of the present invention, step S100 includes: obtaining a wide table of stream processing results from a database; filtering the target order from the wide table of stream processing results, wherein the order status of the target order is incomplete, and the difference between the expiration time of the stream processing status data corresponding to the target order in the key-value database of the stream processing framework and the current time is less than or equal to a preset time threshold. The initial value of the expiration time is the initial creation time of the stream processing status data plus the preset lifespan of the key-value database.

[0045] Specifically, through custom SQL statements, from Figure 2 The target order is selected from a wide table of stream processing results obtained from an online analytics database (e.g., the ClickHouse database).

[0046] Understandably, in the embodiments of the present invention, the time threshold is preset and is greater than zero and less than the preset lifespan of the key-value database. For example, the preset lifespan of the key-value database is 7 days, and the time threshold is preset to 3 days or 2 days. Furthermore, in some embodiments of the present invention, to reduce the amount of data processed while taking into account the update cycle of actual order business data, the time threshold may be preset to 1 day. This considers both minimizing the time threshold to reduce the amount of data and the fact that actual order business data is updated daily, thereby flexibly and accurately maintaining the status of orders whose stream processing status data is about to expire and which are not yet completed, improving maintenance efficiency and saving computing resources.

[0047] Furthermore, in some specific embodiments of the present invention, step S100 is implemented by periodically triggering preset SQL statements to obtain the target order. Multiple periodic triggers ensure that orders whose stream processing status data is about to expire and are incomplete during each monitoring period in the business operation process are filtered out. Each trigger only processes the most urgent orders; that is, each trigger filters out orders that meet the filtering condition that the order status at the current time is incomplete and the stream processing status data is about to expire. In the next trigger, orders whose stream processing status data is about to expire and are incomplete in subsequent periods are filtered out through subsequent periodic triggers, thereby achieving accurate monitoring of orders of any storage duration without increasing storage capacity. In these embodiments, obtaining the stream processing result wide table from the database includes: automatically obtaining the stream processing result wide table from the online analysis and processing database in response to the occurrence of the trigger condition. It can be understood that the occurrence of the trigger condition includes: the current time reaching a preset timing end point, or the current time reaching a time point where the time from the preset timing start point is an integer multiple of a preset time interval. Using a periodic trigger method, daily scheduled triggering can be selected, consistent with the actual update cycle of most order business data.

[0048] S200, based on the information of the target order, and in accordance with the message format of the first message queue, generate a simulated order update message corresponding to the target order.

[0049] Understandably, a message queue is a message queue within an access stream processing framework. Specifically, in some embodiments of the present invention, step S200 includes: selecting one or more sets of fields and corresponding field contents from the target order in the stream processing result wide table; generating an order update simulation message corresponding to the target order according to the one or more sets of fields and corresponding field contents of the target order and the message format of the first message queue. Wherein, for the message queue, the term "first" is used only for purpose and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Features defined by terms such as "first" and "second" in the present invention may explicitly or implicitly include at least one of those features. Thus, the first message queue represents the target for generating the order update simulation message, rather than a limitation on the order update simulation message.

[0050] Understandably, the first message queue, as the source of one or more sets of fields and their corresponding field contents, preferentially selects the message queue that flows through all the stream processing operators in the stream processing framework, such as the main table message queue. This way, only one message queue order update simulation message needs to be generated to update the stream processing status data of the target order in all key-value databases. Optionally, the first message queue may also flow through only some of the stream processing operators' message queues in the stream processing framework. By selecting multiple first message queues, multiple corresponding order update simulation messages are generated, each flowing through the stream processing operator corresponding to its respective first message queue, thus enabling all stream processing operators to update their respective stream processing status data.

[0051] Understandably, the simulated order update message is not a real status update message; the order status has not actually changed. Therefore, the simulated order update message can use the selected corresponding field content as the updated field content, meaning the field content is the same before and after the update. Optionally, the one or more sets of fields and their corresponding field content are order information unrelated to the order status, further isolating the simulated order update message from factors that may affect the order status, to prevent unforeseen situations from impacting business operations.

[0052] S300, the order update simulation message is written into the first message queue, so that the stream processing operator in the stream processing framework consumes the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

[0053] In some embodiments of the present invention, specifically, step S300 includes: the stream processing operator in the stream processing framework processes the first message queue and the second message queue flowing through the stream processing operator, and during the processing, by querying and / or updating the stream processing status data in the key-value database, the expiration time of the stream processing status data corresponding to the target order in the key-value database is updated. After the update, the expiration time is equal to the update time plus the lifespan. The first message queue and the second message queue are not limited to the number of message queues processed by the stream processing operator in the stream processing framework, but are merely used to indicate that the message queues processed by the stream processing operator include both message queues containing simulated order update messages and message queues not containing simulated order update messages. The present invention can simulate changes in order status using one or more message queues in the original order system, which is simple and effective.

[0054] Specifically, such as Figure 2 As shown, the master table stores order information, and the slave tables store order status. The first message queue is a master table message stream obtained by accessing the binary log file of the master table through a stream processing framework, used to store update messages for order information. The second message queue is a slave table message stream obtained by accessing the binary log file of the order status table through a stream processing framework, used to store update messages for order status. In response to the simulated order update message, each stream processing operator processes the data. Through querying and / or updating the stream processing status data in the key-value database during processing, the expiration time of the stream processing status data corresponding to the target order in the key-value database is updated. After the update, the expiration time is the update time plus the lifespan of the key-value database.

[0055] Understandably, in actual business systems, there are situations where orders are unexpectedly delayed or pre-ordered orders have long completion cycles, resulting in the stream processing status data of the target order exceeding its expiration date without business data updates, yet still needing to remain valid. Specifically, this includes: first, an order is unexpectedly delayed, meaning the order could have been processed before the normal expiration time of the stream processing status data, but due to the unexpected lack of business data updates, the stream processing status data needs to remain valid after the expiration time, i.e., exceeding the preset lifespan of the key-value database; second, the order's completion cycle is long, exceeding the preset lifespan of the key-value database, and its stream processing status data, lacking business data updates, still needs to remain valid after the expiration time. In both cases, the lifespan of the stream processing status data corresponding to the target order in the key-value database is longer than the preset lifespan of the stream processing status data in the key-value database. This invention ensures that the stream processing status data in both cases can be normally stored and restored before the order is completed.

[0056] Specifically, for example, an order is created at 8:00 AM on December 1, 2024. The preset lifespan of the key-value database is 7 days, and the initial expiration time of the corresponding stream processing status data is 8:00 AM on December 8, 2024. If the normal completion period for this order is 5 days, but the completion date cannot be determined due to logistics delays caused by weather, or if the completion period for this order is 30 days, the order status would normally not be able to be monitored normally after 8:00 AM on December 8, 2024. Applying the method in this embodiment of the invention and setting the time threshold to 1 day, assuming... If an order is selected at 9:00 AM on December 7, 2024, the expiration time of the corresponding stream processing status data will be updated to 9:00 AM on December 14, 2024. This order can be monitored normally until the new expiration time. If the order is not completed by 9:00 AM on December 13, 2024, the method in this embodiment of the invention will be applied again to select the order, and the expiration time of the corresponding stream processing status data will be updated to 9:00 AM on December 21, 2024. This order can still be monitored normally until this expiration time. Therefore, the method in this embodiment of the invention can monitor orders that are not completed due to unexpected circumstances and orders with long completion periods, even with initial settings for the key-value database's lifespan or more stringent settings, achieving accurate monitoring of orders with any completion period.

[0057] Understandably, in some specific embodiments of the present invention, orders whose streaming processing status data is about to expire but have not been completed can be obtained using only simple SQL statements and task triggering settings (such as timed triggering). Then, an order update simulation message is generated and written to the first message queue to simulate the change of order status, thereby updating the expiration time of the streaming processing status data corresponding to the order. In addition, the present invention avoids intruding into the database of the upstream system (such as the MySQL production database) or calling the upstream system interface to update the order status, and does not operate on the real data of the upstream system. Therefore, the order status simulation update implementation method of the present invention is simple and effective, and does not affect the data storage of the upstream system, which is beneficial to system integration.

[0058] Furthermore, in order to save storage space in the key-value database and ensure the storage and recovery efficiency of stream processing status data during stream processing, some embodiments of the present invention further include: filtering out stream processing status data with an expiration time less than the current time from the key-value database and deleting the stream processing status data.

[0059] Understandably, this embodiment of the invention extends the lifespan of streaming processing status data for orders whose status data is about to expire but are not yet completed. For streaming processing status data of completed orders, it can be deleted if it expires, thereby freeing up storage space in the key-value database, reducing hardware capacity requirements, and lowering costs. Furthermore, due to the shorter lifespan, the amount of storage in the key-value database is reduced, allowing for faster program startup from checkpoints and faster program shutdown and saving of checks. This results in faster program startup and more stable operation during streaming processing.

[0060] To further understand the inventive concept of this invention, Figure 3 A schematic diagram illustrating the principle of a data processing method applying another embodiment of the present invention is provided. The following is in conjunction with... Figure 3 The method of the present invention will be further described in detail below.

[0061] In this embodiment, the business system is a logistics order system, the business database uses a MySQL database, the stream processing framework uses Flink, the key-value database is RocksDB, and the online analysis database uses ClickHouse. Figure 3 As shown, solid lines represent the original order system process, while dashed lines represent the process using the method described in this embodiment of the invention. In the original process, MySQL table A serves as the master table for order information, storing order information. Fields in MySQL table A include order number, order time, product number, payment method, payment time, delivery method, recipient information, delivery address, and delivery method. MySQL table B serves as a slave table for order status, originating from the upstream production system, storing order status. Fields in table B include order number and order status. The "order status" field includes information such as payment on delivery, pending receipt, pending service, completed, and pending evaluation. MySQL table C serves as a slave table for order mini-program check-in information, originating from the mini-program system, storing order mini-program check-in information. Fields in MySQL table C include order number, logistics order number, warehouse check-in information, sorting center check-in information, branch office check-in information, courier check-in information, and recipient signature information.

[0062] Specifically, assume that the lifespan of stream processing state data in the current stream processing framework is 7 days. For example... Figure 3 As shown, the stream processing framework collects data from the binary log files (binlog) of MySQL tables A, B, and C in the upstream system's MySQL database, resulting in three message queues: message stream A, message stream B, and message stream C. The first message queue (corresponding to...) Figure 3The message stream A shown is used to store update messages for order information. When data in MySQL table A changes, for example, the recipient information for order number 369 in MySQL table A changes to "Zhang San", the stream processing framework will collect the corresponding update message from MySQL table A and write it to the first message queue. The format of the update message is, for example, {order number《》}. <369> Change the field <Consignee Information> and the field content <"Zhang San>}. Second message queue (corresponding to Figure 3 Message stream B shown is used to store update messages for order status. When data in MySQL table B changes, for example, the order status corresponding to order number 369 in MySQL table B changes to "completed," the stream processing framework will collect the corresponding update message from MySQL table B and write it to the second message queue. The format of the update message is, for example, {order number}. <369> Change the field <Order Status> and the field content <“Completed”>. Third message queue ( Figure 3 The message stream C shown is used to store update messages for order mini-program check-in information. When data in MySQL table C changes, for example, after the recipient's signature information for order number 369 in MySQL table C is written as "Signed by myself", the stream processing framework will collect the corresponding update message from MySQL table C and write it to the second message queue. The format of the update message is, for example, {order number}. <369> Change the fields <Consignee Signature Information> and <“I Signed for This”>. Widen the data using two association operators in the Flink stream processing framework. Figure 3 The `flowA-flowB association operator`, `mergingflowAB-flowC association operator`, etc., stretch the order information, status information, and mini-program check-in information of an order into a wide table of stream processing results, which is then stored in the online analytics database ClickHouse for business report analysis. For example... Figure 3As shown, the two association operators widening are specifically as follows: First, the status information of the associated order is widened through the association operator of flow A and flow B. The flow processing status data is stored in the key-value database A of the flow processing framework. Second, the mini-program check-in information of an order is obtained by merging the association operators of flow AB and flow C. The flow processing status data is stored in the second key-value database B. The lifespan in key-value databases A and B is 7 days. Each query updates the expiration time and extends it by 7 days. The lifespan of the status related to all association operators in the flow processing framework is fixed. Different lifespans cannot be set for the flow processing status of different association operators. When the flow processing status data in the key-value database is created, the expiration time is determined according to the lifespan. That is, the expiration time is equal to the creation time plus the lifespan. If the key corresponding to the flow processing status data is queried and / or the value corresponding to the key is updated each time, the expiration time of all flow processing status data corresponding to that key will be updated. The updated expiration time is the update time plus the lifespan. In this embodiment, the method of the present invention is applied. First, orders that have not been completed for a certain period (e.g., 5 days or less) are filtered out from the online analytical processing database. Then, simulated order update messages are generated and written to message stream A, i.e., the first message queue that flows through all stream processing operators. For example, order number 369 is selected from the wide table. The simulated update message generated according to the format of the first message queue is {order number}, using the fields "order number" and "369", and "recipient information" and "Zhang San" from the first message queue. <369> The `<Order Status>` field and its content `<"Completed">` are modified. The simulated order update message, consumed by the two associated operators as the first message queue is processed, updates the expiration time of the stream processing status data corresponding to the target order in key-value databases A and B. Furthermore, the lifespan of the stream processing status data in the key-value database of the stream processing framework can be set to 7 days or less, without needing to extend it to 30 days for orders with longer completion cycles. This reduces the amount of cold data or invalid data from completed orders stored in the key-value database, allowing the required storage capacity for the business to remain unchanged or even decrease.

[0063] According to another aspect of the embodiments of the present invention, such as Figure 4 As shown, a data processing apparatus 400 is provided, comprising:

[0064] The order filtering module 401 is used to filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework.

[0065] The message generation module 402 is used to generate an order update simulation message corresponding to the target order based on the information of the target order and in accordance with the message format of the first message queue;

[0066] The message writing module 403 is used to write the order update simulation message into the first message queue, so that the stream processing operator in the stream processing framework consumes the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

[0067] In some embodiments of the present invention, the data processing device is applied to the logistics waybill system in the following specific ways: Figure 5 As shown. The order system logistics order system includes a business database, a stream processing framework, and an online analysis database. The business database contains a master data table and slave data tables. The order filtering module of the data processing device is connected to the online analysis database, and the message writing module is connected to the master table message stream, i.e., connected to the first message queue. The order update simulation message generated and written by the data processing device flows sequentially through each stream processing operator of the stream processing framework along with the first message queue. Each stream processing operator updates the expiration time of the stream processing status data stored in the key-value database corresponding to the target order according to the order update simulation message.

[0068] Figure 6 An exemplary system architecture 600 is shown that can be applied to the data processing method or data processing apparatus of the present invention.

[0069] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, network 605, and server 605. Network 604 is used as a medium to provide a communication link between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0070] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0071] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0072] Server 605 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603 (this is just an example). The backend management server can analyze and process data such as received product information query requests and feed the processing results back to the terminal devices.

[0073] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by the server 605, and correspondingly, the data processing device is generally located in the server 605.

[0074] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0075] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0076] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0077] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0078] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0079] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0081] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a matching module, a deduplication module, and a recombination module. The names of these modules do not necessarily limit the module itself; for example, the recombination module can also be described as "a module that recombines modules according to relationships."

[0082] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: filtering target orders from a database, wherein the order status of the target orders is incomplete and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework; generating a simulated order update message corresponding to the target orders based on the information of the target orders and according to the message format of a first message queue; writing the simulated order update message into the first message queue, causing a stream processing operator in the stream processing framework to consume the simulated order update message, thereby updating the expiration time of the stream processing status data corresponding to the target orders in the key-value database.

[0083] According to the technical solution of the embodiments of the present invention, the present invention has the following advantages or beneficial effects: by filtering orders whose stream processing status data is about to expire but has not been completed, the status maintenance objects can be accurately and flexibly obtained, improving the efficiency of status maintenance; by generating order update simulation messages and adding them to the first message queue, the order update simulation messages can update the expiration time of the stream processing status data corresponding to the order, making full use of the original system performance update, and accurately monitoring orders with long completion cycles without expanding storage capacity, providing a solid foundation for the full utilization and reasonable allocation of data resources, and greatly improving the freedom of business status monitoring and stream processing.

[0084] The specific embodiments described herein do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework. Based on the information of the target order, an order update simulation message corresponding to the target order is generated according to the message format of the first message queue; The simulated order update message is written to the first message queue, so that the stream processing operator in the stream processing framework consumes the simulated order update message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

2. The method according to claim 1, characterized in that, Target orders are filtered from the database, where the order status is incomplete and the corresponding stream processing status data is about to expire in the stream processing framework's key-value database, including: Retrieve the wide table of stream processing results from the database; The target order is selected from the wide table of stream processing results. The order status of the target order is incomplete. Furthermore, the difference between the expiration time and the current time of the stream processing status data corresponding to the target order in the key-value database of the stream processing framework is less than or equal to a preset time threshold.

3. The method according to claim 2, characterized in that, Based on the information of the target order, and according to the message format of the first message queue, a simulated order update message corresponding to the target order is generated, including: Select one or more sets of fields and their corresponding field contents from the first message queue from the wide table of stream processing results; Based on one or more sets of fields of the target order and their corresponding field contents, an order update simulation message corresponding to the target order is generated in accordance with the message format of the first message queue.

4. The method according to claim 3, characterized in that, The set or more fields and their corresponding field contents are order information unrelated to the order status.

5. The method according to claim 1, characterized in that, The lifespan of the stream processing status data corresponding to the target order in the key-value database is longer than the preset lifespan period of the stream processing status data in the key-value database.

6. The method according to claim 1, characterized in that, The stream processing operators in the stream processing framework consume the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order stored in the key-value database, including: The stream processing operator in the stream processing framework processes the first message queue and the second message queue that flow through the stream processing operator. During the processing, the expiration time of the stream processing status data corresponding to the target order in the key-value database is updated by querying and / or updating the stream processing status data in the key-value database. After the update, the expiration time is the update time plus the lifespan of the key-value database. The first message queue is used to store update messages for order information, and the second message queue is used to store update messages for order status.

7. The method according to claim 1, characterized in that, Also includes: Filter out stream processing status data whose expiration time is less than the current time from the key-value database, and delete the stream processing status data.

8. A data processing apparatus, characterized in that, include: The order filtering module is used to filter target orders from the database. The order status of the target orders is incomplete, and the stream processing status data corresponding to the target orders is about to expire in the key-value database of the stream processing framework. The message generation module is used to generate a simulated order update message corresponding to the target order based on the information of the target order and in accordance with the message format of the first message queue. The message writing module is used to write the order update simulation message into the first message queue, so that the stream processing operator in the stream processing framework consumes the order update simulation message, thereby updating the expiration time of the stream processing status data corresponding to the target order in the key-value database.

9. An electronic processing device, characterized in that, include: One or more processors; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images