Data processing methods, apparatus, electronic equipment and software products

By using the BKDR hash algorithm and stack adjustment technology, the problems of uneven queue allocation and uneven throughput were solved, thereby improving the orderliness of data processing and resource utilization.

CN116795556BActive Publication Date: 2026-11-17CHINA MOBILE GROUP JIANGSU +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210249426.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-14
Publication Date
2026-11-17
Estimated Expiration
2042-03-14

AI Technical Summary

Technical Problem

In existing technologies, concurrent data transmission across multiple queues suffers from uneven queue allocation and throughput, leading to resource waste and difficulty in guaranteeing the order of data processing.

Method used

The BKDR hash algorithm is used to evenly distribute the data to be processed into various queues. When messages are not consumed in the preset order, the unconsumed messages are sent to the first stack for order adjustment, ensuring that the data is consumed in order at the consumer end.

Benefits of technology

It achieves queue load balancing and data processing order, improving resource utilization and overall performance, and reducing queue resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116795556B_ABST
    Figure CN116795556B_ABST
Patent Text Reader

Abstract

This application relates to the field of data processing technology, and provides a data processing method, apparatus, electronic device, and program product. The method includes: sending data to be processed to various queues, the data to be processed including multiple messages with the same data identifier; if it is determined that the messages are not consumed in a preset order, sending the unconsumed messages in each queue to a first stack for order adjustment to obtain an adjusted target message; and sending the target message to a consumer for consumption. The data processing method, apparatus, electronic device, and program product provided in this application can ensure the sequentiality of data processing while ensuring queue load balancing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, electronic device, and program product. Background Technology

[0002] Currently, message push notifications use a multi-queue concurrent data transmission method. However, due to differences in transmission speed among different queues or varying consumption capabilities of downstream consumers, out-of-order delivery can occur.

[0003] The current processing method involves first specifying the primary key field of the data, then taking the modulo between the primary key field value and the queue number to obtain the queue number, and finally encapsulating the data into a message and sending it to the queue with the corresponding number. Since data with the same primary key will always be pushed to the same queue, the sequential push of data with the same primary key can be guaranteed.

[0004] The existing solution has the following drawbacks:

[0005] 1. The modulo algorithm is uneven, resulting in unbalanced load. The modulo method used for primary key field values ​​can easily lead to uneven queue allocation, where the overall processing time for a batch of data is equal to the processing completion time of the queue receiving the most messages.

[0006] 2. Different queues have different throughput, resulting in wasted resources. When most primary key modulo values ​​are the same in a batch of data, other queues may quickly complete their push operations while waiting for the last queue to finish, causing uneven throughput and wasting queue resources. Summary of the Invention

[0007] This application provides a data processing method, apparatus, electronic device, and program product to solve the technical problem of not being able to simultaneously guarantee the order of data and queue load balancing.

[0008] In a first aspect, embodiments of this application provide a data processing method, including:

[0009] The data to be processed is sent to each queue, and the data to be processed includes multiple messages with the same data identifier;

[0010] If it is determined that the message is not consumed in the preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message;

[0011] The target message is sent to the consumer for consumption.

[0012] In one embodiment, the fact that the messages are not consumed in a preset order is determined in the following way:

[0013] The sequence number corresponding to the message is detected at the end of each queue;

[0014] If the sequence number corresponding to the message does not match the target sequence number, it is determined that the message was not consumed in the preset order.

[0015] In one embodiment, before sending unconsumed messages from each queue to the first stack for reordering, the method further includes:

[0016] If no unconsumed message is detected, the second stack corresponding to the unconsumed message is determined based on the data identifier;

[0017] Based on the sequence number corresponding to the message, the unconsumed message in the second stack is sent to the first stack.

[0018] In one embodiment, the method further includes:

[0019] The sequence number corresponding to the message is detected at the end of each queue;

[0020] If the sequence number corresponding to the message matches the target sequence number, the message is consumed and the target sequence number is updated.

[0021] In one embodiment, before sending unconsumed messages from each queue to the first stack for reordering, the method further includes:

[0022] Based on the sequence number and queue index corresponding to the message, the unconsumed message is determined, and the queue index is used to indicate the queue where the unconsumed message is located.

[0023] In one embodiment, sending the data to be processed to each queue includes:

[0024] Based on the BKDR hash algorithm, the data to be processed is sent to each queue respectively.

[0025] In one embodiment, the unconsumed message is the message corresponding to the preceding sequence number, where the preceding sequence number is the number before the sequence number corresponding to the message.

[0026] Secondly, embodiments of this application provide a data processing apparatus, including:

[0027] The sending module is used to send the data to be processed to various queues, wherein the data to be processed includes multiple messages with the same data identifier;

[0028] The adjustment module is used to send unconsumed messages from each queue to the first stack for order adjustment when it is determined that the messages have not been consumed in a preset order, so as to obtain the adjusted target message.

[0029] The consumption module is used to send the target message to the consumer for consumption.

[0030] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the data processing method described in the first aspect.

[0031] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect.

[0032] This application provides a data processing method, apparatus, electronic device, and program product that ensures queue load balancing by sending the data to be processed to various queues. Furthermore, if it is determined that messages are not consumed in a preset order, the unconsumed messages in each queue are sent to a first stack for order adjustment to obtain the adjusted target message. The target message is then sent to the consumer for consumption, thus ensuring the sequential nature of data processing while maintaining queue load balancing. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of the shared center structure for applying the data processing method provided in the embodiments of this application;

[0036] Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application;

[0037] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0038] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The data processing method provided in this application can be executed by an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), ATM, or self-service machine, etc., and this application does not impose specific limitations.

[0041] The technical solution of this application will be described in detail below using a computer executing the data processing method provided in this application as an example.

[0042] Figure 1 This is a schematic flowchart illustrating the data processing method provided in an embodiment of this application. (Refer to...) Figure 1 This application provides a data processing method, which may include steps 110, 120 and 130.

[0043] Step 110: Send the data to be processed to each queue. The data to be processed includes multiple messages with the same data identifier.

[0044] It is understood that the data to be processed can follow a producer-consumer model, that is, the producer can generate data to be processed and then put the data to be processed into a queue, and the consumer can take the data to be processed from the queue and consume it. This application does not specifically limit the producer.

[0045] In practice, the data group corresponding to the data to be processed can be encapsulated into multiple messages with the same data identifier. The message includes a message header, which stores the message's data identifier, and the data identifier indicates the common attributes of a group of data.

[0046] Messages corresponding to the same data identifier can be used as a dataset. The message header is also used to record the number of data (messages) in the dataset, as well as the position or preset order of each data (message) in the dataset.

[0047] Data identifiers can be primary key fields for the current data or unique identifiers for messages. Different data identifiers correspond to different data, and the datasets corresponding to different data identifiers are independent of each other.

[0048] In existing technologies, to ensure that messages with the same data identifier are consumed in sequence, they must be pushed to the same queue. However, in this embodiment, the data to be processed can be sent to various queues to ensure queue load balancing.

[0049] In one embodiment, sending the data to be processed to each queue includes:

[0050] Based on the BKDR hash algorithm, the data to be processed is sent to each queue respectively.

[0051] The BKDR hash algorithm is a string hash algorithm that can map strings to integers.

[0052] Based on the BKDR hash algorithm, the data to be processed can be evenly distributed and sent to all queues, so that different queues have similar throughput.

[0053] The data processing method provided in this application embodiment, by sending the data to be processed to each queue based on the BKDR hash algorithm, can maximize the utilization of each queue, make full use of resources, and improve overall performance.

[0054] Step 120: If it is determined that the message has not been consumed in the preset order, send the unconsumed messages in each queue to the first stack for order adjustment to obtain the adjusted target message.

[0055] Messages with the same data identifier need to be consumed in a preset order, and messages are sent to various queues.

[0056] To ensure the sequential nature of data processing, it is necessary to determine whether messages are consumed in the preset order stored in the message header before they reach the consumer.

[0057] If a message is not consumed in the preset order, it means that the data processing order is disordered. In this case, the message and all unconsumed messages corresponding to the data identifier are sent to the first stack for order adjustment.

[0058] For example, in a batch of data, a primary key can correspond to 10 messages. The data sender will sequentially number the messages 1-10 in the message headers of these 10 messages, indicating that the 10 messages need to be pushed to the consumer in numerical order from 1 to 10. When a message reaches the end of the queue, its sequential number will be recorded. The message order received by the consumer should be: message corresponding to number 1, message corresponding to number 2, message corresponding to number 3, ... message corresponding to number 10.

[0059] If it is found that the messages are not arriving on demand, all unconsumed messages in the queues are sent to the first stack for reordering to obtain the adjusted target message.

[0060] Understandably, consumed messages are not subject to reordering in this step.

[0061] For example, if the consumer receives message number 5, it means messages numbered 1 to 4 have already been consumed. However, if the first message to reach the end of a queue is message number 8, then the message order is incorrect and needs to be adjusted. In this case, all unconsumed messages in each queue are sent to the first stack.

[0062] The first stack can be a data structure generated in real time when messages are not consumed in a preset order.

[0063] Based on the sequential numbering of unconsumed messages, messages are placed into the first stack in reverse order. Due to the last-in, first-out (LIFO) characteristic of the stack structure, even if the push order is reversed, the pop order can be forward. Therefore, the sequential nature of unconsumed messages can be guaranteed.

[0064] Step 130: Send the target message to the consumer for consumption.

[0065] Understandably, the first stack is essentially a stack structure independent of the queue. It can take data or messages that are in the queue and put them into the stack to adjust their order, and then give them directly to the consumer. It's similar to inserting messages that are out of queue into the consumer's queue. Inserting messages into the queue is the function, and the stack is the structure.

[0066] Because of the last-in-first-out (LIFO) characteristic of a stack, when data stored in reverse order is pushed, it can be guaranteed that the data will arrive at the consumer in sequential order, so that the consumer can consume the corresponding messages in order.

[0067] The embodiments of this application can be applied to the following scenarios:

[0068] In mobile billing systems, incremental service data needs to be synchronized to the in-memory database through a data sharing center. Because mobile billing systems have high requirements for the accuracy and timeliness of service data, incremental service data needs to be sent to different queues in the data sharing center before being delivered to the consumer. The specific implementation method is as follows:

[0069] 1. Based on the BKDR hash algorithm, the data to be processed can be evenly distributed and sent to all queues.

[0070] 2. When assembling the data corresponding to the data to be processed into a message, a message header is added. The message header includes the primary key of the current data, the number of data in the data set contained in the primary key, and the sequence number of the current data in the data set. The primary key is the data identifier.

[0071] 3. Add a private cache to the sharing center, such as Figure 2 As shown, it is divided into the following three parts:

[0072] Part 1: When a message arrives at the consumer, record the message's primary key, data size, and the sequence number of the consumed message. When the last message corresponding to that primary key arrives at the consumer, clear that cache. Part 1 is the consumed cache, used to record the message consumption status.

[0073] Part Two: When a message enters the queue, its primary key, data size, current message sequence number, and queue number are recorded. When the message reaches the consumer or is moved into the queue stack, its cache is cleared. This second part of the cache is a queue cache, used to record which message is in which queue, similar to an index. Messages can be retrieved directly from their corresponding queues for reordering. This record is maintained for each message entering the queue and is deleted from the cache after being consumed.

[0074] Part Three: Taking the first stack as an example of an interleaving stack, this embodiment will be explained.

[0075] The first part of the cache records the index of all existing queue-jumping stacks and their corresponding primary keys. The third part is the stack cache. Because a batch of data may contain many primary keys, a new queue-jumping stack is generated to adjust the order when messages corresponding to each primary key are out of order. This part of the cache is the index of the primary key and the queue-jumping stack address, used to retrieve the queue-jumping stack address corresponding to the primary key, which can quickly determine the queue-jumping stack.

[0076] By introducing a queue stack, out-of-order data can be retrieved from the queue separately, and then pushed onto the stack in reverse order and popped out in the correct order, thus ensuring the order of the data.

[0077] 3. When a message arrives at the consumer via the queue, the end-of-line interceptor first checks the consumption cache to determine if the message is the first record or if the current sequence number matches the consumption order recorded in the cache. If so, the message's consumed sequence number recorded in the cache is updated. If the message is determined to be the last record, the corresponding cached data for that message is cleared.

[0078] 4. If the message interception at the end of the queue reveals a mismatch between the message's consumption order and the order recorded in the cache, it indicates an incorrect push order. First, create an insertion stack, push the message into the insertion stack, search the queue cache, and retrieve the unconsumed message from the corresponding queue and push it into the insertion stack.

[0079] 5. Repeat the above operation until the processed data sequence number matches the consumption order of that primary key in the consumption cache. This indicates that all data for that primary key has been processed. Then, push all data in the queue stack to the consumer in sequence. Due to the last-in-first-out (LIFO) property of the stack, pushing data stored in reverse order ensures that the data arriving at the consumer is in sequential order.

[0080] The above data process can realize asynchronous processing of queue transmission and queue insertion stack. That is, the queue transmission and queue insertion stack order adjustment operations are independent of each other, and the data push is not affected, which can guarantee the processing performance of the shared center.

[0081] like Figure 2 As shown, KEY001 is the primary key, and DATA01 contains the data and its number. Figure 2 It can be seen that the primary keys KEY001, KEY003 and KEY004 correspond to data in a disordered order. Therefore, the data corresponding to the primary keys are pushed into the corresponding queue stacks to adjust their order.

[0082] Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application. (Refer to...) Figure 3 This application provides a data processing method, which may include:

[0083] Step 310: Send the message to each queue.

[0084] Step 311: Record the primary key, message sequence number, and queue number.

[0085] Step 312: The message arrives at the end of the queue, proceed to step 313.

[0086] Step 313: Determine if there is no consumption cache or if the order is normal. If yes, proceed to step 314; otherwise, proceed to step 315.

[0087] Step 314: Consume messages normally and cache the primary key and the sequence number of the consumed messages.

[0088] Step 315: Create the queue stack and identify unconsumed data.

[0089] Step 316: Retrieve unconsumed data from the queue using the queue index. Determine if unconsumed data exists. If it does, proceed to step 319; otherwise, proceed to step 317.

[0090] Step 317: Find the queue stack with the same primary key through the queue stack index, and proceed to step 319.

[0091] Step 318: Merge the queue stacks.

[0092] Step 319: Push the data onto the queue stack.

[0093] Step 320: Determine whether the consumption order of the data in the stack is consistent with that of the consumption cache.

[0094] Step 321: Push the data in the queue stack to the consumer end in sequence.

[0095] Step 322: Clear all data corresponding to this primary key from the cache.

[0096] The data processing method provided in this application embodiment can ensure queue load balancing by sending the data to be processed to each queue; and when it is determined that the messages are not consumed in a preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message, and the target message is sent to the consumer for consumption, which can ensure the order of data processing while ensuring queue load balancing.

[0097] In one embodiment, the unconsumed message is the message corresponding to the preceding sequence number, where the preceding sequence number is the number before the sequence number corresponding to the message.

[0098] In practice, unconsumed messages can be filtered based on the total amount of data.

[0099] When the total amount of data is too large, the message corresponding to the preceding sequence number can be regarded as an unconsumed message, that is, the message corresponding to the number before the sequence number.

[0100] When the total amount of data is moderate, the remaining unconsumed messages can be considered as unconsumed messages. This application does not impose specific limitations on this.

[0101] For example, the data identifier corresponds to messages with sequence numbers 1 to 10. The message currently arriving at the end of the queue is the message with sequence number 8, while according to the preset order, the message that should arrive at the end of the queue is the message with sequence number 6. Therefore, the unconsumed messages can be those that were not consumed before sequence number 8, namely the messages with sequence numbers 7 and 6. In this case, the messages with sequence numbers 6 and 7 are directly sent to the first stack for order adjustment, and the messages in the first stack are directly pushed to the consumer, ensuring that the messages with sequence numbers 6 and 7 are consumed in the correct order, guaranteeing the sequential processing of messages corresponding to the same data identifier.

[0102] Unconsumed messages can also be messages corresponding to sequence number 6, sequence number 7, sequence number 8, sequence number 9, and sequence number 10. In this case, the remaining unconsumed messages can be directly sent to the first stack for order adjustment, and then the adjusted target message can be sent to the consumer for consumption, thereby ensuring the sequential processing of messages corresponding to the same data representation.

[0103] Understandably, if the unconsumed message is the message corresponding to the previous sequence number, the amount of data sent to the first stack can be reduced.

[0104] The data processing method provided in this application can improve the efficiency of sequence adjustment and reduce the pressure on the first stack for data processing by filtering messages corresponding to the sequence numbers before the sequence number as unconsumed messages.

[0105] In one embodiment, the fact that the messages are not consumed in a preset order is determined in the following way:

[0106] The sequence number corresponding to the message is detected at the end of each queue;

[0107] If the sequence number corresponding to the message does not match the target sequence number, it is determined that the message was not consumed in the preset order.

[0108] Understandably, the sequence number corresponding to a message can be pre-set in the message header to indicate the message's position and processing order in the dataset.

[0109] The target sequence number is the sequential number of the messages that arrive at the end of the queue if they are in the correct order. It can also be understood as the sequential number of the messages that the consumer can currently successfully consume.

[0110] In actual execution, the target sequence number or the preset order of message push can be pre-stored in the corresponding cache. When checking the message consumption order, the target sequence number or preset order stored in the cache can be used to make a judgment, thereby quickly determining whether the messages are consumed in the preset order.

[0111] Since the head of the queue is used for data input and the tail for data output, setting an interception check at the tail of the queue can determine the sequence number of the messages currently reaching the tail. This sequence number is then matched against the target sequence number; if the sequence number and target sequence number do not match, it is determined that the messages were not consumed in the preset order.

[0112] Alternatively, it can detect whether the message corresponding to the previous sequence number of the current message has been consumed. If it has not been consumed, it can also be determined that the message was not consumed in the preset order.

[0113] Alternatively, the message sequence number can be directly matched with the preset order. If no match is found, it can be determined that the message was not consumed in the preset order.

[0114] The data processing method provided in this application embodiment can quickly identify messages with disordered order by matching the sequence number corresponding to the message with the target sequence number.

[0115] In one embodiment, before sending unconsumed messages from each queue to the first stack for reordering, the method further includes:

[0116] If no unconsumed message is detected, the second stack corresponding to the unconsumed message is determined based on the data identifier;

[0117] Based on the sequence number corresponding to the message, the unconsumed message in the second stack is sent to the first stack.

[0118] If no unconsumed messages are detected, it means that the unconsumed messages have been pushed onto another stack, and the second stack can be retrieved based on the data identifier.

[0119] Data can be directly added from the second stack to the first stack, or the first and second stacks can be merged into one stack. Based on the sequential number of the message, the data in the merged stack is stored in reverse order.

[0120] Data identifiers can be associated with stack addresses, and a corresponding index can be created. This index is used to retrieve the stack address corresponding to the data identifier. This index can also be pre-stored in the corresponding cache for quick determination of the stack address corresponding to an unconsumed message.

[0121] The data processing method provided in this application embodiment can determine the second stack corresponding to the unconsumed message through data identification, and send the unconsumed message in the second stack to the first stack to ensure that the out-of-order messages are output in the same stack and avoid causing the out-of-order message again.

[0122] In one embodiment, the method further includes:

[0123] The sequence number corresponding to the message is detected at the end of each queue;

[0124] If the sequence number corresponding to the message matches the target sequence number, the message is consumed and the target sequence number is updated.

[0125] Setting up an interception check at the end of the queue allows us to determine the sequential number of the messages currently reaching the end of the queue. This sequential number is then matched against the target sequence number. If the sequential number matches the target sequence number, it can be determined that the messages are being consumed in the preset order.

[0126] Therefore, the target sequence number needs to be updated before the next message arrives at the end of the queue.

[0127] For example, if the sequence number is 5 and the target sequence number is 5, then after consuming the message corresponding to sequence number 5, the target sequence number needs to be updated to 6.

[0128] The data processing method provided in this application can quickly determine whether messages are consumed in the correct order by matching the sequence number of the message with the target sequence number.

[0129] In one embodiment, before sending unconsumed messages from each queue to the first stack for reordering, the method further includes:

[0130] Based on the sequence number and queue index corresponding to the message, the unconsumed message is determined, and the queue index is used to indicate the queue where the unconsumed message is located.

[0131] In actual execution, when a message enters the queue, the message's data number, data volume, current message sequence number, and queue index are recorded.

[0132] The queue index is the number of the queue in which the message is located, used to indicate which specific message is in which queue.

[0133] Based on the queue index and the current message's sequence number, the queue corresponding to the current message can be determined, meaning the message can be directly retrieved from the corresponding queue for subsequent order adjustment.

[0134] The data processing method provided in this application embodiment determines unconsumed messages by using the sequence number and queue index corresponding to the message, thereby enabling rapid location of unconsumed messages.

[0135] The data processing apparatus provided in the embodiments of this application is described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence.

[0136] Figure 4 This is a schematic diagram illustrating the structure of the data processing method provided in an embodiment of this application. (Refer to...) Figure 4 This application provides a data processing structure, which may include: a sending module 410, an adjustment module 420, and a consumption module 430.

[0137] The sending module 410 is used to send the data to be processed to each queue, wherein the data to be processed includes multiple messages with the same data identifier;

[0138] The adjustment module 420 is used to send unconsumed messages in each queue to the first stack for order adjustment when it is determined that the message has not been consumed in a preset order, so as to obtain the adjusted target message.

[0139] The consumption module 430 is used to send the target message to the consumer for consumption.

[0140] The data processing apparatus provided in this application embodiment can ensure queue load balancing by sending the data to be processed to each queue; and when it is determined that the messages are not consumed in a preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message, and the target message is sent to the consumer for consumption, which can ensure the order of data processing while ensuring queue load balancing.

[0141] In one embodiment, the fact that the messages are not consumed in a preset order is determined in the following way:

[0142] The first detection module is used to detect the sequence number corresponding to the message at the tail of each queue;

[0143] The first determining module is used to determine that the message was not consumed in a preset order if the sequence number corresponding to the message does not match the target sequence number.

[0144] In one embodiment, the data processing apparatus further includes:

[0145] The second determining module is used to determine the second stack corresponding to the unconsumed message based on the data identifier if the unconsumed message is not detected.

[0146] The merging module is used to send the unconsumed messages in the second stack to the first stack based on the sequence number corresponding to the messages.

[0147] In one embodiment, the data processing apparatus further includes:

[0148] The second detection module is used to detect the sequence number corresponding to the message at the tail of each queue;

[0149] An update module is used to consume the message and update the target sequence number when the sequence number corresponding to the message matches the target sequence number.

[0150] In one embodiment, the data processing apparatus further includes:

[0151] The third determining module is used to determine the unconsumed message based on the sequence number and queue index corresponding to the message, wherein the queue index is used to indicate the queue where the unconsumed message is located.

[0152] In one embodiment, the sending module 410 is further configured to:

[0153] Based on the BKDR hash algorithm, the data to be processed is sent to each queue respectively.

[0154] In one embodiment, the unconsumed message is the message corresponding to the preceding sequence number, where the preceding sequence number is the number before the sequence number corresponding to the message.

[0155] Figure 5 An example is a schematic diagram of the structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call a computer program stored in the memory 530 to execute the steps of a data processing method, such as including:

[0156] The data to be processed is sent to each queue, and the data to be processed includes multiple messages with the same data identifier;

[0157] If it is determined that the message is not consumed in the preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message;

[0158] The target message is sent to the consumer for consumption.

[0159] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the data processing methods provided in the above embodiments, such as including:

[0161] The data to be processed is sent to each queue, and the data to be processed includes multiple messages with the same data identifier;

[0162] If it is determined that the message is not consumed in the preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message;

[0163] The target message is sent to the consumer for consumption.

[0164] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:

[0165] The data to be processed is sent to each queue, and the data to be processed includes multiple messages with the same data identifier;

[0166] If it is determined that the message is not consumed in the preset order, the unconsumed messages in each queue are sent to the first stack for order adjustment to obtain the adjusted target message;

[0167] The target message is sent to the consumer for consumption.

[0168] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: The data to be processed is sent to each queue, and the data to be processed includes multiple messages with the same data identifier; If it is determined that the messages are not consumed in the preset order, a first stack is created and the unconsumed messages in each queue are sent to the first stack in reverse order according to the order number of each unconsumed message to adjust the order and obtain the adjusted target message; The target message is sent to the consumer for consumption.

2. The data processing method according to claim 1, characterized in that, The following method is used to determine if the message is not consumed in the preset order: The sequence number corresponding to the message is detected at the end of each queue; If the sequence number corresponding to the message does not match the target sequence number, it is determined that the message was not consumed in the preset order.

3. The data processing method according to claim 1, characterized in that, Before sending unconsumed messages from each queue to the first stack for reordering, the process also includes: If no unconsumed message is detected, the second stack corresponding to the unconsumed message is determined based on the data identifier; Based on the sequence number corresponding to the message, the unconsumed message in the second stack is sent to the first stack.

4. The data processing method according to any one of claims 1-3, characterized in that, The method further includes: The sequence number corresponding to the message is detected at the end of each queue; If the sequence number corresponding to the message matches the target sequence number, the message is consumed and the target sequence number is updated.

5. The data processing method according to any one of claims 1-3, characterized in that, Before sending unconsumed messages from each queue to the first stack for reordering, the process also includes: Based on the sequence number and queue index corresponding to the message, the unconsumed message is determined, and the queue index is used to indicate the queue where the unconsumed message is located.

6. The data processing method according to any one of claims 1-3, characterized in that, The step of sending the data to be processed to each queue includes: Based on the BKDR hash algorithm, the data to be processed is sent to each queue respectively.

7. The data processing method according to claim 5, characterized in that, The unconsumed message is the message corresponding to the preceding sequence number, which is the number preceding the sequence number corresponding to the message.

8. A data processing apparatus, characterized in that, include: The sending module is used to send the data to be processed to various queues, wherein the data to be processed includes multiple messages with the same data identifier; The adjustment module is used to create a first stack and send the unconsumed messages in each queue to the first stack in reverse order according to the order number of each unconsumed message when it is determined that the message has not been consumed in a preset order, so as to obtain the adjusted target message. The consumption module is used to send the target message to the consumer for consumption.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Consumption equilibrium method and system based on message queues

    CN105306552A

  • Message transmission method and device, equipment and storage medium

    CN112965839A