PROCESSING A MESSAGE STREAM

Abstract

A computer-implemented method for processing a message data stream, the method comprising: Obtaining (410) messages of a message stream, each message (PA-1) having an associated timestamp; Storing (420) each message (PA-1) in at least one of a plurality of queues (322), wherein the messages of each queue (322) are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first position in the queue (322); controlling (430) retrieval of a target message from the first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-in-line message (PA-1) of each of the other queues (322) of the plurality of queues (322) that do not include the target queue; Receiving an input signal representative of a timing request; and Determining the threshold based on the received input signal, wherein the threshold is equal to an average of the first message and a second message in each queue (322).

Description

BACKGROUND

[0001] The present disclosure relates generally to processing data and, more particularly, to processing a temporal message stream.

[0002] The present disclosure also relates to a system for processing a message data stream.

[0003] The present disclosure further relates to a computer program product comprising computer-readable program code that enables a processor of a system to implement such a method.

[0004] A conventional data processing system for processing temporal data (or data that does not require strict sequencing but does require processing the data at a similar time) may typically include a processing engine designed to process data streams. Such a processing engine can process fast data streams in real time, and—to accommodate data streams of varying sizes—the processing engine can be scaled based on the size of the data streams. However, this scaling may result in output data being out of sequence, thus rendering the resulting output data unusable.

[0005] For example, applications in a system can send metrics (e.g., response times, transaction rates, and central processing unit (CPU) utilization) to a metrics processing engine via a message hub. The metrics processing engine can then send the metrics to an open-source metrics analysis tool. Scaling the metrics processing engine involves splitting a topic corresponding to the metrics into partitions (or queues) so that the metrics can be shared between instances of the metrics processing engine. However, if a problem occurs in one of the metrics processing engine instances, the instances may become out of synchronization, resulting in unusable output in a user interface of the metrics analysis tool (e.g.,the transaction rate data may show a peak in utilization that is inconsistent with other data, preventing accurate data correlation and problem diagnosis).

[0006] As another example, a company that sells or distributes a large quantity of items (e.g., a company that sells tickets) can handle a large throughput of orders by decoupling actions corresponding to ordering an item and processing the order. Ordering an item may result in a message being placed in a topic specific to the item, with order messages (i.e., messages corresponding to created orders) being arranged across a number of partitions (hereafter referred to as queues) corresponding to the topic. A plurality of processing engines may be designed to process the messages in each queue. However, there is a risk that message processing may be unevenly distributed among the plurality of processing engines.This, in turn, increases the risk of data being received out of sequence, for example, if a person receives the last item in a large batch even if their corresponding order was placed significantly later than an order from another person who also has yet to receive their ordered item. Furthermore, it can increase the risk that a person's order will not be processed due to a "slow" processing engine.

[0007] In this context, the published document US 2006 / 0 039 393 A1 already exists. Although the subject matter described in this document, through its presented components, has already been able to address some of the above-mentioned problem situations, there is still a need to solve the above-mentioned problems more fundamentally. SUMMARY

[0008] This problem is solved by the subject matter of the independent patent claims. Further details are set out in the subordinate and dependent patent claims.

[0009] The present disclosure also aims to provide a system for processing a message data stream.

[0010] The present disclosure aims to provide a computer-implemented method for processing a message data stream.

[0011] The present disclosure further aims to provide a computer program product, such as computer program code, for implementing a proposed method when executed by a controller unit.

[0012] According to one aspect of the present disclosure, a computer-implemented method for processing a message stream is provided. The method comprises obtaining messages of a message stream, each message having an associated timestamp. The method further comprises storing each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed first in the queue. The method further comprises controlling retrieval of a target message from the first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of the first-ordered message of each of the other queues of the plurality of queues.

[0013] According to a further aspect of the disclosure, a computer program product for processing a message data stream is provided. The computer program product comprises a computer-readable storage medium having program instructions contained thereon, wherein the program instructions are executable by a processing unit to cause the processing unit to perform a method according to a proposed embodiment.

[0014] According to another aspect of the disclosure, a processing system is provided, comprising at least one processor and the computer program product according to one embodiment. The at least one processor is configured to execute the computer program code of the computer program product.

[0015] According to another aspect of the disclosure, a system for processing a message stream is provided. The system includes an interface configured to receive messages of a message stream, each message having an associated timestamp. The system further includes a queue component configured to store each message in at least one of a plurality of queues. The messages of each queue are sorted based on their timestamp, such that a message with the earliest timestamp is placed first in the queue.The system further comprises a message retrieval component configured to control retrieval of a target message from the first location in a target queue of the plurality of queues based on a time difference between the timestamp of the target message and the timestamps of the first-in-line message of each of the other queues of the plurality of queues. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Preferred embodiments of the present disclosure will now be described, by way of example only, with reference to the following drawings, in which: Fig. 1 depicts a pictorial representation of an example distributed system in which aspects of the illustrative embodiments may be implemented; Fig. 2 is a block diagram of an example system in which aspects of the illustrative embodiments may be implemented; Fig. 3 is a simplified block diagram of an example embodiment of a system for concurrently processing a temporal message stream; Fig. 4 is a flowchart of a computer-implemented method for simultaneously processing a temporal message data stream; Fig. 5 is a graphical representation of the steps involved in a method of an exemplary embodiment, wherein the vertical axis describes each of a plurality of queues and the horizontal axis describes a time of a timestamp corresponding to a message; and Fig. 6 is a simplified block diagram of an exemplary embodiment of a system for concurrently processing a temporal message stream. DETAILED DESCRIPTION

[0017] The figures are merely schematic and not drawn to scale. It should also be clear that the same reference numerals are used throughout to designate identical or similar parts.

[0018] When embodiments of the present disclosure represent a method, such a method may, in the context of the present application, be a process for execution by a computer, ie, it may be a computer-implementable method. The various steps of the method may thus reflect different parts of a computer program, e.g., different parts of one or more algorithms.

[0019] In the context of the present application, a system may also be a single unit or a collection of distributed units configured to perform one or more embodiments of the methods of the present disclosure. For example, a system may be a personal computer (PC), a server, or a collection of PCs and / or servers connected via a network, such as a local area network, the Internet, etc., to collectively perform at least one embodiment of the methods of the present disclosure.

[0020] Concepts for processing a message stream are proposed. Such concepts can enable concurrent processing of temporal data by scaling a processing engine while maintaining approximate timing of processing. Accordingly, embodiments can enable concurrent processing of a temporal message stream.

[0021] Embodiments can be realized in conjunction with a messaging or event system, e.g. with implementations of Kafka ® , Message Queue (MQ) or Java Message Service (JMS).

[0022] Proposed embodiments may use a concept of defining a group of queues (or partitions, or data streams) within which messages can be stored and time-coordinated (e.g., queued). Messages can be retrieved from the queues in order, and such retrieval can be controlled based on timestamps associated with the messages. For example, the timestamps can be analyzed and used to ensure that messages retrieved from the queues remain relatively synchronized.

[0023] Embodiments of the present disclosure propose providing a message retrieval component. Such a message retrieval component may be configured to control retrieval of a message from the first position in a queue of the plurality of queues based on a time difference between the timestamp of the message and the timestamps of the first-in-line message of each of the other queues of the plurality of queues. For example, by only allowing retrieval of a message if it is within a predetermined allowable time difference of the earliest timestamp across all queues, temporal coordination of retrieved messages may be maintained.

[0024] Embodiments may thus provide for the enhancement of concurrent processing of a temporal message stream by maintaining approximate temporal coordination of concurrently processed data.

[0025] Embodiments may be implemented, for example, in connection with messaging or streaming services. Such embodiments may enable the timing of message streams without requiring strict ordering and / or single-threaded behavior, which prevents scaling of processing engines. Accordingly, message allocation is optimized, reducing the risk of inefficient messaging or streaming services. This is merely one example of advantages over the prior art.

[0026] For the purposes of this disclosure, a target message is one of the messages of a message stream. A message becomes the target message when a client (or other component) attempts to remove the target message from a queue (e.g., the subject of a GET command, where a GET command is code that removes the message from the queue for subsequent processing). Generally, the target message is the first message (e.g., the next message, the message with the earliest timestamp, etc.) in the queue. The queue containing the target message may be referred to as the target queue.

[0027] For the purposes of this disclosure, a topic may be a group, a data stream, and / or a partition. A topic may be any manner of sorting and / or organizing the plurality of messages in a message stream to sort them into different queues. The topics may be (or relate to) a message size, a message type (e.g., a format), a data source, a random, circular, sequential, and / or any other sorting method. For example, the system may relate to selling tickets for an event. At a certain time, the tickets are released, and several purchase messages are generated. The topics for the tickets may be the number of tickets purchased, the total amount purchased, based on the individual ticket price, based on the location of the original purchase, based on the area (e.g.,reserved seating versus open seating) of the requested tickets and / or similar factors.

[0028] In one proposed embodiment, controlling retrieval may comprise determining a time difference between a target message and the first-in-line message from each of the other queues. The time difference is the difference between timestamps of two messages. Controlling retrieval may then comprise identifying the largest time difference value from the determined time differences and then allowing retrieval of the target message from the target queue based on whether the largest time difference value exceeds a threshold. In this way, message retrieval may be controlled so that messages are retrieved from queues evenly across all queues. This may optimize the timing of the data corresponding to the messages so that temporally similar data can be processed at a similar time.In this way, the risk of disrupting the order of temporal data due to scaling can be reduced. This provides the advantage of efficient scaling of processes.

[0029] In some embodiments, controlling retrieval may further comprise receiving an input signal representative of a timing requirement and determining the threshold based on the received input signal. Thus, the threshold used to establish retrieval of a message may be dependent on a timing requirement. As a result, establishing retrieval of a message may be based on the requirements for the method and / or system and may be automatically adjusted with respect to the requirements. In this way, the efficiency of the method and / or system for simultaneously processing a temporal message stream is optimized.

[0030] In a proposed embodiment, retrieval of the target message may be permitted if the largest value of a time difference does not exceed the threshold. Furthermore, retrieval of the target message may be denied if the largest value of a time difference exceeds the threshold. Thus, message retrieval may be maintained evenly across each of the plurality of queues. This may reduce the risk of message retrieval bias within the plurality of queues and thereby reduce the risk of a queue retrieving a significantly larger / smaller number of messages compared to an average amount across the plurality of queues.

[0031] In some embodiments, an error message may be generated if the retrieval of the target message from the target queue is rejected. In this way, a user may be alerted to the rejection of a message retrieval. This allows the queue corresponding to the rejected retrieval to be identified, allowing the retrieval rate for each of the plurality of queues to be determined and analyzed. Thus, if a comparatively large number of retrievals in a particular queue are rejected, the number of which can be determined, the queue can be identified, and remedial action can be taken.

[0032] The method for concurrently processing a temporal message stream may further comprise generating a timestamp for each received message, wherein the timestamp represents a time associated with the message. Thus, each message has a corresponding time identified by the timestamp. For example, the timestamp associated with a message may identify the time at which the message was received (e.g., received by the interface). This allows the messages to be organized in a time-dependent manner, thus optimizing scheduling.

[0033] An example of a timestamp might include the time the message is queued using a PUT command, where PUT is a command in a programming language. This can be generated by the client publishing the message or by the messaging system. This can occur at the time the message is queued or at the time the message data is generated. The time can be a timespan or date string, which can be located in the message body (or in the message header / metadata).

[0034] In some embodiments, each message may relate to at least one of a plurality of topics. In some embodiments, the method for concurrently processing a temporal message stream may further comprise, for each of the plurality of topics, defining one or more queues associated with each topic. The method may further comprise storing each message in at least one of the plurality of queues that relate to at least one topic in the queue associated with the same topic(s). Thus, the method may be extended to group more than one topic—and consequently, the corresponding plurality of queues.This allows a plurality of topics related to the messages to be processed in parallel, thereby improving the efficiency of the method and / or system and increasing the complexity of the method and / or system.

[0035] Fig. 1 depicts a pictorial representation of an exemplary distributed system in which aspects of the illustrative embodiments may be implemented. Distributed system 100 may include a network of computers in which aspects of the illustrative embodiments may be implemented. Distributed system 100 includes at least one network 102, which is the medium used to provide data communications links between various devices and computers interconnected within distributed computing system 100. Network 102 may include connections such as wired and wireless data communications links or fiber optic cables.

[0036] In the illustrated example, a first server 104 and a second server 106, along with a storage device 108, are connected to the network 102. Additionally, clients 110, 112, and 114 are also connected to the network 102. The clients 110, 112, and 114 may be, for example, personal computers, network computers, or the like. In the illustrated example, the first server 104 provides data such as boot files, operating system images, and applications to the clients 110, 112, and 114. In the illustrated example, the clients 110, 112, and 114 are clients of the first server 104. The distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0037] In the illustrated example, the distributed system 100 is the Internet, where the network 102 represents a worldwide collection of networks and gateways that use the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol suite to exchange data with each other. The core of the Internet is formed by a structure of high-speed data transmission links between key nodes or host computers, consisting of thousands of commercial, government, educational, and other computer systems that relay data and messages. The distributed system 100 may also be implemented as a number of different types of networks, for example, as an intranet, a local area network (LAN), a wide area network (WAN), or the like. As mentioned above, Fig. 1 is intended as an example, not as an architectural limitation of various embodiments of the present disclosure, which is why the Fig. 1 are not intended to limit the environments in which the illustrative embodiments of the present disclosure may be practiced.

[0038] Fig. 2 is a block diagram of an example system 200 in which aspects of the illustrative embodiments may be implemented. The system 200 is an example of a computer such as the client 110 of Fig. 1, in which computer-usable code or computer-usable instructions may be located that implement the processes for illustrative embodiments of the present disclosure.

[0039] In the illustrated example, system 200 uses a hub architecture that includes a North Bridge & Memory Controller Hub (NB / MCH) 202 and a South Bridge & Input / Output (I / O) Controller Hub (SB / ICH) 204. A processing unit 206, a main memory 208, and a graphics processor 210 are connected to the NB / MCH 202. The graphics processor 210 can be connected to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0040] In the illustrated example, a LAN adapter 212 is connected to the SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, USB (Universal Serial Bus) and other data transmission ports 232, and PCI / PCIe devices 234 are connected to the SB / ICH 204 via a first bus 238 and a second bus 240. The PCI / PCIe devices can include, for example, Ethernet adapters, expansion cards, and PC cards for notebook computers. PCI uses a CardBus controller, whereas PCIe does not. The ROM 224 can be, for example, a Flash BIOS (Binary Input / Output System).

[0041] The HDD 226 and the CD-ROM drive 230 are connected to the SB / ICH 204 via the second bus 240. The HDD 226 and the CD-ROM drive 230 can use, for example, an IDE (Integrated Drive Electronics) or a SATA (Serial Advanced Technology Attachment) interface. A SIO (Super I / O) unit 236 can be connected to the SB / ICH 204.

[0042] An operating system is executed in the processing unit 206. The operating system coordinates various components within the system 200 from Fig. 2 and provides control thereof. As a client, the operating system can be a commercially available operating system. An object-oriented programming system, such as the Java™ programming system, can run alongside the operating system and provide calls to Java™ programs or applications running on the system 200.

[0043] As a server, the System 200 can, for example, be an IBM® eServer™ System p ® -Computer system that runs the Advanced Interactive Executive (AIX) operating system ® ) or the LINUX ® operating system. System 200 may be a symmetric multiprocessor (SMP) system that includes a plurality of processors in processing unit 206. Alternatively, a single-processor system may be used.

[0044] Instructions for the operating system, the programming system, and applications or programs are located on storage devices such as HDD 226 and can be loaded into main memory 208 for execution by processing unit 206. Accordingly, one or more message processing programs may be configured to be stored by the storage devices and / or main memory 208, according to one embodiment.

[0045] The processes for illustrative embodiments of the present disclosure may be performed by processing unit 206 using computer-usable program code that may be located in memory such as main memory 208, ROM 224, or one or more peripheral units 226 and 230.

[0046] A bus system such as the first bus 238 or the second bus 240 from Fig. 2 may include one or more buses. The bus system may be implemented using any type of data transmission structure or architecture that provides for the transmission of data between various components or units connected to the structure or architecture. A data transmission unit, such as the modem 222 or the network adapter 212 of Fig. 2 may contain one or more units used to transmit and receive data. A working memory may be, for example, the main memory 208, the ROM 224, or a cache, as in the NB / MCH 202 of Fig. 2 is present.

[0047] The expert knows that the hardware in the Fig. 1 and Fig. 2 may vary depending on the implementation and may be modified without departing from the scope of the claims. In addition to or instead of the Fig. 1 and Fig. 2, other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, and the like, may also be used. Furthermore, the processes of the illustrative embodiments may also be applied to a multiprocessor data processing system other than the system mentioned above without departing from the spirit and scope of the present disclosure.

[0048] Furthermore, system 200 may take the form of a variety of computing systems, including, but not limited to, client computing devices, server computing devices, a tablet computer, a laptop computer, a telephone or other data transmission device, a personal digital assistant (PDA), or the like. In some illustrative examples, system 200 may be a portable computing device configured with flash memory to provide non-volatile memory, for example, for storing operating system files and / or user-generated data. Thus, system 200 may be essentially any computing system known or later developed without architectural limitation.

[0049] With reference to Fig. 3 illustrates a simplified block diagram of an exemplary embodiment of a system for concurrently processing a temporal message stream.

[0050] The system includes an interface 310 configured to receive messages of a message stream, each message having an associated timestamp. The system further includes a queue component 320 configured to store each message in each of a plurality of queues 322, wherein the messages of each queue 322 are sorted based on their timestamp such that a message with the earliest timestamp is positioned first in queue 322 (i.e., a first-in, first-out (FIFO) queue), a message PB-1 in a queue 322B.The system further includes a message retrieval component 330 configured to control retrieval of a target message from the first location in a target queue of the plurality of queues 322 based on a time difference between the timestamp of the target message and the timestamps of the first-in-order message of each of the other queues 322 of the plurality of queues 322. The message retrieval component 330 may communicate with a plurality of consumers 332, wherein each of the consumers 332 may retrieve messages from a corresponding queue 322, a message PA-1 in a consumer 332A. Although the consumers 332 may have a 1:1 mapping to a queue, a consumer 332 may retrieve messages from more than one queue 322, and a queue 322 may have messages retrieved by more than one of the consumers 332.

[0051] In some embodiments, the message retrieval component 330 is further configured to determine, for each of the other queues 322, a time difference between the timestamp of the target message and the first-order message of each queue 322. The message retrieval component is further configured to identify the largest time difference value from the determined time differences and to allow or deny retrieval of the target message from the target queue 322 based on whether the largest time difference value exceeds a threshold.

[0052] In one embodiment, message retrieval component 330 is further configured to allow retrieval of the target message from the target queue if the largest value of a time difference does not exceed the threshold, and to deny retrieval of the target message from the target queue if the largest value of a time difference exceeds the threshold. In some embodiments, the message with the earliest timestamp (or the message associated with the largest difference) is retrieved, and the time differences are recalculated. There will then be a second largest difference or a remaining largest difference. These are calculated in the same way as the largest difference.

[0053] In one embodiment, the message retrieval component 330 is further configured to generate an error message when retrieval of the target message from the queue 322 is rejected.

[0054] In one embodiment, the system further comprises a coordination configuration component (not shown) configured to receive an input signal representative of a timing request and determine the threshold based on the received input signal. In various embodiments, the coordination configuration component is a component of, and / or is incorporated into, interface 310, queue component 320, message retrieval component 330, and / or is a separate component.

[0055] In some embodiments, the system further includes a time component configured to generate, for each component, a timestamp representative of a time associated with the message and to assign the generated timestamp to the message. For example, a timestamp for a message may represent the time at which the message is generated or queued with a PUT command. In various embodiments, the time component is a component of, and / or incorporated into, interface 310, queue component 320, message retrieval component 330, and / or is a separate component.

[0056] In some embodiments, each message relates to at least one of a plurality of topics. Queue component 320 is further configured for each of the plurality of topics to assign one or more of queues 322 to one or more topics and to store each message in at least one of the plurality of queues 322 associated with the topic(s) to which the message relates.

[0057] With reference to Fig. Figure 4 illustrates a flowchart of a computer-implemented method for processing a message stream according to one embodiment. Such a method may enable concurrent processing of a temporal message stream by scaling a processing engine while maintaining approximate timing of the processing.

[0058] A step 410 includes obtaining messages of a message stream, each message having an associated timestamp. In some embodiments, step 410 includes generating a timestamp for each message.

[0059] In one embodiment, each message relates to at least one of a plurality of topics. In some embodiments, the method further comprises, for each of the plurality of topics, assigning at least one topic to each queue. In some embodiments, the method further comprises dividing the topic into a plurality of partitions and assigning one or more consumers to each queue.

[0060] For example, in a messaging system, a user can create a topic that contains five queues (e.g., partitions, streams, and / or topics). Five consumers can be assigned to the topic as group consumers, so that each queue is provided with one consumer. Each message is distributed to one of the queues based on a certain sorting factor (e.g., size, content, origin, data type, circulation, randomness, etc.). Each consumer processes messages according to its respective queue once they are retrieved from the queue. This example is described in Fig. 3, wherein the five queues 322A, 322B, 322C, 322D, 322E and the five associated consumers 332A, 332B, 332C, 332D, 332E process the messages PA-1, PA-2, PA-3 for the queue 322A, PB-1 for the queue 322B and PC-1 for the queue 322C. Fig. Figure 5 is a graphical representation of the steps involved in this example, where the vertical axis describes each of the plurality of queues and the horizontal axis describes the time of the timestamp corresponding to a message. The example described below refers to the Fig. 3 and Fig. 5 information presented.

[0061] Step 420 includes storing each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp so that a message with the earliest timestamp is placed first in the queue. Here, each message is stored in at least one of the queues associated with the topic(s) to which the message relates.

[0062] Step 430 includes controlling retrieval of a target message from the first location in a target queue based on a time difference between the message's timestamp and the timestamps of the first-placed message of each of the other queues in the plurality of queues. In the example, messages to consumers may be retrieved uniformly from the plurality of queues and sequentially from each of the plurality of queues.

[0063] In one embodiment, controlling retrieval for each of the other queues may comprise determining a time difference between the timestamp of the target message and the first-order message of each of the other queues. Controlling retrieval may then comprise identifying the largest time difference from the determined time differences and then allowing retrieval of the target message from the target queue based on whether the largest time difference exceeds a threshold.

[0064] In the example, the user can define a threshold of 100 ms, which can also be referred to as a timing. However, the threshold can also be more or less than 100 ms. For example, a first queue 322A from the five queues can receive a message containing a comparatively large event, event PA-1, with a timestamp of 20:00:00,000 and a processing time of 200 ms. The message corresponding to event PA-1 is retrieved by consumer 332A, which corresponds to the first queue. The next message in the first queue 322A can contain another event, event PA-2, with a timestamp of 20:00:00,100. The consumer 332B, which corresponds to the next subsequent queue, i.e.a second queue 322B, may attempt to retrieve the next event from the first-place message of the second queue 322B, e.g., event PB-1 with a timestamp of 20:00:00,190. The timestamps of each event corresponding to messages in the first position in each queue may be determined.

[0065] In one embodiment, retrieval of the target message from the target queue is allowed if the largest value of a time difference does not exceed the threshold, and retrieval of the target message from the target queue is denied if the largest value of a time difference exceeds the threshold. In this way, when retrieving a message, the timestamps at the first position in each queue are compared. If the value of a time difference between an earliest timestamp across all queues exceeds the timestamp corresponding to a target message in a target queue, the message corresponding to the earliest timestamp is retrieved.

[0066] In the example, the time difference between the timestamp corresponding to event PB-1 (20:00:00,190) and the timestamp corresponding to event PA-1 (20:00:00,100) is 90 ms, which does not exceed the threshold of 100 ms. This is obtained by subtracting the timestamp of the front-of-line message from the timestamp of the target message (e.g., (target message timestamp) - (first message of another queue), 20:00:00,190 - 20:00:00,100). In this way, the message containing event PB-1 can be retrieved by consumer 332B corresponding to the target queue, in this case, second queue 322B.

[0067] In one embodiment, permitting retrieval of the target message further comprises generating an error message if retrieval of the target message from the target queue is rejected. The error message may include a return code or a message indicating that the output data is out of synchronization. If retrieval of the target message from the target queue is rejected, the process may be temporarily halted until the timestamps corresponding to the messages at the first position in each queue allow a corresponding time difference that does not exceed the threshold.

[0068] Controlling a retrieval may further comprise receiving an input signal representative of a timing request and then determining the threshold based on the received input signal.

[0069] In the example, a message from the next subsequent queue, a third queue 322C, may contain another event, event PC-1, with a timestamp of 20:00:00,250. The timestamps of each event corresponding to messages in the first position in each queue can be determined. At this time, the timestamp corresponding to event PA-2 is the earliest, and the time difference between the timestamp corresponding to event PA-2 (20:00:00,100) and the timestamp corresponding to event PC-1 (20:00:00,250) is 150 ms, which exceeds the threshold of 100 ms. Thus, the request to fetch from queue 322C is rejected, and event PC-1 is not fetched. On the other hand, if PA-2 is the target message, the difference of 150 ms is negative, which is less than the threshold of 100 ms, so PA-2 can be retrieved.

[0070] Processing of the message corresponding to event PA-1 is complete, and the message corresponding to the next event in the first queue 322A, event PA-2, is retrieved. A message from the first queue 322A may contain another event, such as event PA-3, with a timestamp of 20:00:00,180. The timestamps of each event corresponding to messages in the first position in each queue may be determined. The time difference between the timestamp corresponding to event PC-1 (20:00:00,250) and the timestamp corresponding to event PA-3 (20:00:00,180) is 70 ms, which does not exceed the threshold of 100 ms. This allows the message containing event PC-1 to be retrieved by the consumer 332C corresponding to the destination queue, in this case, the third queue 322C.

[0071] In some embodiments, the threshold is determined dynamically. It may be based on the number of messages in the queues, the relative processing time of the messages in the queues, the time differences between the timestamps of the messages, and / or any other similar factors. For example, the threshold may be the average of differences between the timestamp of the first and second messages in each queue. In another example, the threshold may be the message with the longest processing time (e.g., 00:00,100) in any queue plus a predetermined amount of time (e.g., x + 00:00,150). Thus, if the longest processing time for a message in the queues is 00:00,100, the threshold would be 00:00,250.

[0072] The above example can be extended to group more than one topic (and subsequently the corresponding queues). In this embodiment, each group can be given an identifier to correlate consumers between each topic.

[0073] As in Fig. 6, in another example, embodiments may include a computer system 70 that may be part of a networked system 7. Without being limited thereto, the components of the computer system / server 70 may include one or more processing devices, including, for example, processors or processing units 71, a system memory 74, and a bus 90 connecting various system components, such as the system memory 74, to the processing unit 71.

[0074] Processing unit 71 is configured to define a target object type; the target object type is configured to contain results of a middleware function. Furthermore, processing unit 71 is configured to receive a first object for an application and then perform a process of the middleware function using the first object to generate a process result. Processing unit 71 is configured to generate a second object of the target object type based on the process result and to provide the second object to the application.

[0075] Bus 90 represents at least one of any one of several types of bus structures, such as a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port (AGP), and a processor or local bus, and any of a variety of bus architectures may be used. By way of example and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0076] The computer system / server 70 typically includes a variety of media readable by a computer system. Such media may be any available media accessible to the computer system / server 70 and may include both volatile and non-volatile, removable and non-removable media.

[0077] The system memory 74 may include a computer system-readable medium in the form of volatile memory, such as random access memory (RAM) 75 and / or cache memory 76. The computer system / server 70 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the system memory 74 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and commonly referred to as a "hard disk"). Although not depicted here, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk, e.g.,a CD-ROM, a DVD-ROM, or other optical medium. In these cases, each drive may be connected to bus 90 via one or more data media interfaces. As illustrated and described in more detail below, memory 74 may include at least one program product having a set of (e.g., at least one) program modules configured to perform the functions of embodiments of the disclosure.

[0078] For example, and without limitation, a program / utility 78 having a set of (at least one) program modules 79 may be stored in memory 74, as may an operating system, one or more application programs, other program modules, and program data. The operating system, the one or more application programs, the other program modules, and the program data, or a combination thereof, may each include an implementation of a network environment. The program modules 79 generally perform the functions and / or procedures of embodiments of the disclosure described herein.

[0079] The computer system / server 70 may also exchange data with one or more external devices 80, e.g., a keyboard, a pointing device, a display 85, etc.; with one or more devices that allow a user to interact with the computer system / server 70; and / or with any devices (e.g., network card, modem, etc.) that allow the computer system / server 70 to exchange data with one or more other computing devices. Such data transfer may occur via input / output (I / O) interfaces 72. Furthermore, the computer system / server 70 may exchange data with one or more networks, e.g., a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 73. As shown, the network adapter 73 exchanges data with the other components of the computer system / server 70 via the bus 90.It should be understood that, although not depicted here, other hardware and / or software components could be used in conjunction with the computer system / server 70. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive stacks, RAID systems, tape drives, and data archival storage systems, etc.

[0080] When embodiments of the present disclosure represent a method, it should be understood in the context of the present application that such a method is a process for execution by a computer, ie, a computer-implementable method. The various steps of the method thus reflect different parts of a computer program, e.g., different parts of one or more algorithms.

[0081] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0082] The computer-readable storage medium may be any physical device capable of retaining and storing instructions for use by an instruction execution unit. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM).Flash memory), an SCM (Storage Class Memory), a static random access memory (SRAM), a portable CD-ROM, a DVD (Digital Versatile Disc), a memory stick, a floppy disk, a mechanically encoded device such as punched cards or raised structures in a groove on which instructions are stored, and any suitable combination thereof. A computer-readable storage medium, as used herein, shall not be construed as containing transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., pulses of light traveling through fiber optic cables), or electrical signals transmitted through a wire.

[0083] Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to respective computing / processing units or to an external computer or storage unit via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routers, firewalls, switching units, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing unit receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing unit.

[0084] Computer-readable program instructions for performing operations of the present disclosure may be assembly language instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, as well as traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server.In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, over the Internet using an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present disclosure.

[0085] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that instructions executing via the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions / steps specified in the flowchart and / or block diagram block(s). These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium on which instructions are stored comprises an article of manufacture, including instructions implementing aspects of the function(s) specified in the flowchart and / or block diagram block(s).implement the function / step specified in the blocks of the flow charts and / or block diagrams.

[0087] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of process steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / steps specified in the block(s) of flowcharts and / or block diagrams.

[0088] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions comprising one or more executable instructions for performing the particular logical function or functions. In some alternative implementations, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown in succession may actually execute substantially concurrently, or the blocks may sometimes execute in reverse order depending on the corresponding functionality.It is further to be understood that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or by combinations of special purpose hardware and computer instructions.

[0089] The descriptions of the various embodiments of the present disclosure have been presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Those skilled in the art will appreciate that numerous modifications and variations are possible without departing from the spirit and scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical application, or technical improvement over commercially available technology, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0090] In summary, various embodiments have been discussed, which in turn are specified in the following numbered examples:

[0091] Example 1 is as follows. A computer-implemented method for processing a message stream, the method comprising: obtaining messages of a message stream, each message having an associated timestamp; storing each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first location in the queue; and controlling retrieval of a target message from the first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-order message of each of the other queues of the plurality of queues not including the target queues.

[0092] Example 2 is as follows. The method of example 1, wherein controlling the retrieval comprises: for each of the other queues, determining a time difference between the timestamp of the target message and a first-in-line message of the queue; identifying a largest time difference from the determined time differences; and permitting retrieval of the target message from the target queue based on whether the largest time difference exceeds a threshold.

[0093] Example 3 is as follows. The method of examples 1 or 2, wherein permitting retrieval of the target message comprises: permitting retrieval of the target message from the target queue when the largest value of a time difference is less than the threshold.

[0094] Example 4 is as follows. The method of any of the preceding examples, further comprising: rejecting retrieval of the message from the destination queue if the largest value of a time difference exceeds the threshold; and generating an error message in response to rejecting retrieval of the message from the destination queue.

[0095] Example 5 is as follows. The method of any of the preceding examples, further comprising: generating an error message in response to the retrieval of the message from the destination queue being rejected.

[0096] Example 6 is as follows. The method according to any of the preceding examples, further comprising: delaying a rejected retrieval of the message from the destination queue.

[0097] Example 7 is as follows. The method of any of the preceding examples, further comprising: determining that the message associated with the longest value is no longer in its associated queue; identifying a remaining largest value of time differences from the determined time differences; and allowing retrieval of the target message from the target queue if the remaining largest value of a time difference is less than the threshold.

[0098] Example 8 is as follows. The method of any of the preceding examples, further comprising: receiving an input signal representative of a timing request, and determining the threshold based on the received input signal.

[0099] Example 9 is as follows. The method according to any one of the preceding examples, wherein the threshold is equal to an average of the first message and a second message in each queue.

[0100] Example 10 is as follows. The method of any of the preceding examples, further comprising: for each received component, generating a timestamp representative of a time associated with the message; and assigning the generated timestamp to the message.

[0101] Example 11 is as follows. The method of any of the preceding examples, wherein: each message relates to at least one of a plurality of topics; the method further comprises, for each of the plurality of topics, defining a plurality of queues associated with at least one topic; and wherein storing each message in at least one of a plurality of queues comprises: storing each message in at least one of the plurality of queues associated with the topic(s) to which the message relates.

[0102] Example 12 is as follows. A computer program product for processing a message stream, the computer program product comprising a computer-readable storage medium having program instructions embodied thereon, the program instructions executable by a processing unit to cause the processing unit to: obtain messages of a message stream, each message having an associated timestamp; store each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first position in the queue;and control retrieval of a target message from a first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of the first-in-line message of each of the other queues of the plurality of queues;

[0103] Example 13 is as follows. The computer program product of example 12, wherein the program instructions are further configured to cause the processing unit to: determine, for each of the other queues, a time difference between the timestamp of the target message and a first-in-line message of each queue; and identify a largest value of a time difference from the determined time differences.

[0104] Example 14 is as follows. The computer program product of examples 12 or 13, wherein the program instructions are further configured to cause the processing unit to: allow retrieval of the target message from the target queue if the largest value of a time difference is less than the threshold.

[0105] Example 15 is as follows. The computer program product of any of Examples 12 to 14, wherein the program instructions are further configured to cause the processing unit to: reject a retrieval of the message from the target queue if the largest value of a time difference exceeds the threshold; and in response to rejecting the retrieval of the message from the target queue, generate an error message.

[0106] Example 16 is as follows.A system for processing a message stream, the system comprising: an interface configured to receive messages of a message stream, each message having an associated timestamp; a queue component configured to store each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first location in the queue; and a message retrieval component configured to control retrieval of a target message from a first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-order message of each of the other queues of the plurality of queues that are not the target queue.

[0107] Example 17 is as follows. The system of Example 16, wherein the message retrieval component is configured to: determine, for each of the other queues, a time difference between the timestamp of the target message and the first-in-line message of the queue; identify the largest value of a time difference from the determined time differences; and permit retrieval of the target message from the requested queue based on whether the largest value of a time difference exceeds a threshold.

[0108] Example 18 is as follows. The system of examples 16 or 17, wherein the message retrieval component is further configured to: allow retrieval of the target message from the requested queue if the largest value of a time difference does not exceed the threshold.

[0109] Example 19 is as follows. The system of any of Examples 16 to 18, wherein the message retrieval component is further configured to generate an error message in response to a retrieval of the target message from the requested queue being rejected when the largest value of a time difference exceeds the threshold.

[0110] Example 20 is as follows. The system of any of Examples 16 to 19, further comprising: a coordination configuration component configured to receive an input signal representative of a timing coordination request and to determine the threshold based on the received input signal.

[0111] Example 21 is as follows. The system of any of examples 16 to 20, further comprising: a time component configured to generate, for each received component, a timestamp representative of a time associated with the message, and to assign the generated timestamp to each message.

[0112] Example 22 is as follows. The system of any of examples 16 to 21, wherein each message relates to at least one of a plurality of topics, wherein the queue component is further configured to define, for each of the plurality of topics, a plurality of queues associated with the topic; and wherein the queue component is further configured to store each message in each of the plurality of queues associated with the topic(s) to which the message relates.

[0113] Example 23 is as follows. The system of any of examples 16 to 22, wherein a user defines the threshold.

[0114] Example 24 is as follows. A computer-implemented method for processing a message stream, the method comprising: obtaining messages of a message stream, each message having an associated timestamp, each message relating to at least one of a plurality of topics; defining a plurality of queues associated with at least one topic; and storing each message in at least one of the plurality of queues associated with the at least one topic to which each message relates; storing each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first position in the queue; generating a timestamp representative of a time associated with the message;Assigning the generated timestamp to the message; controlling retrieval of a target message from the first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-placed message of each of the other queues of the plurality of queues not including the target queue; for each of the other queues, determining a time difference between the timestamp of the target message and a first-placed message of the queue; identifying a largest value of a time difference from the determined time differences; receiving an input signal representative of a timing request; determining the threshold based on the received input signal; and permitting retrieval of the target message from the target queue in response to the largest value of a time difference being less than the threshold.

[0115] Example 25 is as follows. A system comprising: a processor; and a computer-readable storage medium communicatively coupled to the processor and storing program instructions that, when executed by the processor, are configured to cause the processor to: receive messages of a message stream, each message having an associated timestamp, each message relating to at least one of a plurality of topics; define a plurality of queues associated with at least one topic; store each message in at least one of the plurality of queues associated with the at least one topic to which each message relates;store each message in at least one of a plurality of queues, wherein the messages of each queue are sorted based on their timestamp such that a message with the earliest timestamp is placed at a first position in the queue; generate a timestamp representative of a time associated with the message; assign the generated timestamp to the message; control retrieval of a target message from the first position in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-placed message of each of the other queues of the plurality of queues not including the target queue; determine, for each of the other queues, a time difference between the timestamp of the target message and a first-placed message of the queue;to identify a largest value of a time difference from the determined time differences; to receive an input signal representing a timing request; to determine the threshold value based on the received input signal; to delay a rejected retrieval of the message from the target queue; to determine that the message associated with the longest value is no longer in its associated queue; to identify a remaining largest value of time differences from the determined time differences; and to allow retrieval of the target message from the target queue if the remaining largest value of a time difference is less than the threshold value.

Claims

[1] A computer-implemented method for processing a message data stream, the method comprising: Obtaining (410) messages of a message stream, each message (PA-1) having an associated timestamp; Storing (420) each message (PA-1) in at least one of a plurality of queues (322), wherein the messages of each queue (322) are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first position in the queue (322); controlling (430) retrieval of a target message from the first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-in-line message (PA-1) of each of the other queues (322) of the plurality of queues (322) that do not include the target queue; Receiving an input signal representative of a timing request; and Determining the threshold based on the received input signal, wherein the threshold is equal to an average of the first message and a second message in each queue (322). [2] The method of claim 1, wherein controlling (430) the retrieval comprises: for each of the other queues (322), determining a time difference between the timestamp of the target message and a first-in-line message of the queue (322); Identifying a largest value of a time difference from the determined time differences; and Allowing retrieval of the target message from the target queue (322) based on whether the largest value of a time difference exceeds a threshold. [3] The method of claim 2, wherein allowing retrieval of the target message comprises: Allow retrieval of the target message from the target queue if the largest value of a time difference is less than the threshold. [4] The method of claim 2, further comprising: Reject retrieval of the message from the destination queue if the largest value of a time difference exceeds the threshold. [5] The method of claim 4, further comprising: Generate an error message in response to the message being rejected from the destination queue. [6] The method of claim 2, further comprising: Delaying a rejected message retrieval from the destination queue. [7] The method of claim 6, further comprising: Determine that the message corresponding to the longest value is no longer in its associated queue; Identifying a remaining largest value of time differences from the determined time differences; and Allow retrieval of the target message from the target queue if the remaining largest value of a time difference is less than the threshold. [8] The method of claim 1, further comprising for each received message (PA-1): Generating a timestamp representing a time associated with the message; and Assign the generated timestamp to the message (PA-1). [9] The method of claim 1, wherein: each message (PA-1) refers to at least one of a plurality of topics; the method further comprising, for each of the plurality of topics, defining a plurality of queues (322) associated with at least one topic; and wherein storing (420) each message in at least one of a plurality of queues (322) comprises: storing each message in at least one of the plurality of queues associated with the topic(s) to which the message relates. [10] A computer program product for processing a message data stream, the computer program product comprising a computer-readable storage medium having program instructions contained thereon, the program instructions being executable by a processing unit to carry out the method according to any one of claims 1 to 9. [11] A system for processing a message data stream, the system comprising: an interface (310) configured to receive messages (PA-1) of a message stream, each message (PA-1) having an associated timestamp; a queue component (320) configured to store each message (PA-1) in at least one of a plurality of queues (322), wherein the messages of each queue (322) are sorted based on their timestamp such that a message with the earliest timestamp is placed in a first position in the queue (322); and a message retrieval component (310) configured to control retrieval of a target message from a first location in a target queue based on a time difference between the timestamp of the target message and the timestamps of a first-place message of each of the other queues (322) of the plurality of queues (322) that are not the target queue, wherein the message retrieval component (330) is further configured to for each of the other queues (322), determine a time difference between the timestamp of the target message and the first message in the queue; to identify the largest value of a time difference from the determined time differences; and permit retrieval of the target message from the requested queue (322) based on whether the largest value of a time difference exceeds a threshold; allow retrieval of the target message from the requested queue based on whether the largest value of a time difference does not exceed the threshold; and The system also features: a coordination configuration component configured to receive an input signal representative of a timing coordination request and to determine the threshold based on the received input signal. [12] The system of claim 11, wherein the message retrieval component (330) is further configured to generate an error message in response to a retrieval of the target message from the requested queue being rejected when the largest value of a time difference exceeds the threshold. [13] The system of claim 11, further comprising: a time component configured to generate, for each received component, a timestamp representing a time associated with the message (PA-1) and to assign the generated timestamp to each message (PA-1). [14] The system of claim 11, wherein each message relates to at least one of a plurality of topics, wherein the queue component (320) is further configured to define, for each of the plurality of topics, a plurality of queues (322) associated with the topic; and wherein the queue component (320) is further configured to store each message (PA-1) in each of the plurality of queues (322) associated with the topic(s) to which the message relates. [15] The system of claim 11, wherein a user defines the threshold.

Images