Buffering elements to be processed

Abstract

Techniques relating to buffers are disclosed. A first buffer is for storing a number of elements to be processed. The disclosed techniques include providing a second buffer for storing elements to be added to the first buffer, and adding acquired elements to the second buffer when the number of elements in the first buffer is greater than a first threshold. These techniques can prevent overrun of the first buffer while allowing acquired elements to be added to the buffer.

Description

[Technical Field]

[0001] This application claims priority to U.S. Patent Application No. 18 / 084,276, filed December 19, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] In the computer field, a buffer is generally understood to refer to a portion of memory used to hold data elements to be processed. For example, incoming data elements to be processed by a processor may be added to a buffer. When the processor is ready to process one or more of the elements, the processor (or its thread) may remove one or more elements from the buffer and process them. The buffer is configured so that the rate of element processing does not necessarily match the rate at which elements are received into the buffer. [Brief explanation of the drawings]

[0003] With reference to the following drawings, specific embodiments are disclosed. [Figure 1] FIG. 1 illustrates a block diagram of an exemplary computing device that may be used to implement certain embodiments. [Figure 2] FIG. 2 illustrates a block diagram of an example system in which certain embodiments may be implemented. [Figure 3] FIG. 3 illustrates a block diagram of another example system in which certain embodiments may be implemented. [Figure 4] FIG. 4 shows a block diagram of queues and worker threads according to one embodiment. [Figure 5] FIG. 5 shows a block diagram of data flow between queues according to one embodiment. [Figure 6] FIG. 6 shows a block diagram of a connector according to one embodiment. [Figure 7] FIG. 7 illustrates a flow diagram of a process performed by a connector according to one embodiment. [Figure 8]FIG. 8 illustrates a flow diagram of another process performed by a connector according to one embodiment. [Figure 9] FIG. 9 illustrates a flow diagram of a method performed by a connector according to one embodiment. [Figure 10] FIG. 10 illustrates a block diagram of an exemplary electronic trading system in which certain embodiments may be employed. [Figure 11] FIG. 11 illustrates a block diagram of another exemplary electronic trading system in which certain embodiments may be employed.

[0004] Certain embodiments will be better understood when read in conjunction with the provided drawings, which illustrate examples. It should be understood, however, that the embodiments are not limited to the arrangements and apparatus shown in the accompanying drawings. DETAILED DESCRIPTION OF THE INVENTION

[0005] The disclosed embodiments relate generally to buffers, and more particularly to systems and methods in which elements are buffered for processing. Buffers, such as ring buffers, have a capacity for a fixed number of elements. This creates a risk of buffer overrun, where the number of elements added to the buffer exceeds the buffer's capacity and the elements may overwrite adjacent memory locations. In some cases, these adjacent memory locations may contain unprocessed data in the buffer, and thus overwriting this unprocessed data may result in data loss.

[0006] To avoid buffer overruns, flow control techniques can be employed. Specifically, flow control can be activated when the number of elements in a buffer approaches or reaches the buffer's capacity. When flow control is activated, threads are restricted from adding another element to the buffer; instead, threads may spin (i.e., repeatedly attempt to add elements to the buffer) until the number of elements in the buffer decreases.

[0007] However, this technique has drawbacks. One drawback is the impact it has on processes that add elements to the buffer. Specifically, such processes may be forced to wait until the number of elements in the buffer decreases before they can proceed, which can block the process.

[0008] In certain situations, this blocking can, in turn, pose a risk of deadlock. A deadlock is a situation in which neither of two (or more) interdependent processes can logically proceed. One example is when a particular thread is processing elements in a particular buffer and adding elements to a particular buffer. However, if the buffer is approaching or at capacity, at which point the flow control described above is activated, the thread will spin when attempting to add elements to the buffer. Because the thread is spinning, it cannot complete its work on the elements in the buffer, and therefore the elements are not removed from the buffer. Thus, the number of elements in the buffer does not decrease, and the thread may spin indefinitely. Because no logical progress can be made in adding or removing elements, a deadlock occurs. Another example is when a loop is established between two or more buffers, which can result in a deadlock. For example, if a first thread removing elements from a first buffer and adding elements to a second buffer is spinning (e.g., because the second buffer is approaching or at capacity and therefore flow control is activated), and a second thread removing elements from the second buffer and adding elements to the first buffer is spinning (e.g., because the first buffer is approaching or at capacity and therefore flow control is activated), then neither thread can logically make progress and a deadlock occurs.

[0009] Deadlocks are undesirable because they halt processing and may require it to be restarted. This is an inefficient use of resources and may have undesirable consequences for the processing itself. For example, a deadlock in a real-time message processing system may lead to message loss, which is highly undesirable. Furthermore, the risk of deadlock itself is undesirable because it may impose restrictions on code used in connection with buffers. For example, code that loops between buffers or is at risk of looping may need to be avoided or rewritten to reduce the risk of deadlock. This may limit how buffers can be used. It is desirable to mitigate at least some of these drawbacks.

[0010] I. BRIEF DESCRIPTION OF SPECIFIC EMBODIMENTS

[0011] Certain embodiments provide a system including a memory providing a first buffer for storing a number of elements to be processed and a second buffer for storing elements to be added to the first buffer. The system also includes at least one processor for retrieving elements to be processed, determining that the number of elements in the first buffer is greater than a first threshold, and adding the retrieved elements to the second buffer. The system also provides a second buffer for storing elements to be added to the first buffer, and adds the retrieved elements to the second buffer if the number of elements in the first buffer is greater than the first threshold. This allows the processor to add elements to the buffer while preventing overrun of the first buffer. Thus, a process attempting to add an element to the first buffer does not have to wait until the number of elements in the first buffer decreases, but rather, at least one processor adds the element to the second buffer. This avoids blocking of such processes, thereby improving the overall speed of such processes. Furthermore, this eliminates the risk of deadlock. Specifically, even if the number of elements in the first buffer is greater than the first threshold, at least one processor (specifically, its worker thread) adds elements to the second buffer rather than spinning while attempting to add elements to the first buffer. Therefore, the worker thread can logically proceed with processing even when the number of elements in the first buffer is greater than the first threshold, thereby eliminating the risk of deadlock associated with the first buffer. Eliminating the risk of deadlock can avoid or reduce the risk of needing to restart processing, improving overall system efficiency and / or preventing the loss of elements in the buffer. Furthermore, eliminating the risk of deadlock in the first buffer can increase the flexibility of how the first buffer can be used. For example, eliminating the risk of deadlock allows code to be written without having to avoid loops between buffers, resulting in more concise and efficient code.Furthermore, eliminating the risk of deadlock due to the first buffer may allow the size of the first buffer to be reduced, potentially reducing memory requirements. For example, without the second buffer, the first buffer may need to be relatively large to reduce the risk of deadlock, depending, for example, on the expected peak element addition rate and buffer usage. However, eliminating the risk of deadlock due to the second buffer means that the size of the first buffer (which may be provided by, for example, a ring buffer) does not need to be determined by these parameters and can be made smaller without the risk of deadlock.

[0012] In certain embodiments, the at least one processor is further configured to add the retrieved element to the first buffer based at least in part on determining that the number of elements in the first buffer is less than a first threshold. This indicates that in “normal” operation, i.e., when the number of elements in the first buffer is less than the first threshold, elements are added to the first buffer. This indicates efficient processing of elements, i.e., in normal operation, only the first buffer is relevant, and when the first buffer is approaching or at its capacity, the second buffer is used, thereby, for example, eliminating the risk of deadlock.

[0013] In particular embodiments, the at least one processor is configured to set an overflow flag if the number of elements in the first buffer is greater than a first threshold, and the at least one processor is configured to determine that the number of elements in the first buffer is greater than the first threshold by determining that the overflow flag is set. This may provide an efficient implementation of determining that the number of elements in the first buffer is greater than the threshold. The at least one processor may test the overflow flag, and if it is set, the at least one processor may add the retrieved elements directly to the second buffer. This may be more computationally efficient than, for example, obtaining a count of the number of elements in the first buffer and comparing it to the first threshold.

[0014] In certain embodiments, the at least one processor is further configured to determine that the number of elements in the first buffer is less than a second threshold, where the second threshold is less than the first threshold, and to remove elements from the second buffer and add them to the first buffer. This ensures that elements are removed from the second buffer and added to the first buffer when the number of elements in the first buffer is sufficiently reduced, e.g., so that there is no longer a risk of overrunning the first buffer. This ensures that the first buffer (which may be configured for high performance, e.g., as a ring buffer) is used to process elements. Note that as long as the second threshold is equal to or greater than one element, the use of the second buffer does not add latency to the processing of elements from the first buffer.

[0015] In certain embodiments, the first buffer is a first queue that queues elements to be processed, and the second buffer is a second queue that queues elements for addition to the first queue, allowing elements to be processed in order, i.e., in the order in which they were retrieved, which may be important for certain elements, such as protocol messages for maintaining a session between a client device and a host, or other types of messages.

[0016] In particular embodiments, the at least one processor is further configured to determine that the number of elements in the second buffer is greater than zero and add the retrieved element to the second buffer. This ensures that the elements are processed in the order in which they were retrieved. For example, if there are already waiting elements in the second queue, the at least one processor posts to the second queue instead of the first queue. As noted above, processing elements in the order in which they were retrieved may be important for certain types of elements, and this may be useful for implementing such sequential processing requirements.

[0017] In particular embodiments, the at least one processor is further configured to: determine that the second buffer is empty and that the number of elements in the first buffer is less than a first threshold; and add the retrieved element to the first buffer. This ensures that the elements are processed in the order in which they were retrieved. For example, the at least one processor may add the retrieved message to the first buffer only if there are no elements waiting in the second buffer and the number of messages in the first buffer is less than the first threshold. This may help implement sequential processing requirements.

[0018] In particular embodiments, the at least one processor is configured to set an overflow flag if the number of elements in the first buffer is greater than a first threshold and to reset the overflow flag when the at least one processor determines that the second buffer is empty, and to determine that the second buffer is empty and the number of elements in the first buffer is less than the first threshold by determining that the overflow flag is reset. This allows the determination that the second buffer is empty and the number of elements in the first buffer is less than the first threshold to be efficiently implemented by referencing a single flag, which may be more computationally efficient than, for example, performing each determination individually and / or explicitly each time.

[0019] In a particular embodiment, the first buffer is a ring buffer, which allows for high performance. Specifically, the contiguous nature of the memory in a ring buffer allows elements to be read quickly. Because ring buffers are inherently limited in size, the second buffer maintains the performance of a ring buffer while eliminating the risk of deadlock.

[0020] In certain embodiments, the second buffer is a queue that queues elements for addition to the first buffer. This may be useful for implementing sequential processing requirements. In certain embodiments, the second buffer is an unbounded buffer. This may ensure that the second buffer never fills up. The second buffer may be practically unbounded. That is, while the second buffer technically has a capacity, that capacity may be so large that in practice the second buffer may be considered unbounded. That is, in practice there is little risk of the second buffer filling up. This may be useful for reducing or eliminating the risk of buffer overruns in the second buffer.

[0021] In certain embodiments, at least one processor is configured to implement a number of worker threads that add to the first buffer, where the first threshold is at most the total number of elements the first buffer has capacity for minus the number of worker threads. This may ensure the maximum value of the first threshold while ensuring that the first buffer does not overrun. For example, each worker thread may add elements to the first buffer one element at a time. Even if all of the total number of worker threads simultaneously add an element to the first buffer, the first buffer will not overrun, and additional elements may be added by the worker threads to the second buffer. This may ensure that elements are not lost or overwritten and may further ensure that the capacity of the first buffer is fully utilized.

[0022] In a particular embodiment, the element is a message, and the at least one processor is configured to obtain the element by receiving the message from an external entity via the communication interface, thereby ensuring that the process of receiving messages at the communication interface is not blocked when the number of messages in the first buffer is greater than a first threshold, which in turn ensures that messages are not lost.

[0023] In certain embodiments, at least one processor is configured to retrieve elements by removing them from a third buffer. In an example embodiment, processing elements from a first buffer may include adding elements to a third buffer. This situation may typically result in a risk of deadlock. However, if the number of elements in the first buffer is greater than a threshold, the at least one processor adds the retrieved elements to a second buffer, thereby eliminating this risk of deadlock. Thus, even if processing elements from a first buffer includes adding elements to a third buffer, elements can be retrieved safely, i.e., without risk of deadlock, from any third buffer. In an example embodiment, the first buffer may be an order queue, and the third buffer may be a worker queue.

[0024] In a particular embodiment, the elements are each references to one or more message queues. The first queue is a worker queue, to which one or more references to message queues, such as a session queue or an action queue, can be added. This eliminates the risk of process blocking and / or deadlock, even if the first buffer is a buffer to which one or more references to message queues are added by a worker thread.

[0025] In particular embodiments, the element is associated with a message, and the system includes a server system including at least one processor and memory, the server system configured to facilitate messaging between a host system and one or more client systems. For example, the server system may implement a connector that facilitates messaging between the host system and one or more client systems. In such server systems, avoiding process blocking and / or deadlock may be particularly important because connections between the host system and one or more client systems may be severed if messages, such as protocol messages, are not processed within a certain amount of time. In examples, the element is associated with a message, and the element may be a message, a reference to a message, a reference to a queue of messages, or a reference to a queue of references to messages.

[0026] In certain embodiments, one or more of the client systems includes a trading device and the host system includes an electronic exchange. In such systems, avoiding process blocking or deadlock can be particularly important because timing of messages between the trading device and the electronic exchange is critical. Furthermore, eliminating the risk of deadlock reduces the risk of requiring a system restart, which could result in lost messages and / or resource-intensive message reconciliation processes. Thus, a more efficient and / or consistent trading connector may be provided.

[0027] In a particular embodiment, the server system is configured to establish multiple connections between the host system and a respective plurality of client systems, the memory is configured to provide, for each of the multiple connections, a first buffer and an associated second buffer, and the at least one processor is configured, for at least some of the connections, to add an element (e.g., a message or a reference to a message) to the respective first buffer or second buffer for that connection. Having different first and second buffers for each different connection can ensure that bursts of messages in a first connection do not affect the speed of processing of a second connection. Rather, bursts of messages are added to the second buffer for the first connection. This allows the system to operate consistently across connections. This also allows the processing of messages from different connections to be parallelized, helping to improve processing speed.

[0028] In certain embodiments, messaging between the host system and one or more client systems is bidirectional, the memory is configured to provide a first buffer and an associated second buffer for each of a first messaging direction and a second messaging direction, and the at least one processor is configured, for each messaging direction, to add message elements (e.g., messages or references to messages) to the respective first buffer or second buffer for that direction. Having different first and second buffers for each messaging direction can ensure that bursts of messages in one direction do not affect the speed of processing in the other direction. This allows, for example, messages from the host to be delivered to client devices even if processing of messages from one or more client devices to the host is relatively slow. This can also parallelize the processing of messages in different messaging directions, which can help improve processing speed.

[0029] In certain embodiments, the server system includes an order connector of an electronic trading platform, the order connector including at least one processor and memory. The electronic trading platform may be in a particular situation where it is important that messages (e.g., trade orders and / or execution reports) are processed quickly, that connections between trading devices and exchanges are not lost (because a loss could result in the cancellation of all trade orders in a session), that certain ordering requirements are implemented, and / or that the speed of processing on one connection is not affected by the speed of processing on another connection. Accordingly, this is a situation in which certain embodiments are particularly useful.

[0030] Certain embodiments provide a method that includes providing, in memory, a first buffer for storing a number of elements to be processed and a second buffer for storing elements to be added to the first buffer, The method also includes, by at least one processor, obtaining the elements to be processed, determining that the number of elements in the first buffer is greater than a first threshold, and adding the obtained elements to the second buffer.

[0031] Certain embodiments provide a tangible computer-readable medium comprising instructions that, when executed, cause a machine to perform at least the following steps: providing, in memory, a first buffer for storing elements to be processed and a second buffer for storing elements to be added to the first buffer, by at least one processor, obtaining the elements to be processed, determining that the number of elements in the first buffer is greater than a first threshold, and adding the obtained elements to the second buffer.

[0032] II. Examples of Computing Devices

[0033] FIG. 1 shows a block diagram of an exemplary computing device 100. Computing device 100 may be used to implement certain embodiments described herein. In other examples, other computing devices may be used. Computing device 100 includes a communication bus 110, a processor 112, a memory 114, a network interface 116, input devices 118, and output devices 120. Processor 112, memory 114, network interface 116, input devices 118, and output devices 120 are coupled to communication bus 110. Computing device 100 is connected to an external network 140, such as a local area network (LAN) or a wide area network (WAN) such as the Internet. Computing device 100 is connected to external network 140 via network interface 116. Computing device 100 may include additional, different, or fewer components. For example, multiple communication buses (or other types of component interconnects), multiple processors, multiple memory devices, multiple interfaces, multiple input devices, multiple output devices, or any combination thereof may be provided. As another example, computing device 100 may not include input devices 118 or output devices 120. As another example, one or more components of computing device 100 may be combined into a single physical element, such as a field programmable gate array (FPGA) or a system on a chip (SoC).

[0034] Communication bus 110 may include a channel, electrical or optical network, circuit, switch, fabric, or other mechanism for communicating data between components within computing device 100. Communication bus 110 is communicatively coupled to any of the components of computing device 100 and may transfer data therebetween.

[0035] Processor 112 may be any suitable processor, processing unit, or microprocessor. Processor 112 may include, for example, one or more general-purpose processors, digital signal processors, application-specific integrated circuits, FPGAs, analog circuits, digital circuits, programmed processors, and / or combinations thereof. Processor 112 may be a multi-core processor that may include multiple processing cores of the same or different types. Processor 112 may be a single device or a combination of devices, such as one or more devices associated with a network or distributed processing system. Processor 112 may support various processing strategies, such as multiprocessing, multitasking, parallel processing, and / or remote processing. Processing may be performed locally or remotely and may be moved from one processor to another. In particular embodiments, computing device 100 is a multi-processor system and, as such, may include one or more additional processors communicatively coupled to communication bus 110.

[0036] Processor 112 is operable to execute logic and other computer-readable instructions encoded on one or more tangible media, such as memory 114. As used herein, logic encoded on one or more tangible media includes instructions executable by processor 112 or another processor. Logic may be stored as part of software, hardware, an integrated circuit, firmware, and / or microcode, for example. Logic may be received from an external communication device via communications network 140. Processor 112 may execute logic to perform the functions, operations, or tasks described herein.

[0037] Memory 114 may be one or more tangible media, such as, for example, a computer-readable storage medium. Computer-readable storage media may include various types of volatile and non-volatile storage media, such as, for example, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, any combination thereof, or other tangible data storage devices. As used herein, the terms "non-transitory" or "tangible computer-readable medium" are expressly defined to include any type of computer-readable medium and to exclude propagating signals. Memory 114 may include any desired type of mass storage device, such as a hard disk drive, optical media, magnetic tape or disks, etc.

[0038] Memory 114 may include one or more memory devices. For example, memory 114 may include cache memory, local memory, mass storage, volatile memory, non-volatile memory, or a combination thereof. Memory 114 may be adjacent to processor 112, part of processor 112, programmed with processor 112, networked with processor 112, and / or remote from processor 112, such that data stored in memory 114 may be retrieved and processed by processor 112, for example. Memory 114 may store instructions executable by processor 112. The instructions may be executed to perform one or more operations or functions described herein.

[0039] The memory 114 may store an application 130 that implements the disclosed techniques. In particular embodiments, the application 130 may be accessed from or stored in different locations. The processor 112 may access the application 130 stored in the memory 114 and execute computer-readable instructions included in the application 130.

[0040] Network interface 116 may include one or more network adapters. The network adapters may be wired or wireless network adapters. Network interface 116 enables computing device 100 to communicate with an external network 140. Computing device 100 may communicate with other devices through network interface 116 using one or more network protocols, such as Ethernet, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), a wireless network protocol such as Wi-Fi, a Long Term Evolution (LTE) protocol, or other suitable protocols.

[0041] The input devices 118 may include positional input devices such as a mouse, touchpad, touchscreen, keyboard, buttons, switches, and / or other human-machine interface devices. The output devices 120 may include a display, such as a liquid crystal display (LCD), cathode ray tube (CRT), light emitting diode (LED) display (e.g., an OLED display), or other suitable display.

[0042] In particular embodiments, during the installation process, the application may be transferred from input device 118 and / or network 140 to memory 114. When computing device 100 is executing or preparing to execute application 130, processor 112 may obtain instructions from memory 114 via communication bus 110.

[0043] III. Example Systems

[0044] FIG. 2 illustrates an exemplary system 200 in which certain embodiments disclosed herein may be employed. The system 200 includes client devices 202a, 202b, a connector 204, and a host 210. Each of the client devices 202a, 202b, the connector 204, and the host 210 may be provided by a respective computing device, such as the computing device 100 according to any of the embodiments described above with reference to FIG. 1. The connector 204 may be provided by a server system. As shown in FIG. 2, the connector 204 includes a processor 206 and a memory 208. The processor 206 and / or the memory 208 may be the same as or similar to the processor 112 and / or the memory 114, respectively, of the computing device 100 described above with reference to FIG. 1. Each client device 202a, 202b communicates with the connector 204 via a first network 211. The first network 211 may include, for example, a local area network (LAN) and / or a wide area network (WAN) such as the Internet. Connector 204 communicates with host 210 via second network 212. Second network 212 may include a LAN, a WAN, a shared memory system, and / or a proprietary network. In an embodiment, first network 211 and second network 212 may be the same network.

[0045] The connector 204 is disposed between the client devices 202a, 202b and the host 210. In this embodiment, the connector 204 acts as an intermediary between the client devices 202a, 202b and the host 210. Specifically, the connector 204 performs protocol conversion for data communicated between the client devices 202a, 202b and the host 210. In this embodiment, the connector 204 processes messages received from the client devices 202a, 202b into a data format understandable by the host 210. The connector 204 also processes messages received from the host 210 into a format understandable by the client devices 202a, 202b. For example, the client devices 202a, 202b can send request messages to the host 210 via the connector 204, the host 210 can execute the request in the request message, and can send response messages indicating the extent to which the request has been executed to the client devices 202a, 202b via the connector 204. Messages are received by connector 204 via a network interface (not shown in FIG. 1, but see network interface 116 of exemplary computing device 100 in FIG. 1) of connector 204. Messages received by connector 204 are processed by processor 206. Before being processed, the received messages are temporarily stored in memory 208, and specifically, in a buffer implemented by memory 208, as described in more detail below.

[0046] Figure 3 illustrates an example system 300 in which certain embodiments disclosed herein may be employed. The example system 300 may be the same as or similar to the example system 200 described above with reference to Figure 2. The example system includes client devices 302a, 302b, a connector 304 having a processor 306 and memory 308, and a host 310. The client devices 302a, 302b, the connector 304, and the host 310 may be the same as or similar to the client devices 202a, 202b, the connector 204, and the host 210, respectively, of any of the examples described above with reference to Figures 1 and 2.

[0047] As shown in FIG. 3 , each client device 302 a, 302 b establishes a respective connection or session 320, 330 with the host 310 through a connector 304. The connector 304 may establish a separate connection with the host 310 for each session 320, 330. Each session 320, 330 may be established using an appropriate communication protocol. As an example of a message in a first session 320, the first client device 302 a may send a first message 322, such as a request, to the connector 304. The connector 304 may process the first message 322 to generate a transformed version of the first message 324, which the connector 304 may then send to the host 310. The host 310 may perform an action based on the received message 324 and send a second message 326, such as a response, to the connector 304. The connector 304 processes the second message 326 to generate a transformed version of the second message 328, and the connector 304 can send the transformed message 328 to the client device 302a. Each of the messages 322, 324, 326, and 328 is in the same first session 320. Therefore, each of the messages 322, 324, 326, and 328 has the same session identifier. Similarly, in the second session 330, the second client device 302b can send a first message 332, such as a request, to the connector 304, the connector 304 can process the first message 332 to generate a transformed version of the first message 334, and the connector 304 can send the transformed message 334 to the host 310. The host 310 can perform an action based on the received message 334 and send a second message 336, such as a response, to the connector 304. The connector 304 processes the second message 336 to generate a transformed version of the second message 338, and the connector 304 can send this transformed message 328 to the client device 302a. The messages 332, 334, 336, and 338 are each in the same second session 330.Thus, each of the messages 332, 334, 336, 338 has the same session identifier (although different from the session identifier of the messages of the first session 320).

[0048] Messages 322, 332 received by the connector 304 from client devices 302a, 302b are referred to in this disclosure as outbound messages, and messages 326, 336 received by the connector 304 from the host 310 are referred to in this disclosure as inbound messages.

[0049] IV. Queuing

[0050] In embodiments, it may be important for the connector 304 to process some messages in the order they are received. One example is messages from the client devices 302a, 302b (or alternatively, the host 310) related to a particular action on the host 310. For example, for a particular action on the host 310, the particular client device 302a may send a request message requesting that the particular action be performed, followed by a cancel message or a modify message that cancels or modifies, respectively, the request in the request message. It may be important for the connector 304 to process these messages in the order they are received at the order connector 304. If the cancel message or the modify message were sent to the host 310 before the request message, the request would not be canceled or modified, respectively. Another example is protocol messages, e.g., messages between the particular client devices 302a, 302b and the host 310, that establish or maintain sessions 320, 330. An example of a protocol message is a "heartbeat" message, which may be sent periodically from the host 310 to a particular client device 302a, 302b (or from the client device 302a, 302b to the host 310) to confirm that the particular client device 302a, 302b (or host 310) still exists and therefore maintains the associated session 320, 330. It may be important for the connector 304 to process protocol messages in the order received at the connector 304, because otherwise, protocol steps may not be performed in the expected order and the session may be terminated.

[0051] Connector 304 utilizes queues to process particular messages in the order they are received. A queue is a buffer that holds elements to be processed in a particular order. Specifically, a queue is a first-in, first-out (FIFO) buffer, meaning that elements in a queue are processed in the same order they were added to the queue. As messages are received at connector 304, the messages (or references to those messages) are placed in a queue in memory 308 where they are held until processor 306 is available to process them.

[0052] Processing all of the messages in the order in which they are received by the connector 304 can limit the rate at which messages can be processed. Specifically, speed can be increased by parallelizing the processing of some messages, i.e., processing some messages simultaneously. In particular, messages from different sessions 320, 330 can be processed simultaneously if the messages from those sessions are unordered relative to one another. Furthermore, messages associated with different actions on the host 310 can be processed simultaneously if the messages are unordered relative to one another. Furthermore, inbound and outbound messages received at the connector 304 can be processed simultaneously if the messages received from different directions (i.e., inbound and outbound messages) are unordered relative to one another.

[0053] FIG. 4 illustrates a buffer and worker thread architecture 400 that may be employed by connectors 204, 304 according to one embodiment. In this example, each buffer is a queue. Architecture 400 allows for processing some messages concurrently, as well as processing some messages in the order they are received. Referring to FIG. 4, architecture 400 includes, for a first session (such as first session 320 described above with reference to FIG. 3), an outbound session queue 402 and an inbound session queue 404, and a set of outbound action queues 406 and a set of inbound action queues 408. Architecture 400 also includes, for a second session (such as second session 330 described above with reference to FIG. 3), an outbound session queue 412 and an inbound session queue 414, and a set of outbound action queues 416 and a set of inbound action queues 418. Architecture 400 also includes a worker queue 420 and a pool of worker threads 422.

[0054] Each session queue 402, 404, 412, and 414 is associated with a particular session. For simplicity, FIG. 4 shows queues associated with only two sessions. However, it will be understood that architecture 400 may include more or fewer session queues as needed, depending on the number of sessions established between the client device and the host. Each set of action queues 406, 408, 416, and 418 is associated with a particular session, and within a particular set, each action queue is associated with a different action on the host. Each different action may be associated with a different action ID. The action ID may be included in all messages associated with a particular working action on the host and may identify the working action to which the message is associated. Outbound queues 402, 412, 406, and 416 are associated with messages received from client devices, and inbound queues 404, 408, 414, and 418 are associated with messages received from the host. In this example, each session queue 402, 404, 412, 414 and action queue 406, 408, 416, 418 is a queue of one or more messages. In an embodiment, rather than storing one or more messages, the session queues 402, 404, 412, 414 and / or action queues 406, 408, 416, 418 may store one or more references to a respective one or more messages. For example, for a particular message, the message itself may be stored elsewhere in memory 208, 308, and the references in queues 402, 404, 412, 414, 406, 408, 416, 418 may indicate where the message is stored and from where it can be accessed. Worker queue 420 may include references to one or more queues of messages, specifically session queues and / or action queues, as described in more detail below. That is, worker queue 420 may be a queue of queue references.

[0055] The queues 402, 404, 406, 408, 412, 414, 416, 418, and 420 are each portions of the connector 304's memory (such as memory 308 described above with reference to FIG. 3). Specifically, each queue 402, 404, 406, 408, 412, 414, 416, 418, and 420 may be implemented as a ring buffer. Each worker thread 422 may represent a virtual core of a physical processor (such as processor 306 described above with reference to FIG. 3). Each worker thread 422 in the pool is assigned a task, performs the task, and upon completion, the worker thread 422 returns to the pool ready to be assigned another task. Each worker thread 422 processes one task at a time, although different worker threads 422 in the pool can process different tasks simultaneously. While three worker threads 422 are shown in FIG. 4, it will be understood that more or fewer worker threads may be used as desired. Worker thread 422 obtains a queue reference from worker queue 420 and processes one or more messages in the referenced queue (such as session queues 402, 404, 412, 414 and action queues 406, 408, 416, 418).

[0056] To illustrate the operation of architecture 400, a first message 410 is received from a client device in a first session. The message 410 is added to an outbound session queue 402 associated with the first session. At this point, there may be one or more messages in the session queue 402, depending on the number of messages added to the session queue 402. The session queue 402 is added to a worker queue 420. That is, a reference to the session queue 402 is added to the worker queue 420. A worker thread 422 obtains a reference to the session queue 402 from the worker queue 420 and processes one or more messages in the referenced session queue 402. Specifically, for each of the one or more messages in the referenced session queue 402, the worker thread 422 deserializes the message. If the message is not associated with a particular action and therefore does not include an action ID, for example, a protocol message, the worker thread 422 may process the message as needed and forward it to the host as needed. However, if the message is associated with a particular action, the worker thread 422 determines the action ID contained in the message and adds the deserialized message to the action queue 406 associated with the determined action ID. The worker thread 422 then adds the action queue 406 to the worker queue 420. That is, a reference to the action queue 406 is added to the worker queue 420. The worker thread 422 obtains a reference to the action queue 406 from the worker queue 420 and performs processing on one or more messages in the referenced action queue 406. For example, this may include converting the message so that it can be understood by the host and forwarding the converted message to the host.

[0057] Concurrently with the receipt of the first message 410, a second message 411 may be received from the host in the second session. The second message 411 is added to an inbound session queue 414 associated with the second session. The session queue 414 is added to a worker queue 420. That is, a reference to the session queue 414 is added to the worker queue 420. A worker thread 422 obtains a reference to the session queue 414 from the worker queue 420 and processes one or more messages in the referenced session queue 414. Specifically, for each of the one or more messages in the session queue 414, the worker thread 422 deserializes the message. If the message is not associated with a particular action and therefore does not include an action ID, e.g., a protocol message, the worker thread 422 may process the message as needed and forward it to the host as needed. However, if the message is associated with a particular action, the worker thread 422 determines the action ID included in the message and adds the deserialized message to the action queue 418 associated with the determined action ID. Next, worker thread 422 adds action queue 418 to worker queue 420. That is, a reference to action queue 418 is added to worker queue 420. Worker thread 422 obtains a reference to action queue 418 from worker queue 420 and performs processing on one or more messages in the referenced action queue 418. For example, this may include converting the messages so that they can be understood by the client device and forwarding the converted messages to the client device.

[0058] According to architecture 400, protocol messages and messages associated with a particular action ID for a particular session and message direction are processed in the order in which they are received. However, messages associated with different sessions, different message directions, and / or different action IDs may be processed concurrently. Thus, architecture 400 can process messages at high speed while processing some messages in the order in which they are received.

[0059] V. Flow Control and Deadlock

[0060] As previously mentioned, each queue 402, 404, 406, 408, 412, 414, 416, 418, and 420 can be implemented as a fixed-size ring buffer. When implemented as a ring buffer, each queue has a fixed capacity for messages (or references to messages or queues, as appropriate). Therefore, for each queue, it is possible for the queue to become full. Adding a message to an already full queue implemented as a ring buffer can cause a buffer overrun, specifically overwriting the first message in the queue. This causes the first message in the queue to be lost. To avoid this, a flow control scheme may be used. According to the flow control scheme, when the number of messages in a particular queue 402, 404, 406, 408, 412, 414, 416, 418, 420 reaches a certain threshold, the worker thread 422 is restricted from adding another message to the particular queue; instead, the worker thread 422 spins (i.e., repeatedly attempts to add a message to the particular queue) until the number of messages in the particular queue decreases. While this prevents overruns, this control scheme has drawbacks, a particular drawback being the risk of deadlock.

[0061] Figure 5 illustrates an example situation in which the flow control scheme described above may cause deadlock. The block diagram of Figure 5 shows a session queue 502, an action queue 506, a worker queue 520, and two worker threads 522a and 522b. Session queue 502 and action queue 506 may be, for example, session queues 402, 404, 412, and 414 and action queues 406, 408, 416, and 418 for any one of the particular sessions and message directions described above with reference to Figure 4. Worker queue 520 and worker threads 522a and 522b may be the same as or similar to worker queue 420 and worker thread 422, respectively, described above with reference to Figure 4.

[0062] Similar to the process described above with reference to FIG. 4, in FIG. 5, a reference to session queue 502 is added to worker queue 520. A first worker thread 522a obtains a reference to session queue 502 from worker queue 520, obtains a message from the referenced session queue 502, deserializes the message, and attempts to add the deserialized message to action queue 506. However, if action queue 506 is approaching or at capacity, a flow control scheme is activated for action queue 506. In this case, first worker thread 522a is prevented from adding the deserialized message to action queue 506; instead, worker thread 522a spins, i.e., repeatedly attempts to add the deserialized message to action queue 506 (denoted by an "X" in FIG. 5). Meanwhile, worker queue 520 itself may be approaching or at capacity, resulting in a flow control scheme being activated for worker queue 520. The second worker thread 522b may attempt to add a reference to action queue 506 to the worker queue 520. However, because a flow control scheme is activated for the worker queue 520, the second worker thread 522b is prevented from adding a reference to action queue 506 to the worker queue 520, and instead, the second worker thread 522b spins, that is, repeatedly attempts to add a reference to action queue 506 to the worker queue 520 (denoted by an "X" in FIG. 5). In this situation, the first worker thread 522a is spinning, waiting for the number of deserialized messages in the action queue 506 to decrease, but is unable to decrease the number of deserialized messages in the action queue 506 because it is unable to add a reference to action queue 506 to the worker queue 520, and therefore the second worker thread 522b is spinning, waiting for the number of queue references in the worker queue 520 to decrease.Similarly, the first worker thread 522a is spinning and is therefore unable to decrease the number of queue references in the worker queue 520. The processing of messages reaches a deadlock and cannot continue.

[0063] Because the deadlock prevents further processing of messages, protocol messages such as "heartbeat" messages are not processed. If more than a certain number of protocol messages between the client device and the host are not processed, or if the interval between protocol message exchanges between the client device and the host exceeds a certain time, either the host or the client device, or both, may determine that the other is no longer participating in the session and terminate the session. This means that messages sent by either the client device or the host, or both, may be lost.

[0064] While a deadlock can be broken by restarting the connectors 204 and 304, this not only takes time and consumes resources, but also carries the risk of losing queued or in-flight messages. Therefore, deadlocks are undesirable, and eliminating the risk of deadlocks would be beneficial. Furthermore, the risk of deadlock itself is undesirable because it can impose restrictions on the code that can be used in conjunction with queues. For example, code that involves worker threads 522a and 522b adding messages to the session queue 502, or that poses such a risk, should be avoided or rewritten to mitigate the risk of a deadlock occurring between the worker queue 520 and the session queue 502. This can limit how queues can be used. It would be useful to be able to use queues, or more generally, buffers, while eliminating the risk of deadlocks.

[0065] It is understood that Figure 5 is merely an example, and that deadlock, or the risk of deadlock, may occur or exist in other example situations or systems. For example, even when a single queue is used, the risk of deadlock may exist. For example, a particular thread may process messages from a particular queue and also add messages to that particular queue. However, according to the control scheme described above, when the queue becomes full, the thread spins trying to add messages to the queue. Because the thread is in a spinning state, it cannot process any messages from the queue. Therefore, the number of messages in the queue does not decrease, and the thread continues to spin indefinitely. Logically, no progress can be made in adding or processing messages, and a deadlock condition occurs.

[0066] More broadly, even in the absence of a risk of deadlock, the above control scheme has drawbacks. For example, the control scheme affects processes that add messages to a queue, or more generally, elements to a buffer. Specifically, such processes may be forced to wait until the number of messages in the queue decreases before proceeding, which may prevent the process from progressing. Such blocking may slow down the rate at which the process executes, and consequently the overall processing performed by, for example, connectors 204 and 304. As an example, this process includes the reception of messages by connectors 204 and 304, as described above with reference to FIGS. 2 through 4. For example, the control scheme may cause connector 304 to block the reception of outbound messages 322, 332, and 410 and / or inbound messages 326, 336, and 411, which may cause instability in the functioning of the overall system 300. It would be beneficial to mitigate such drawbacks.

[0067] VI. Overflow Queue

[0068] FIG. 6 illustrates a system 600 in which an embodiment may be employed. The system 600 includes a memory 604 and a processor 602. The memory 604 may be the same as or similar to the memory 114, 208, or 308 according to any one of the embodiments described above with reference to FIGS. 1 through 4. The processor 602 may be the same as or similar to the processor 112, 206, or 306 according to any one of the embodiments described above with reference to FIGS. 1 through 4. The processor 602 and the memory 604 may be part of a server system 600. In an embodiment, the server system 600 may be configured to facilitate messaging between a host system (such as the host 210 or 310 according to any one of the embodiments described above with reference to FIGS. 1 through 4) and one or more client systems (such as the client devices 202a, 202b, 302a, or 302b according to any one of the embodiments described above with reference to FIGS. 1 through 5). For example, the server system 600 may be part of or implement the connector 204, 304 according to any one of the embodiments described above with reference to FIGS.

[0069] In system 600, memory 604 is configured to provide a standard queue 612 (an example of a first buffer) and an overflow queue 610 (an example of a second buffer). Standard queue 612 is configured to store multiple elements to be processed. For example, in a particular queue 612, an element may be one of a message, a reference to a message, and a reference to a queue, as appropriate. Overflow queue 610 is configured to store elements for addition to standard queue 612. Overflow queue 610 is related to standard queue 612 in that it is configured to store elements for addition to standard queue 612. Standard queue 612 has a fixed capacity, i.e., it has the capacity to store only a certain number of elements. In an embodiment, standard queue 612 is a ring buffer. In an embodiment, overflow queue 610 is unbounded, i.e., it can have an unlimited capacity for elements. This can ensure that overflow queue 610 does not become full. It is understood that while all memory may theoretically be limited, overflow queue 610 may be referred to as unbounded in the sense that its capacity is or can be made large enough that, in practice, overflow queue 610 is considered unbounded. That is, in some embodiments, there is practically no risk of overflow queue 610 filling up, and therefore, practically no risk of buffer overrun in overflow queue 610.

[0070] 6 may represent any one of the queues 402-420, 502, 506, and 520 described above with reference to FIGS. 1-5. That is, in an embodiment, the standard queue 612 may represent any one of the session queues 402, 404, 412, and 414, the action queues 406, 408, 416, and 418, and the worker queue 420, as described above with reference to FIGS. 1-5. For example, if the standard queue 612 represents the session queues 402, 404, 412, and 414, the elements are messages or references to messages, e.g., serialized messages. If the standard queue 612 represents the action queues 406, 408, 416, and 418, the elements are messages or references to messages, e.g., deserialized messages. When the standard queue 612 represents a worker queue 420, the elements are queue references, i.e., references to session queues 402, 404, 412, 414 and / or references to action queues 406, 408, 416, 418. While FIG. 6 shows one standard queue 612 and one overflow queue 610 for simplicity, it will be understood that in some examples, the memory 604 may be configured to provide multiple standard queues 612. For example, one standard queue 612 may be configured for each of the session queues 402, 404, 412, 414, action queues 406, 408, 416, 418, and worker queue 420 described above with reference to FIGS. 1 through 5. In this case, the memory 604 may be configured to provide multiple overflow queues 610, respectively. For example, the memory 604 may be configured to provide one overflow queue 610 for each of the standard queues 612. In this case, each overflow queue 610 may be associated with a respective standard queue 612. That is, each overflow queue 610 is for storing elements for addition to a respective standard queue 612.

[0071] As described above with reference to Figures 3 and 4, the connector 304 may establish multiple connections or sessions between the host system 310 and each of the multiple client systems 302a, 302b. In these cases, when the system 600 implements the connector 304, the memory 604 may be configured to provide a standard queue 612 and an associated overflow queue 610 for each of the multiple sessions. For example, there may be a first standard queue (e.g., session queues 402, 404 or action queues 406, 408 of Figure 4) and an associated first overflow queue for a first session, and a second standard queue (e.g., session queues 412, 414 or action queues 416, 418 of Figure 4) and an associated second overflow queue for a second session. Furthermore, as described above with reference to Figures 3 and 4, messaging between the host system 310 and the client systems 302a, 302b is bidirectional. In those cases where system 600 implements connector 304, memory 604 may be configured to provide, for each of a first messaging direction (e.g., outbound) and a second messaging direction (e.g., inbound), a standard queue 612 and an associated overflow queue 610. For example, for the first messaging direction (e.g., outbound), there may be a first standard queue (e.g., session queue 402 or action queue 406 in FIG. 4) and an associated first overflow queue, and for the second messaging direction (e.g., inbound), there may be a second standard queue (e.g., session queue 404 or action queue 408 in FIG. 4) and an associated second overflow queue.

[0072] Referring to Figure 6, processor 602 is configured to obtain an element to be processed. In some examples, processor 602 obtains the element by receiving a message from an external entity via a communication interface (such as network interface 116 of Figure 1). For example, in the case of an outbound message, the external entity may be client device 202a, 202b, 302a, 302b described above with reference to Figures 2-3. As another example, in the case of an inbound message, the external entity may be host 210, 310 described above with reference to Figures 2-3. In an embodiment, processor 602 receives the message and may determine to which session queue 402, 404, 412, 414, 502 the message (or a reference to the message) is to be added, for example, based on the session in which the message was received and / or the messaging direction of the message, as described above with reference to Figures 3-5. In such an example, the standard queue 612 may represent a particular session queue 402 , 404 , 412 , 414 , 502 , and the overflow queue 610 may represent the overflow queue 610 for and associated with that particular session queue 402 , 404 , 412 , 414 .

[0073] In some examples, the processor 602 obtains the element by removing the element from a third queue (not shown in FIG. 6 ). For example, the processor 602 may implement a worker thread (e.g., worker threads 422, 522a, 522b described above with reference to FIGS. 4 and 5 ) that removes the message from the referenced session queue 402, 404, 412, 414, 502, deserializes the message, and adds the deserialized message (or a reference thereto) to the appropriate action queue 406, 408, 416, 418, 506. This is done, for example, as described above with reference to FIGS. 4 and 5 . In these examples, the standard queue 612 may represent the action queue 406, 408, 416, 418, 506, and the third queue (not shown) may represent the session queue 402, 404, 412, 414, 502. In these examples, the overflow queue 610 may represent the overflow queue 610 associated with the action queues 406 , 408 , 416 , 418 , 506 .

[0074] In some examples, the processor 602 obtains an element by obtaining a reference to a queue, such as a reference to a session queue 402, 404, 412, 414 or a reference to an action queue 406, 408, 416, 418, 506. For example, a worker thread may add a message (or a reference to a message) to a session queue 402, 404, 412, 414 or an action queue 406, 408, 416, 418, 506, and then add a reference to the session queue 402, 404, 412, 414 or action queue 406, 408, 416, 418, 506 to a worker queue 420, 520, e.g., as described above with reference to Figures 4 and 5. In these examples, the standard queue 612 may represent the worker queue 420, 520, and the overflow queue 610 may represent the overflow queue 610 associated with the worker queue 420, 520.

[0075] Regardless of how processor 602 retrieves the element, for a particular standard queue 612 and associated overflow queue 610, processor 602 is configured to add the retrieved element to either standard queue 612 or the overflow queue 610 associated with standard queue 612. Specifically, processor 602 is configured to add the retrieved element to overflow queue 610 based on determining that the number of elements in standard queue 612 is greater than a first threshold T1. Processor 602 is configured to add the retrieved element to standard queue 612 based, at least in part, on determining that the number of elements in standard queue 612 is less than the first threshold T1. Thus, even if the number of elements in standard queue 612 exceeds the first threshold, processor 602 (or its worker threads) can still add the element to the queue, thereby avoiding process blocking and eliminating the risk of deadlock associated with standard queue 612.

[0076] The first threshold T1 is less than the element capacity of the standard queue 612. In particular, the first threshold T1 may be set for the standard queue 612 to avoid overrunning the standard queue 612. That is, the first threshold T1 may be set to the number of elements at which the standard queue 612 reaches or approaches its element capacity. For example, the standard queue 612 may be implemented as a ring buffer with a fixed number of element slots. Each element slot may have a capacity for only one element. Thus, the ring buffer may have a capacity for a fixed number of elements. For example, if the elements are messages and the ring buffer holds messages, the ring buffer may have a capacity for a fixed number of messages. If the elements are references to messages and the ring buffer holds references to messages, the ring buffer may have a capacity for a fixed number of references to messages. If the elements are references to queues and the ring buffer holds references to queues, the ring buffer may have a capacity for a fixed number of references to queues. The first threshold may be set to be less than the total number of elements for which the standard queue 612 has a capacity. This may ensure that if the standard queue 612 reaches or approaches its capacity, new elements are instead added to the overflow queue 610, reducing the risk of overrun. In an embodiment, the processor 602 may implement a specific number of worker threads (e.g., worker threads 422) that can be added to the standard queue 612. In an embodiment, the first threshold T1 may be at most the total number of elements that the standard queue 612 has capacity for minus the number of worker threads. This may ensure a maximum value for the first threshold while ensuring that the first buffer does not overrun. For example, each worker thread adds one element to the standard queue 612. If all of the total number of worker threads were to add elements to the standard queue 612 at the same time, the first buffer would not overrun, but the first threshold would be exceeded for any additional elements, and the processor would add the additional elements to the overflow queue 610.

[0077] For elements added to the standard queue 612, the processor 602 (e.g., its worker thread) may remove the element from the standard queue 612 for processing. For example, if the standard queue 612 is a session queue 402, 404, 412, 414, the processor 602 may remove the message from the standard queue 612, deserialize the message, and add the deserialized message (or a reference thereto) to the appropriate action queue 406, 408, 416, 418, 506 (associated overflow queues may be implemented in a similar manner). As another example, if the standard queue 612 is a worker queue 420, 520, the processor 602 may remove a reference to the session queue 402, 404, 412, 414 or action queue 406, 408, 416, 418, 506 from the worker queue 420, 520 and process one or more messages in the referenced queue. As another example, if the standard queue 612 is an action queue 406, 408, 416, 418, 506, the processor 602 may remove the deserialized message from the action queue 406, 408, 416, 418, 506 and perform processing on the deserialized message, for example, to convert the message into a format that the client device 202a, 202b, 302a, 302b or the host 210, 310 can understand as appropriate.

[0078] For elements added to the overflow queue 610, the processor 602 may add these messages to the standard queue 612. In an embodiment, the functionality of the processor 602 may be provided by at least one processor. In an embodiment, at the appropriate time, a worker thread (such as the pool worker thread 422 described above with reference to FIG. 4) may use the processor 602 to remove messages from the overflow queue 610 and add them to the associated standard queue 612. In this manner, the processor 602 may remove messages from multiple overflow queues 610 and add them to the appropriate associated standard queue 612. For each overflow queue 610, the processor 602 is configured to determine that the number of elements in the standard queue 612 is less than a second threshold T2, and, based on determining that the number of elements in the standard queue 612 is less than the second threshold T2, remove elements from the overflow queue 610 and add the removed elements to the standard queue 612. The second threshold T2 may be lower than the first threshold T1.

[0079] The second threshold T2 for the standard queue 612 may be set at a point where there is low or no risk of overrunning the standard queue. Once the number of elements in the standard queue 612 is sufficiently reduced (e.g., by being removed from the standard queue 612 for processing as described above), elements are removed from the overflow queue 610 (e.g., one by one) and added to the standard queue 612. This ensures that elements are processed from the standard queue 612. In an embodiment, the second threshold T2 is set to one or more elements. This ensures that when there is an element in the overflow queue 610, there is at least one element in the standard queue 612. In this manner, use of the overflow queue 610 ensures that the added latency to processing elements from the standard queue 612 is minimized. In an embodiment, the first threshold T1 may be between 80% and 98% of the capacity of the standard queue 612. In an embodiment, the second threshold T1 may be between 60% and 90% of the capacity of the standard queue 612. In a specific example, the capacity of session queues 402, 404, 412, 414 may be 1024, the first threshold T1 may be 1000 (i.e., 98% of capacity), and the second threshold T2 may be 900 (i.e., 88% of capacity). In another specific example, the capacity of action queues 406, 408, 416, 418, 506 may be 64, the first threshold T1 may be 54 (i.e., 84% of capacity), and the second threshold T2 may be 44 (i.e., 69% of capacity). Other implementations may also be used.

[0080] When an element is removed from the overflow queue 610 and added to the associated standard queue 612, the element may be processed from the standard queue 612, for example, as described above.

[0081] In an embodiment, processor 602 is configured to add the retrieved element to overflow queue 610 based in part on determining that the number of elements in overflow queue 610 is greater than zero. Processor 602 may be configured to add the retrieved element to standard queue 612 based in part on determining that the number of elements in overflow queue 610 is zero, thereby ensuring that elements are processed in the order in which they were retrieved. For example, if there are already waiting elements in overflow queue 610, processor 602 may not add the newly retrieved element to standard queue 612 because this would cause the newly retrieved element to be out of order with the elements already in overflow queue 610. Instead, processor 602 may add the newly retrieved element to overflow queue 610.

[0082] 7, a block diagram of decisions the processor 602 may make when a message is retrieved for a particular standard queue 612 is shown, according to one embodiment. In block 702, the processor 602 determines whether the number of elements in the standard queue 612 is greater than a first threshold T1. If the number of elements in the standard queue 612 is greater than the first threshold T1, the processor 602 adds an element to the overflow queue 610 associated with the standard queue 612, according to block 708. If the number of elements in the standard queue 612 is not greater than the first threshold T1, the processor 602 determines whether the number of elements in the overflow queue 610 is greater than zero, according to block 704. If the number of elements in the overflow queue 610 is greater than zero, the processor 602 adds an element to the overflow queue, according to block 708. If the number of elements in the overflow queue 610 is zero, the processor adds an element to the standard queue 612, according to block 706. As mentioned above, this means that overruns of the standard queue 612 are avoided, the risks of blocking or deadlock associated with standard queues are eliminated, and elements are processed in the order in which they are retrieved.

[0083] 8, a block diagram of decisions that the processor 602 may make regarding elements in the overflow queue 610, according to one embodiment, is shown. For a particular overflow queue 610 that contains an element, the processor 602 (specifically its worker thread) determines, at block 802, whether the number of elements in the associated standard queue 612 is less than a second threshold T2. If the number of elements in the associated standard queue 612 is less than the second threshold T2, the processor 602 removes an element from the overflow queue 610 and adds the removed element to the associated standard queue 612, according to block 806. If the number of elements in the associated standard queue 612 is greater than or equal to the second threshold T2, the processor 602 does not add the element from the overflow queue 610 to the standard queue 612 and returns to block 802 to again determine whether the number of elements in the associated standard queue 612 is less than the second threshold T2.

[0084] Referring again to FIG. 6 , in an embodiment, the processor 602's determination may be made with reference to an overflow flag 614. The overflow flag 614 is stored in memory 604. The overflow flag 614 may be either set (e.g., set to either true or false) or reset (e.g., set to the other of true or false). The overflow flag 614 is associated with the standard queue 612 and the overflow queue 610. In an embodiment where there are multiple standard queues 612 and associated overflow queues 610 (not shown in FIG. 6 ), each standard queue 612 and associated overflow queue 610 may have an associated overflow flag 614. The overflow flag 614 may be used by the processor 602 to efficiently determine whether to add the retrieved message to the standard queue 612 or the overflow queue 610.

[0085] Specifically, in an embodiment, processor 602 (specifically, its worker thread) may set overflow flag 614 if the number of elements in standard queue 612 is greater than a first threshold T1. By determining that overflow flag 614 is set, processor 602 may determine that the number of elements in standard queue 612 is greater than the first threshold T1. In this manner, an efficient method of implementing determining that the number of elements in standard queue 612 is greater than the first threshold T1 is provided. If processor 602 tests overflow flag 614 and it is set, processor 602 may add the retrieved message directly to overflow queue 610. This may be more computationally efficient than, for example, counting the number of messages in standard queue 612 and comparing it to the first threshold T1 for every message. On the other hand, processor 602 may reset the overflow flag if it is determined that overflow queue 610 is empty (i.e., the number of elements in overflow queue 610 is zero). By determining that overflow flag 614 is reset, processor 602 may determine that overflow queue 610 is empty (i.e., the number of elements in overflow queue 610 is zero) and that the number of elements in standard queue 612 is less than first threshold T1. This allows processor 602 to efficiently determine that the number of messages in standard queue 612 is less than first threshold T1 and that overflow queue 610 is empty (i.e., the number of elements in overflow queue 610 is zero). When processor 602 tests overflow flag 614 and it is reset, processor 602 may add the retrieved element to standard queue 612. This may be more computationally efficient than, for example, individually and / or explicitly making this determination for every element.

[0086] As an example, a particular overflow flag 614 is associated with a particular standard queue 612 and overflow queue 610. The processor 602 (specifically, its worker thread) may obtain an element and check the overflow flag 614 to determine whether it is set. If the overflow flag 614 is set (e.g., set to true), the processor 602 may add the element to the overflow queue 610. If the overflow flag 614 is reset (e.g., set to false), the processor 602 may check a counter indicating the number of elements currently in the standard queue 612 and compare this number to a first threshold T1. If the processor 602 determines that the number of elements does not exceed the first threshold T1, the processor may add the element to the standard queue 612 and increment the counter by one. If processor 602 determines that the number of elements is equal to first threshold T1, processor 602 may add the element to standard queue 612, increment a counter by one, and set overflow flag 614. At this point, processor 602 may add more retrieved elements directly to overflow queue 610 because overflow flag 614 is set. This may continue until overflow flag 614 is again reset (e.g., set to false). Specifically, as described above, processor 602 (and more specifically, its worker thread) waits until the number of elements in standard queue 612 is less than second threshold T2, at which point processor 602 removes an element from overflow queue 610 and adds the removed element to standard queue 612. This continues until processor 602 removes the last element from overflow queue 610. For example, processor 602 may increment a counter of overflow queue 610 by one each time an element is added to overflow queue 610 and may decrement the counter by one each time a message is removed from overflow queue 610.If processor 602 determines that the number of elements in overflow queue 610 is zero, processor 602 resets overflow flag 614 (e.g., sets overflow flag 614 to false). Processor 602 (specifically, its worker thread) adds more retrieved elements to standard queue 612 until the number of elements in standard queue 612 again exceeds first threshold T1, at which point overflow flag 614 is again set.

[0087] In some situations, there is a risk that between the time one worker thread checks the overflow flag 614 and adds an element to the queue, another worker thread may set or reset the flag. For example, between the time one worker thread of the processor 602 checks the overflow flag 614, determines that it is set, and adds an element to the overflow queue 610, another worker thread may reset the overflow flag. This risks sending the element to the wrong queue. In some examples, locking can be used. For example, the processor 602 (specifically, its worker thread) may check the overflow flag 614, and if it is set (e.g., set to true), the processor 602 may lock the overflow flag 614 and check the locked overflow flag 614. If the locked overflow flag 614 is set (e.g., set to true), the processor 602 may add the retrieved element to the overflow queue 610. If the locked overflow flag 614 is reset (e.g., set to false), the processor may add the retrieved element to the standard queue 612. If the overflow flag 614 is reset (e.g., set to false), the processor 602 does not lock the overflow flag 614 but instead adds the retrieved element directly to the standard queue. Thus, during "normal" operation (i.e., when an element is added directly to the standard queue 612), no locking is used and checking the overflow flag 614 during "normal" operation can be computationally inexpensive. Because "normal" operation can account for a large portion of the overall operation of the processor 602, this can ensure that the correct overflow flag 614 value is used while making the overall operation of the processor relatively computationally inexpensive.

[0088] VII. Variations

[0089] The system 600, its components, and the operation of the components described above with reference to Figures 1-6 relate to an example embodiment. In other embodiments, variations on the above-described embodiment may be made.

[0090] For example, while in the above embodiments the system 600 of Figure 6 is described as implementing the connectors 204, 304 of Figures 2 and 3, it will be appreciated that this is not necessarily the case and that in other embodiments the system 600 may be used in other contexts. For example, the system 600 may be used with any standard queue 612, or more generally a buffer, such as those used in computing devices.

[0091] As another example, while the above embodiments refer to a standard queue 612 and / or an overflow queue 610, it will be understood that these are not necessarily queues, but may more generally be first and second buffers, respectively, which may be portions of memory used to hold elements to be processed.

[0092] As another example, it is understood that the elements retrieved by the processor 602 and stored in the queues 610, 612 do not necessarily have to be messages, but may more generally be elements such as data elements. For example, for a particular first buffer, the elements may be messages, references to messages, references to queues of one or more messages, and references to queues of references to one or more messages. It is further understood that the elements do not necessarily have to be related to messages, but may in some examples be any elements, such as data elements. For example, the first buffer may store a number of elements and have a capacity for a certain number of elements. For example, the buffer may have a certain number of spaces, with one element occupying one space in the buffer. For example, each element may have a fixed size. It is understood that other variations are possible, and in some examples, the system 600 may include more or fewer components than those described in the above examples.

[0093] Thus, in an embodiment, it is understood that a system 600 is provided that includes: a memory 604 configured to provide: a first buffer 612 that stores a number of elements to be processed; and a second buffer 610 that stores elements to be added to the first buffer 612; and a processor configured to: retrieve the elements to be processed; determine that the number of elements in the first buffer 612 is greater than a first threshold T1; and add the retrieved elements to the second buffer 610.

[0094] VIII. Exemplary Methods

[0095] Referring to FIG. 9 , a flow diagram illustrating a method according to one embodiment is shown. The method may be performed by a server system, such as system 600 according to any of the embodiments described above with reference to FIGS. 1-8 . For example, the method may be performed by connectors 204, 304 according to any of the embodiments described above with reference to FIGS. 1-8 . In an embodiment, the server system includes at least one processor (e.g., processor 112, 206, 306, 602 according to any of the embodiments described above with reference to FIGS. 1-8 ) and memory (e.g., memory 114, 208, 308, 604 according to any of the embodiments described above with reference to FIGS. 1-8 ). Aspects of the method may be performed by the at least one processor. The at least one processor may be configured to execute aspects of the method. The memory may store instructions that, when executed, cause the at least one processor to perform aspects of the method or functions defined by the method. For example, the memory may store an application (e.g., application 130 of FIG. 1 ) including instructions.

[0096] The method includes providing a first buffer in memory for storing a number of elements to be processed, in step 902. For example, according to any of the embodiments described above with reference to Figures 1-8, the memory may be memory 604, the first buffer may be standard queue 612, and the elements may be messages.

[0097] The method includes providing a second buffer in memory for storing elements to be added to the first buffer, in step 904. For example, the second buffer may be an overflow queue 610 according to any of the embodiments described above with reference to Figures 1-8.

[0098] The method includes, in step 906, obtaining, by at least one processor, elements to be processed. For example, the at least one processor may include processor 602 according to any of the embodiments described above with reference to Figures 1-8.

[0099] The method includes determining, by the at least one processor, that the number of elements in the first buffer is greater than a first threshold value in step 908. For example, the first threshold value may be a first threshold value T1 according to any of the embodiments described above with reference to Figures 1 through 8. The method includes, at step 910, adding, by the at least one processor, the retrieved elements to a second buffer, i.e., in response to determining that the number of elements in the first buffer is greater than a first threshold, adding the retrieved elements to the second buffer.

[0100] IX. Illustrative Electronic Trading System

[0101] FIG. 10 shows a block diagram depicting an example electronic trading system 1000 in which certain embodiments may be employed. The system 1000 includes a trading device 1010, a gateway 1020, and an exchange 1030. The trading device 1010 communicates with the gateway 1020. The gateway 1020 communicates with the exchange 1030. As used in this disclosure, the term "communicate" encompasses direct communication and / or indirect communication via one or more intermediary components. The trading device 1010, the gateway 1020, and / or the exchange 1030 may include one or more computing devices 100 of FIG. 1. The example electronic trading system 1000 shown in FIG. 10 may communicate with additional components, subsystems, and elements that provide additional functionality and capabilities without departing from the teachings and teachings presented in this disclosure.

[0102] During operation, the trading device 1010 may receive market data from the exchange 1030 via the gateway 1020. The trading device 1010 may send messages to the exchange 1030 via the gateway 1020. A user may utilize the trading device 1010 to monitor market data and make decisions to send order messages to buy or sell one or more tradable objects to the exchange 1030. The trading device 1010 may use the market data to perform trading actions, such as sending order messages to the exchange 1030. For example, the trading device may execute an algorithm that uses the market data as input and outputs a trading action, such as sending an order message to the exchange 1030. The algorithm may or may not require input from a user to perform a trading action.

[0103] Market data may include data about the market for a tradable object. For example, market data may include the inside market, market depth, last traded price ("LTP"), last traded quantity ("LTQ"), or a combination of these. The inside market refers to the highest bid (best bid) and lowest ask (best ask or best offer) prices in the market for a tradable object at a particular time (as the inside market may change over time). Market depth refers to the quantity available at price levels including and beyond the inside market. There may be "gaps" in the market depth due to prices with no quantity based on orders in the market.

[0104] Price levels associated with the inside market and market depth may be provided as value levels that include not only the price itself but also derived and / or calculated representations. For example, a value level may be displayed as the net change from the opening price. As another example, a value level may be presented as a value calculated from the prices of two other markets. As another example, a value level may include an integrated price level.

[0105] A tradable object is anything that can be traded. For example, a certain amount of a tradable object can be bought or sold at a particular price. Tradable objects may include, for example, financial instruments, stocks, options, bonds, futures contracts, currencies, warrants, fund-related derivatives, securities, commodities, swaps, interest rate-related products, index-linked products, traded events, goods, or combinations thereof. Tradable objects may include instruments listed and / or managed by an exchange, instruments defined by a user, a combination of real or synthetic instruments, or any combination thereof. Synthetic tradable objects may exist that correspond to and / or are similar to real tradable objects.

[0106] An order message is a message that includes a trade order, which may be, for example, a command to place an order to buy or sell a tradable object, a command to initiate management of an order according to a defined trading strategy, a command to change, modify, or cancel an order, instructions to an electronic exchange related to the order, or a combination thereof.

[0107] The trading device 1010 may include one or more electronic computing platforms. For example, the trading device 1010 may include a desktop computer, a handheld device, a laptop, a server, a portable computing device, a trading terminal, an embedded trading system, a workstation, an algorithmic trading system such as a "black box" or "gray box" system, a computer cluster, or a combination thereof. As another example, the trading device 1010 may include a single or multi-core processor in communication with memory or other storage media configured to accessibly store one or more computer programs, applications, libraries, computer-readable instructions, etc., for execution by the processor.

[0108] By way of example, trading device 1010 may include a computing device, such as a personal computer or mobile device, in communication with one or more servers, where the computing device and one or more servers together constitute trading device 1010. For example, trading device 1010 may be a computing device and one or more servers that jointly execute the TT® Platform, an electronic trading platform offered by Trading Technologies International, Inc. (hereinafter, "Trading Technologies") of Chicago, Illinois. For example, one or more servers may execute a portion of the TT Platform, such as a portion that provides a web server, and a computing device may execute another portion of the TT Platform, such as a portion that provides user interface functionality on a web browser. The computing device and server may communicate with each other to implement the TT Platform, for example, using browser session requests and responses or web sockets. As another example, trading device 1010 may include a computing device, such as a personal computer or mobile device, that executes an application, such as TT® Desktop or TT® Mobile, both of which are electronic trading applications offered by Trading Technologies. As another example, trading device 1010 may be one or more servers running trading tools such as ADL®, AUTOSPREADER®, AUTOTRADER®, and / or MD TRADER®, also provided by Trading Technologies.

[0109] Trading device 1010 is controlled or otherwise used by a user. As used in this disclosure, the term "user" may include, but is not limited to, a human being (e.g., a trader), a trading group (e.g., a group of traders), or an electronic trading device (e.g., an algorithmic trading system). One or more users may be involved in controlling or otherwise using a trading device.

[0110] The trading device 1010 may include one or more trading applications. As used in this disclosure, a trading application is an application that facilitates or enhances electronic trading. A trading application provides one or more electronic trading tools. For example, a stored trading application may be executed by the trading device to arrange and display market data in one or more trading windows. In another example, a trading application may include an automated spread trading application that provides spread trading tools. In yet another example, a trading application may include an algorithmic trading application that automatically processes algorithms and performs specific actions, such as placing orders, modifying existing orders, or deleting orders. In yet another example, a trading application may provide one or more trading screens. A trading screen may provide one or more trading tools that enable interaction with one or more markets. For example, trading tools may enable a user to obtain and display market data, set order entry parameters, send order messages to an exchange, deploy trading algorithms, and / or monitor positions while implementing various trading strategies. The electronic trading tools provided by a trading application may be available all the time, or may be available only in a particular configuration or mode of operation of the trading application.

[0111] The trading application may be implemented using computer-readable instructions stored on a computer-readable medium and executable by a processor. The computer-readable medium may include various types of volatile and non-volatile storage media, such as, for example, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, any combination thereof, or other tangible data storage devices. As used in this disclosure, the terms "non-transitory" or "tangible" computer-readable medium are expressly defined to include all types of computer-readable storage media and to exclude propagating signals.

[0112] One or more components or modules of the trading application may be loaded into the computer-readable medium of the trading device 1010 from another computer-readable medium. For example, the trading application (or an update to the trading application) may be stored on one or more CDs, DVDs, or USB drives by a manufacturer, developer, or publisher and then loaded onto the trading device 1010, or loaded onto a server from which the trading device 1010 retrieves the trading application. As another example, the trading device 1010 may receive the trading application (or an update to the trading application) from a server, e.g., via the Internet or an internal network. The trading device 1010 may receive the trading application or update when requested by the trading device 1010 (e.g., "pull delivery") and / or when not requested by the trading device 1010 (e.g., "push delivery").

[0113] The trading device 1010 may be adapted to send order messages. For example, the order messages may be sent to the exchange 1030 via the gateway 1020. As another example, the trading device 1010 may be adapted to send order messages to a simulated exchange in a simulated environment that does not implement real-world transactions.

[0114] The order message may be transmitted at the request of a user. For example, a trader may utilize the trading device 1010 to transmit the order message or may manually enter one or more parameters for a trade order (e.g., order price and / or quantity). As another example, an automated trading tool provided by a trading application may calculate one or more parameters of a trade order and automatically transmit the order message. In some cases, the automated trading tool may prepare the order message to be transmitted but not actually transmit it without confirmation from the user.

[0115] The order message may be transmitted in one or more data packets or via a shared memory system. For example, the order message may be transmitted from the trading device 1010 to the exchange 1030 via the gateway 1020. The trading device 1010 may communicate with the gateway 1020 using a local area network, a wide area network, a multicast network, a wireless network, a virtual private network, an internal network, a cellular network, a peer-to-peer network, a point of presence, a dedicated line, the Internet, a shared memory system, and / or a proprietary network.

[0116] Gateway 1020 may include one or more electronic computing platforms. For example, gateway 1020 may be implemented as one or more desktop computers, handheld devices, laptops, servers, portable computing devices, trading terminals, embedded trading systems, workstations with single or multi-core processors, algorithmic trading systems such as "black box" or "gray box" systems, computer clusters, or any combination thereof.

[0117] The gateway 1020 facilitates communications. For example, the gateway 1020 may perform protocol conversion for data communicated between the trading device 1010 and the exchange 1030. The gateway 1020 may, for example, process order messages received from the trading device 1010 into a data format understandable by the exchange 1030. Similarly, the gateway 1020 may transform market data received from, for example, the exchange 1030 in an exchange-specific format into a format understandable by the trading device 1010. As described in more detail below with reference to FIG. 11 , in some examples, the gateway 1020 may communicate with cloud services that may support the functionality of the gateway 1020 and / or the trading device 1010.

[0118] The gateway 1020 may include trading applications similar to those described above that facilitate or improve electronic trading. For example, the gateway 1020 may include a trading application that tracks orders from the trading devices 1010 and updates the status of the orders based on fill confirmations received from the exchange 1030. As another example, the gateway 1020 may include a trading application that aggregates market data from the exchange 1030 and provides it to the trading device 1010. In yet another example, the gateway 1020 may include a trading application that provides risk processing, calculates implieds, handles order processing, handles market data processing, or any combination thereof.

[0119] In particular embodiments, the gateway 1020 communicates with the exchange 1030 using a local area network, a wide area network, a multicast network, a wireless network, a virtual private network, an internal network, a cellular network, a peer-to-peer network, a point of presence, a dedicated line, the Internet, a shared memory system, and / or a proprietary network.

[0120] The exchange 1030 may be owned, operated, controlled, or used by an exchange entity. Exemplary exchange entities include CME Group, Chicago Board Options Exchange, Intercontinental Exchange, and Singapore Exchange. The exchange 1030 may be an electronic exchange including an electronic matching system, such as a computer, server, or other computing device, adapted to, for example, facilitate the buying and selling of tradable objects offered for trading by the exchange. The exchange 1030 may include, for example, an entity that lists and / or manages tradable objects and a separate entity that receives and matches orders. The exchange 1030 may include, for example, an electronic communications network ("ECN").

[0121] The exchange 1030 is adapted to receive order messages and match bilateral trade orders to buy or sell tradeable objects. Unmatched trade orders may be posted by the exchange 1030 as trade objects. Once an order to buy or sell a tradeable object is received and confirmed by the exchange, the order is considered a working order until it is executed or canceled. If only a portion of the order quantity is matched, the partially executed order remains a working order. Trade orders may include, for example, trade orders received from the trading device 1010 or other devices in communication with the exchange 1030. For example, the exchange 1030 typically communicates with a variety of other trading devices (which may be similar to the trading device 1010) that provide trade orders to be matched.

[0122] The exchange 1030 is adapted to provide market data. The market data may be provided in one or more messages or data packets, or via a shared memory system. For example, the exchange 1030 may publish a data feed to subscribing devices, such as trading devices 1010 and gateways 1020. The data feed may include market data.

[0123] System 1000 may include additional, different, or fewer components. For example, system 1000 may include multiple trading devices, gateways, and / or exchanges. In another example, system 1000 may include other communication devices, such as middleware, firewalls, hubs, switches, routers, servers, exchange-specific communication equipment, modems, security managers, and / or encryption / decryption devices.

[0124] In an embodiment, trading device 1010 may be provided by client device 202a, 202b, 302a, 302b according to any of the embodiments described above with reference to Figures 1-9. For example, trading device 1010 may be configured to provide the same or similar functionality as client device 202a, 202b, 302a, 302b according to any of the embodiments described above with reference to Figures 1-9. In an embodiment, exchange 1030 may be provided by host 210, 310 according to any of the embodiments described above with reference to Figures 1-9. For example, exchange 1030 may be configured to provide the same or similar functionality as host 210, 310 according to any of the embodiments described above with reference to Figures 1-9. In an embodiment, gateway 1020 may be provided by or include system 600 and / or connectors 204, 304 according to any of the embodiments described above with reference to Figures 1-9. For example, the gateway 1020 may be configured to provide the same or similar functionality as the system 600 and / or connectors 204, 304 according to any of the embodiments described above with reference to FIGS.

[0125] In embodiments, the messages referenced above with reference to Figures 1-9 may include trade orders sent from trading device 1010 to exchange 1030 via gateway 1020. For example, gateway 1020 may receive (outbound) trade orders from trading device 1010, and gateway 1020 may add the trade orders to a standard queue or an overflow queue, e.g., as described above with reference to Figures 1-9. Gateway 1020 may process the trade orders by converting them into a format understandable by exchange 1030.

[0126] In embodiments, the messages referenced above with reference to Figures 1-9 may include an execution report sent from the exchange 1030 to the trading device 1010 via the gateway 1020 when a trade order is executed at the exchange 1030. For example, the gateway 1020 may receive an (inbound) execution report from the exchange 1030 and add the execution report to a standard queue or an overflow queue, e.g., as described above with reference to Figures 1-9. The gateway 1020 may process the execution report by converting it into a format understandable by the trading device 1010.

[0127] In some examples, the actions performed by the host referenced above with reference to FIGS. 1 through 9 may include the exchange 1030 performing an action with respect to a particular working order. For example, the trading device 1010 may send a trade order to the exchange 1030 via the gateway 1020. The trade order may include an order ID that indicates the particular working order with which the trade order is associated. All trade orders from the trading device 1010 that are associated with the same working order may include the same order ID. The exchange 1030 executes the received trade order (e.g., placing, modifying, and / or canceling the order). The exchange 1030 may then send an execution report back to the trading device 1010 via the gateway 1020. For example, the execution report may detail the results of the trade order. Each execution report may also include an order ID that indicates the particular working order with which the execution report is associated. In these examples, each of the action queues 406, 408, 416, 418 referenced above with reference to Figures 1-9 may each be an order queue. That is, each order queue 406, 408, 416, 418 is associated with a particular working order, and messages in a particular order queue 406, 408, 416, 418 may all have the same order ID. It is important that trade orders associated with a particular working order are processed in the order they are received by the gateway 1020. If the orders are received out of order at the exchange 1030, the exchange 1030 may not correctly execute the trade orders.

[0128] In embodiments, the messages referenced above with reference to Figures 1 through 9 may include protocol messages between the trading device 1010 and the exchange 1030. For example, the trading device 1010 (or applications running thereon) and the exchange 1030 may communicate with each other using Financial Information Exchange (FIX) messages. Some of these messages may be related to establishing or maintaining a FIX connection or session. For example, the messages may include heartbeat messages sent from the exchange 1030 to the trading device 1010 (and / or from the trading device 1010 to the exchange 1030) to confirm that the trading device 1010 (and / or the exchange 1030) is still on the connection and therefore should maintain the connection.

[0129] X. Examples of Electronic Trading Systems

[0130] FIG. 11 shows a block diagram illustrating an example electronic trading system 1100 in which certain embodiments may be employed. Electronic trading system 1100 includes trading device 1110, hybrid cloud system 1120, and exchange 1130. Trading device 1110 may be the same as or similar to trading device 1010 described above with reference to FIG. 10. Exchange 1130 may be the same as or similar to exchange 1030 described above with reference to FIG. 10. Hybrid cloud system 1120, or one or more components thereof, may provide one or more functions of gateway 1020 described above with reference to FIG. 10. That is, the functionality of gateway 1020 described above with reference to FIG. 10, or one or more portions of its functionality, may be included within hybrid cloud system 1120.

[0131] Hybrid cloud system 1120 includes cloud service 1140 and data center 1160. In the example shown in FIG. 11 , cloud service 1140 and its components are separate from data center 1160. However, in other examples (not shown), one or more or all of the components and / or functionality of cloud service 1140 may instead be implemented in data center 1160. In such examples, or in other cases, electronic trading system 1100 may not include cloud service 1140. In such examples, one or more functions of gateway 1020 described above with reference to FIG. 10 may be provided solely by data center 1160, or one or more components thereof.

[0132] To maintain low latency for time-sensitive processing, data center 1160 may be co-located with exchange 1130 or located near exchange 1130. Thus, functions of hybrid cloud system 1120 that are time-sensitive or that benefit from low latency with exchange 1130 may be performed by data center 1160. In general, functions of hybrid cloud system 1120 that are not time-sensitive or that do not benefit from low latency with exchange 1130 may be performed by cloud services 1140. Hybrid cloud system 1120 allows for scalability of electronic trading system 1100 for non-time-sensitive functions while also achieving relatively low latency with exchange 1130.

[0133] In the example of FIG. 11 , trading device 1110 communicates with cloud service 1140 via a first network 1171. For example, first network 1171 may be a wide area network such as the Internet using a Hypertext Transfer Protocol (HTTP) connection. Trading device 1110 communicates with data center 1160 via a second network 1172. For example, trading device 1110 may communicate with data center 1160 via a virtual private network or using a secure WebSocket or TCP connection. First network 1171 and second network 1172 may be the same network. Data center 1160 communicates with cloud service 1140 via a third network 1173. For example, data center 1160 may communicate with cloud service 1140 via a private network or a virtual private network (VPN) tunnel. Third network 1173 may be the same as first network 1171 and / or second network 1172. The data center 1160 communicates with the exchange 1130 via a fourth network 1174. For example, the data center 1160 may communicate with the exchange 1130 using a local area network, a wide area network, a multicast network, a wireless network, a virtual private network, an internal network, a cellular network, a peer-to-peer network, a point of presence, a dedicated line, the Internet, a shared memory system, and / or a proprietary network. The fourth network 1174 may be the same as the first network 1171, the second network 1172, and / or the third network 1173.

[0134] Cloud services 1140 may be implemented as a virtual private cloud, which may be provided by logically isolated sections of the overall web services cloud. In this example, cloud services 1140 includes web database 1141 and associated web server 1142, product database 1143 and associated product data server (PDS) 1144, user setup database 1145 and associated user setup server 1146, and transaction database 1147 and associated transaction server 1148.

[0135] The trading device 1110 may communicate with a web server 1142. As one example, the trading device 1110 may execute a web browser, referred to herein as a browser, and establish a browsing session with the web server 1142. This may occur after proper domain name resolution to the IP address of the cloud service 1140 and / or proper authentication of the trading device 1110 (or its user) by the cloud service 1140. The browser sends requests to the web server 1142, which returns responses to the browser using, for example, Hypertext Transfer Protocol (HTTP) or Secure Hypertext Transfer Protocol (HTTPS) protocols. The web server 1142 provides a user interface to the browser through which a user can interact with the electronic trading platform. This user interface may enable the display of market data and / or the submission of trade orders. As another example, the trading device 1110 may execute an application that communicates with the web server 1142, such as via an application programming interface (API), to enable a user to interact with the electronic trading platform. This application may provide a user interface through which a user can interact with the electronic trading platform.

[0136] The trading device 1110 may communicate with the PDS 1144. The PDS 1144 interfaces with a product database 1143. The product database 1143 stores instrument definitions and user permissions for the instruments. Specifically, the product database 1143 stores definitions of tradable objects and user permissions to place trade orders for the tradable objects. This information may be provided to the trading device 1110. This information may be used by the user interface of the trading device 1110 to determine which tradable objects a given user of the trading device 1110 may place trade orders for.

[0137] The trading device 1110 may communicate with a user setup server 1146. The user setup server 1146 interfaces with a user setup database 1145. The user setup database 1145 stores user settings, preferences, and other information associated with a user's account. This information may be provided by the trading device 1110 to the user setup server 1146 upon user registration or at some point thereafter, and the user setup server 1146 may store this information in the user setup database 1145. This information may be provided to the trading device 1110. This information may be used by the user interface of the trading device 1110 to determine what market data to display and in what format.

[0138] Transaction database 1147 stores information about transactions conducted using electronic trading system 1100. Transaction database 1147 may store all of the trade orders submitted by users and all of the corresponding order execution reports provided by exchange 1130 upon execution of the trade orders. Transaction server 1148 may query transaction database 1147 and generate, for example, an audit trail 1149 for a given user. This audit trail 1149 may be provided to trading device 1110 (or other device) so that the trading activity of a given user can be reviewed and / or analyzed.

[0139] Data center 1160 includes a multicast bus 1161, a price server 1162, an edge server 1163, a risk server 1164, a ledger uploader server 1165, an order connector 1166, and a strategy engine server 1167. The various components within data center 1160 communicate with each other using multicast bus 1161. This allows for efficient and scalable communication between components within data center 1160. For example, information provided by one of the components may be received by multiple other components. By sending this information on the multicast bus 1161 to which the other components subscribe, the information can be sent in a single message regardless of the number of receiving components.

[0140] The price server 1162 receives market data from the exchange 1130. The price server 1162 converts this information into a format and / or syntax associated with (e.g., used by) the electronic trading system 1100. The price server 1162 transmits the converted information as one or more multicast messages on the multicast bus 1161. Specifically, the price server 1162 multicasts this information to a first multicast bus A, where it is received by the price clients. The edge server 1163 and the strategy engine server 1167 subscribe to the first multicast bus A and receive market data from the price server 1162. The price server 1162 may communicate with the cloud service 1140. For example, the price server 1162 may provide information about products or tradable objects to the PDS server 1144, which may use the information in defining the tradable objects.

[0141] The edge server 1163 communicates with the trading devices 1110. For example, the trading devices 1110 may communicate with the edge server 1163 via a secure WebSocket or TCP connection. In some examples, the edge server 1163 may be implemented as a server cluster. The number of servers in the cluster may be determined and expanded as needed based on utilization. The edge server 1163 receives market data via a first multicast bus A and routes the market data to the trading devices 1110. A user of the trading device 1110 may decide to place a trade order based on the market data. The edge server 1163 routes the trade order from the trading device 1110 to the exchange 1130. Specifically, when the edge server 1163 receives an order message from the trading device 1110, it multicasts the order message (or at least a portion of its contents) to a second multicast bus B for receipt by the order client. The risk server 1164 subscribes to the second multicast bus B and receives order messages from the edge server 1163 .

[0142] The risk server 1164 is used to determine the pre-trade risk for a given trade order contained in a given order message. For example, for a given trade order, the risk server 1164 may determine whether the user submitting the trade order is authorized to submit the trade order. The risk server 1164 may determine whether the user is authorized to trade the quantity of the tradable object specified in the trade order. The risk server 1164 may prevent the submission of fraudulent trade orders. The risk server 1164 receives order messages from the edge server 1163 via the second multicast bus B and processes the order messages to determine the risk for the trade order in the message. If the risk server 1164 determines that the trade order should not be submitted (e.g., if the risk associated with the trade order exceeds a threshold), the risk server 1164 prevents the submission of the trade order. For example, in this case, the risk server 1164 does not send the order message to the order connector 1166 but instead sends a message to the user indicating that the trade order was not submitted. If the risk server 1164 determines that the trade order should be submitted (e.g., if the risk associated with the trade order is below a threshold), the risk server 1164 forwards the order message to the order connector 1166. Specifically, the risk server 1164 multicasts the order message to a second multicast bus B. The order connector 1166 and the ledger uploader 1165 subscribe to the second multicast bus B and receive the order message from the risk server 1164.

[0143] The ledger uploader server 1165 is in communication with the transaction database 1147 of the cloud service 1140. The ledger uploader server 1165 receives order messages from the risk server 1164 and sends the order messages to the transaction database 1147. The transaction database 1147 then stores the order messages (or at least a portion of their contents) in a ledger stored in the transaction database 1147.

[0144] The order connector 1166 communicates with the exchange 1130. The order connector 1166 receives order messages from the risk server 1164, processes the order messages for transmission to the exchange 1130, and transmits the processed order messages to the exchange 1130. Specifically, this processing includes processing the order messages into a data format understandable by the exchange 1130. If the trade order in the order message is executed by the exchange 1130, the exchange 1130 sends a corresponding execution report message to the order connector 1166. The execution report message includes an execution report that details the execution of the trade order. The order connector 1166 processes the execution report message. Specifically, this processing includes processing the execution report message into a data format understandable by the electronic trading system and trading device 1110. The order connector 1166 multicasts the processed execution report message to a third multicast bus C for reception by execution report clients. The edge server 1163 and the ledger uploader 1165 subscribe to the third multicast bus C to receive the processed execution report message. The ledger uploader 1165 communicates with the transaction database 1147 and updates the ledger with the execution report message (or at least a portion of its contents). The edge server 1163 forwards the execution report message to the trading device 1110. Based on the execution report message, the trading device 1110 may display information indicating that the trade order has been executed.

[0145] In some embodiments, the order message may be sent by a strategy engine server 1167. For example, the strategy engine server 1167 may implement one or more strategy engines using an algorithmic strategy engine and / or an autospreader strategy engine. The strategy engine 1167 may receive market data (from the price server 1162 via the first multicast bus A) and automatically generate order messages based on the market data and appropriately configured algorithms. The strategy engine server 1167 may send the order message to the order connector 1166 (via the risk server 1164 and the second multicast bus B), which processes the order message in the same manner as described above. Similarly, when the exchange 1130 fills an order, the strategy engine 1167 may receive a corresponding order fill report message from the order connector 1166 (via the third multicast bus C). The order message and the fill report message may be sent to the ledger uploader 1165 in the same manner as described above, which may update the ledger stored in the transaction database 1147.

[0146] In some embodiments, trade orders sent by the trading device 1110 may not be placed by a human. For example, the trading device 1110 may be a computing device implementing an algorithmic trading application. In these examples, the trading device 1110 may not communicate with the web server 1142, the PDS 1144, and / or the user setup server 1146 and may not utilize a browser or user interface to place trades. An application executing on the trading device 1110 may communicate with an adapter associated with the edge server 1163. For example, the application and the adapter may communicate with each other using Financial Information Exchange (FIX) messages. In these examples, the adapter may be a FIX adapter. An application executing on the trading device 1110 may receive market data in FIX format (the market data is provided by the price server 1162 and converted to FIX format by the FIX adapter associated with the edge server 1163). Applications executing on trading devices 1110 may generate trade orders based on the received market data and send order messages in FIX format to a FIX adapter associated with edge server 1163. The FIX adapter associated with edge server 1163 may process the order messages received in FIX format into a format understandable by components of data center 1160.

[0147] It should be noted that electronic trading system 1100 is merely an example, and other electronic trading systems may be used. As one example, electronic trading system 1100 need not necessarily include cloud service 1140. As another example, data center 1160 may include more or fewer components than those described above with reference to FIG. 11. As another example, forms of messaging between components of data center 1160 other than multicast messaging may be used.

[0148] In an embodiment, order connector 1166 may be provided by or may include system 600 and / or connectors 204, 304 of any of the embodiments described above with reference to Figures 1-9. For example, order connectors 204, 304 may be configured to provide the same or similar functionality as system 600 and / or connectors 204, 304 of any of the embodiments described above with reference to Figures 1-9. Thus, in an embodiment, server system 600 and / or connectors 204, 304 of any of the embodiments described above with reference to Figures 1-9 may implement order connector 1166 of example electronic trading system 1100 or electronic trading platform provided therewith. Trading device 1110 and / or strategy engine 1167 may be provided by and / or provide the same or similar functionality as client devices 202a, 202b, 302a, 302b of any of the embodiments described above with reference to Figures 1-9. The exchange 1130 may be provided by and / or provide the same or similar functionality as the host 210, 310 according to any of the embodiments described above with reference to Figures 1-9.

[0149] In embodiments, the order connector 1166 may receive a message, such as a trade order or a protocol message, from the trading device 1110 (or the strategy engine 1167). The order connector 1166 may then add the message to a standard queue or an overflow queue, as described above. In the case of a trade order message, the order connector 1166 may process the message, convert it to a format understandable by the exchange 1130, and send the converted message to the exchange 1130. In the case of a protocol message, such as a FIX heartbeat message, the order connector 1166 may forward the message to the exchange 1130 as needed. The exchange 1130 may process the message. For example, in the case of a trade order message, the exchange 1130 may execute the trade order. The exchange 1130 may then send an execution report back to the trading device 1110. In the case of a protocol message, the exchange 1130 may process the protocol message and, as needed, send a response protocol message back to the trading device 1110. The order connector 1166 may receive messages from the exchange 1130 and add the messages to a standard queue or an overflow queue, as described above. In the case of an execution report message, the order connector 1166 may process the message, convert it to a format understandable by the trading device 1110 (or strategy engine 1167), and send the converted message to the trading device 1110 (or strategy engine 1167). In the case of a protocol message, the order connector 1166 may forward the message to the trading device 1110 (or strategy engine 1167) as needed.

[0150] As a particular example, a trading device 1110 (or strategy engine 1167) may send a trade order message for a particular working order to an order connector 1166. The order connector 1166 may add the trade order message for the particular working order to an outbound session queue associated with the session between the trading device 1110 and the exchange 1130 (or, as appropriate, to an overflow queue associated with the outbound session queue). The order connector 1166 may then add a reference to the session queue to a worker queue (or, as appropriate, to an overflow queue associated with the worker queue). The order connector 1166 may then remove the trade order message from the referenced session queue, determine the particular working order to which the trade order message pertains (e.g., by referencing the order ID included in the message), and add the trade order message to an outbound order queue associated with the particular working order (or, as appropriate, to an overflow queue associated with the outbound order queue). The order connector 1166 may then add a reference to the order queue to the worker queue (or to an overflow queue associated with the worker queue, as appropriate). The order connector may then remove the trade order message from the referenced order queue, process and convert the trade order message into a format understood by the exchange 1130, and send the converted trade order message to the exchange 1130.

[0151] The exchange 1130 may process the trade order and, in response, send a fill report related to the particular working order. The order connector 1166 may add the fill report message to an inbound session queue associated with the session between the trading device 1110 and the exchange 1130 (or, as appropriate, an overflow queue associated with the inbound session queue). The order connector 1166 may then add a reference to the inbound session queue to a worker queue (or, as appropriate, an overflow queue associated with the worker queue). The order connector 1166 may then remove the fill report message from the referenced inbound session queue, determine the particular working order to which the fill report message relates (e.g., by referencing the order ID included in the message), and add the fill report message to an inbound order queue associated with the particular working order (or, as appropriate, an overflow queue associated with the inbound order queue). The order connector 1166 may then add a reference to the inbound order queue to the worker queue (or an overflow queue associated with the worker queue, as appropriate). The order connector 1166 may process the execution report message from the referenced inbound order queue, convert the execution report message to a format understood by the trading device 1110 (or strategy engine 1167), and send the converted execution report to the trading device 1110 (or strategy engine 1167). The trading device 1110 (or strategy engine 1167) may process the execution report appropriately.

[0152] Thus, messages on different sessions (i.e., associated with different trading devices 1110) may be processed by the order connector 1166 simultaneously. Furthermore, messages associated with different working orders (whether within the same session or across sessions) may be processed by the order connector 1166 simultaneously. This allows the order connector 1166 to process messages quickly. Furthermore, because each queue has an associated overflow queue, if a burst of messages occurs such that one of the standard queues is at or near capacity, messages (or, more generally, elements) can be added to the associated overflow queue. This avoids process blocking and / or deadlock, thereby preventing process blocking and / or deadlock from affecting the speed of processing by the order connector 1166. This allows the order connector 1166 to achieve overall speed and consistency. Furthermore, a burst of messages in a particular session (i.e., for a particular trading device 1110) does not affect the speed at which the order connector 1166 processes messages from another session. Rather, bursts of messages are added to a standard queue or an overflow queue in a particular session, as appropriate. This allows the order connector 1166 to operate consistently across sessions. Thus, the order connector 1166 may protect the processing of messages in one session (i.e., for a particular trading device 1110 or strategy engine 1167) from interference by bursts of messages in another session (i.e., for another trading device 1110 or strategy engine 1167). Thus, the operation of the order connector 1166 may be improved.

[0153] As used in this disclosure, the phrases "configured to" and "adapted to" encompass an element, structure, or device being modified, arranged, changed, or transformed to perform a particular function or for a particular purpose.

[0154] Some of the illustrated figures depict example block diagrams, systems, and / or flow diagrams that illustrate methods that may be used to implement all or portions of particular embodiments. One or more of the components, elements, blocks, and / or functions of the example block diagrams, systems, and / or flow diagrams may be implemented, alone or in combination, in, for example, hardware, firmware, discrete logic, a set of computer-readable instructions stored on a tangible computer-readable medium, and / or any combination thereof. The example block diagrams, systems, and / or flow diagrams may be implemented, for example, using any combination of application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable logic devices (FPLDs), discrete logic, hardware, and / or firmware.

[0155] The example block diagrams, systems, and / or flow diagrams may be implemented using, for example, one or more processors, controllers, and / or other processing devices. For example, the embodiments may be implemented using coded instructions, e.g., computer-readable instructions, stored on a tangible computer-readable medium. The tangible computer-readable medium may include various types of volatile and non-volatile storage media, including, for example, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), flash memory, hard disk drives, optical media, magnetic tape, file servers, any other tangible data storage devices, or any combination thereof. The tangible computer-readable medium is non-transitory.

[0156] Furthermore, although example block diagrams, systems, and / or flow diagrams have been described above with reference to the figures, other implementations may be employed. For example, the order of execution of the components, elements, blocks, and / or functions may be changed, and / or some of the described components, elements, blocks, and / or functions may be changed, eliminated, subdivided, or combined. Furthermore, any or all of the components, elements, blocks, and / or functions may be executed sequentially and / or in parallel, for example, by separate processing threads, processors, devices, discrete logic, and / or circuits.

[0157] While embodiments have been disclosed, various modifications may be made and equivalents may be substituted. In addition, many modifications may be made to adapt to a particular situation or material. Therefore, it is not intended that the disclosed technology be limited to the particular embodiments disclosed, but rather that the invention will include all embodiments falling within the scope of the appended claims.

[0158] Terms

[0159] Clause 1. A system comprising: a memory configured to include a first buffer for storing a number of elements to be processed; and a second buffer for storing elements to be added to the first buffer; and at least one processor configured to: retrieve elements to be processed; determine that the number of elements in the first buffer is greater than a first threshold; and add the retrieved elements to the second buffer.

[0160] Clause 2. The system of clause 1, wherein the at least one processor is further configured to add the retrieved element to the first buffer based at least in part on determining that the number of elements in the first buffer is less than the first threshold.

[0161] Clause 3. The system of clause 1 or clause 2, wherein the at least one processor is configured to set an overflow flag if the number of elements in the first buffer is greater than the first threshold, and the at least one processor is configured to determine that the number of elements in the first buffer is greater than the first threshold by determining that the overflow flag is set.

[0162] Clause 4. The system of any one of clauses 1 to 3, wherein the at least one processor is further configured to: determine that the number of elements in the first buffer is less than a second threshold, where the second threshold is lower than the first threshold; remove elements from the second buffer; and add the removed elements to the first buffer.

[0163] Clause 5. The system of any one of clauses 1 to 4, wherein the first buffer is a first queue for queuing a plurality of elements to be processed, and the second buffer is a second queue for queuing elements for addition to the first queue.

[0164] Clause 6. The system of any one of clauses 1 to 5, wherein the at least one processor is further configured to: determine that the number of elements in the second buffer is greater than 0; and add the retrieved element to the second buffer.

[0165] Clause 7. The system described in any one of clauses 1 to 6, wherein the at least one processor is further configured to determine that the second buffer is empty and the number of elements in the first buffer is less than the first threshold, and add the acquired element to the first buffer.

[0166] Clause 8. The system of clause 7, wherein the at least one processor is configured to set an overflow flag if the number of elements in the first buffer is greater than the first threshold and to reset the overflow flag if it is determined that the second buffer is empty, and wherein the at least one processor is configured to determine that the second buffer is empty and that the number of elements in the first buffer is less than the first threshold by determining that the overflow flag is reset.

[0167] Clause 9. The system of any one of clauses 1 to 8, wherein the first buffer is a ring buffer.

[0168] Clause 10. A system according to any one of clauses 1 to 9, wherein the second buffer is a queue for queuing elements for addition to the first buffer.

[0169] Clause 11. The system of any one of clauses 1 to 10, wherein the at least one processor is configured to implement a number of worker threads to add to the first buffer, and the first threshold is at most the total number of elements for which the first buffer has capacity minus the number of worker threads.

[0170] Clause 12. A system according to any one of clauses 1 to 11, wherein the element is a message and the at least one processor is configured to obtain the element by receiving a message from an external entity via a communication interface.

[0171] Clause 13. The system of any one of clauses 1 to 11, wherein the at least one processor is configured to obtain the element by removing the element from a third buffer.

[0172] Clause 14. A system according to any one of clauses 1 to 13, wherein each of the elements is a reference to one or more message queues.

[0173] Clause 15. A system as described in any one of clauses 1 to 14, wherein the element is associated with a message, the system including a server system including the at least one processor and the memory, the server system configured to facilitate messaging between a host system and one or more client systems.

[0174] Clause 16. The system of clause 15, wherein the one or more client systems include trading devices and the host system includes an electronic exchange.

[0175] Clause 17. A system as described in Clause 15 or Clause 16, wherein the server system is configured to establish a plurality of connections between the host system and each of a plurality of client systems, the memory is configured to provide a first buffer and an associated second buffer for each of the plurality of connections, and the at least one processor is configured to, for at least some of the connections, add an element to the respective first buffer or second buffer of the connection.

[0176] Clause 18. A system described in any one of clauses 15 to 17, wherein the messaging between the host system and the one or more client systems is bidirectional, the memory is configured to provide a first buffer and an associated second buffer for each of a first messaging direction and a second messaging direction, and the at least one processor is configured to, for each messaging direction, add an element to the respective first buffer or second buffer for that direction.

[0177] Clause 19. The system of any one of clauses 15 to 18, wherein the server system includes an order connector of an electronic trading platform, the order connector including the at least one processor and the memory.

[0178] Clause 20. A method comprising: providing in a memory a first buffer for storing a number of elements to be processed and a second buffer for storing elements to be added to the first buffer; by at least one processor, obtaining elements to be processed; determining that the number of elements in the first buffer is greater than a first threshold; and adding the obtained elements to the second buffer.

[0179] Clause 21. The method of clause 20, further comprising adding, by the at least one processor, the retrieved element to the first buffer based at least in part on determining that the number of elements in the first buffer is less than the first threshold.

[0180] Clause 22. The method of clause 20 or clause 21, wherein the step of setting an overflow flag by the at least one processor if the number of elements in the first buffer is greater than the first threshold, and the step of determining that the number of elements in the first buffer is greater than the first threshold relies on the step of determining that the overflow flag is set.

[0181] Clause 23. The method of any one of clauses 20 to 22, further comprising: determining, by the at least one processor, that the number of elements in the first buffer is less than a second threshold, wherein the second threshold is less than the first threshold; and removing elements from the second buffer and adding the removed elements to the first buffer.

[0182] Clause 24. The method of any one of clauses 20 to 23, wherein the first buffer is a first queue for queuing a plurality of elements to be processed, and the second buffer is a second queue for queuing elements for addition to the first queue.

[0183] Clause 25. The method of any one of clauses 20 to 24, further comprising the steps of: determining, by the at least one processor, that the number of elements in the second buffer is greater than 0; and adding the retrieved element to the second buffer.

[0184] Clause 26. A method according to any one of clauses 20 to 25, further comprising the steps of: determining, by the at least one processor, that the second buffer is empty and that the number of elements in the first buffer is less than the first threshold; and adding the retrieved element to the first buffer.

[0185] Clause 27. The method of clause 26, further comprising the steps of: setting, by the at least one processor, an overflow flag if the number of elements in the first buffer is greater than the first threshold; and resetting the overflow flag if it is determined that the second buffer is empty, wherein determining that the second buffer is empty and the number of elements in the first buffer is less than the first threshold is based on determining that the overflow flag is reset.

[0186] Clause 28. The method of any one of clauses 20 to 27, wherein the first buffer is a ring buffer.

[0187] Clause 29. The method of any one of clauses 20 to 28, wherein the second buffer is a queue for queuing elements for addition to the first buffer.

[0188] Clause 30. A method according to any one of clauses 20 to 29, comprising a step of determining, by the at least one processor, a number of worker threads to add to the first buffer, wherein the first threshold is at most the total number of elements for which the first buffer has capacity minus the number of worker threads.

[0189] Clause 31. A method according to any one of clauses 20 to 30, wherein the element is a message and the step of obtaining the element relies on receiving a message from an external entity via a communication interface.

[0190] Clause 32. The method of any one of clauses 20 to 30, wherein the step of obtaining the element relies on the step of removing the element from a third buffer.

[0191] Clause 33. A method according to any one of clauses 20 to 32, wherein the elements are each a reference to one or more message queues.

[0192] Clause 34. A method according to any one of clauses 20 to 33, wherein the element is associated with a message, the method is performed by a server system including the at least one processor and the memory, and the method includes a step of facilitating messaging between a host system and one or more client systems by the at least one processor.

[0193] Clause 35. The method of clause 34, wherein one or more of the client systems include trading devices and the host system includes an electronic exchange.

[0194] Clause 36. A method as described in clause 34 or clause 35, further comprising the steps of: establishing a plurality of connections between the host system and each of a plurality of client systems; providing in the memory a second buffer associated with a first buffer for each of the plurality of connections; and, for at least some of the plurality of connections, adding, by the at least one processor, an element to the respective first buffer or second buffer for that connection.

[0195] Clause 37. A method according to any one of clauses 34 to 36, wherein the messaging between the host system and the one or more client systems is bidirectional, and further comprising the steps of providing a first buffer and an associated second buffer for each of a first messaging direction and a second messaging direction, and adding, by the at least one processor, for each messaging direction, an element to the respective first buffer or second buffer for that direction.

[0196] Clause 38. The method of any one of clauses 34 to 37, wherein the memory and the at least one processor are part of an order connector of an electronic trading platform.

[0197] Clause 39. A tangible computer-readable medium containing instructions that, when executed, cause a machine to perform at least the following steps: providing in a memory a first buffer for storing elements to be processed and a second buffer for storing elements to be added to the first buffer; by at least one processor, obtaining elements to be processed; determining that the number of elements in the first buffer is greater than a first threshold; and adding the obtained elements to the second buffer.

[0198] Clause 40. A tangible computer-readable medium as recited in Clause 39, the instructions of which, when executed, cause a machine to perform the step of adding the retrieved element to a first buffer based at least in part on determining, by the at least one processor, that the number of elements in the first buffer is less than the first threshold.

[0199] Clause 41. A tangible computer-readable medium according to clause 39 or clause 40, the instructions of which, when executed, cause a machine to perform the step of, by the at least one processor, setting an overflow flag if the number of elements in the first buffer is greater than the first threshold, wherein determining that the number of elements in the first buffer is greater than the first threshold is based on determining that the overflow flag is set.

[0200] Clause 42. A tangible computer-readable medium according to any one of clauses 39 to 41, the instructions of which, when executed, cause a machine to perform the steps of: determining, by the at least one processor, that the number of elements in the first buffer is less than a second threshold, wherein the second threshold is less than the first threshold; removing elements from the second buffer; and adding the removed elements to the first buffer.

[0201] Clause 43. The tangible computer-readable medium of any one of clauses 39 to 42, wherein the first buffer is a first queue that queues a plurality of elements to be processed, and the second buffer is a second queue that queues elements for addition to the first queue.

[0202] Clause 44. A tangible computer-readable medium according to any one of clauses 39 to 43, the instructions of which, when executed, cause a machine to perform the steps of: determining, by the at least one processor, that the number of elements in the second buffer is greater than 0; and adding the retrieved element to the second buffer.

[0203] Clause 45. A tangible computer-readable medium according to any one of clauses 39 to 44, the instructions of which, when executed, cause a machine to perform the steps of: determining, by the at least one processor, that the second buffer is empty and that the number of elements in the first buffer is less than the first threshold; and adding the retrieved element to the first buffer.

[0204] Clause 46. A tangible computer-readable medium as set forth in Clause 45, the instructions of which, when executed, cause a machine to perform, by the at least one processor, the steps of: setting an overflow flag if the number of elements in the first buffer is greater than the first threshold; and resetting the overflow flag if it is determined that the second buffer is empty, wherein determining that the second buffer is empty and the number of elements in the first buffer is less than the first threshold is based on determining that the overflow flag is reset.

[0205] Clause 47. The tangible computer-readable medium of any one of clauses 39 to 46, wherein the first buffer is a ring buffer.

[0206] Clause 48. The tangible computer-readable medium of any one of clauses 39 to 47, wherein the second buffer is a queue that queues elements for addition to the first buffer.

[0207] Clause 49. A tangible computer-readable medium according to any one of clauses 39 to 48, the instructions of which, when executed, cause a machine to perform the step of: by the at least one processor, establishing a number of worker threads to add to the first buffer, wherein the first threshold is at most the total number of elements for which the first buffer has capacity minus the number of worker threads.

[0208] Clause 50. A tangible computer-readable medium according to any one of clauses 39 to 49, wherein the element is a message and the element is obtained by receiving a message from an external entity via a communications interface.

[0209] Clause 51. The tangible computer-readable medium of any one of clauses 39 to 49, wherein the element is obtained by removing an element from a third buffer.

[0210] Clause 52. A tangible computer-readable medium according to any one of clauses 39 to 51, wherein the elements are each a reference to one or more message queues.

[0211] Clause 53. A tangible computer-readable medium according to any one of clauses 39 to 52, wherein the elements are message-related and the instructions, when executed, cause a machine to implement a server system including the memory and the at least one processor, and facilitate messaging between a host system and one or more client systems by the at least one processor.

[0212] Clause 54. The tangible computer-readable medium of clause 53, wherein one or more of the client systems include trading devices and the host system includes an electronic exchange.

[0213] Clause 55. A tangible computer-readable medium according to clause 53 or clause 54, the instructions of which, when executed, cause a machine to perform the steps of establishing a plurality of connections between the host system and each of a plurality of client systems; providing in the memory a first buffer and an associated second buffer for each of the plurality of connections; and, for at least some of the plurality of connections, adding, by the at least one processor, an element to the respective first buffer or second buffer for that connection.

[0214] Clause 56. A tangible computer-readable medium as described in any one of clauses 53 to 55, wherein messaging between the host system and the one or more client systems is bidirectional, and the instructions, when executed, cause the machine to: provide a first buffer and an associated second buffer for each of a first messaging direction and a second messaging direction; and, for each messaging direction, add an element to the respective first buffer or second buffer for that direction by the at least one processor.

[0215] Clause 57. A tangible computer-readable medium according to any one of clauses 53 to 56, the instructions of which, when executed, cause a machine to implement an order connector of an electronic trading platform, the order connector comprising the at least one processor and the memory.

Claims

1. a first buffer for storing the number of elements to be processed; and a second buffer for storing elements to be added to the first buffer; and a memory configured to provide Get the element to be processed, determining that the number of elements in the first buffer is greater than a first threshold; Add the obtained element to the second buffer. at least one processor configured to A system comprising:

2. The at least one processor further comprises: adding the retrieved element to the first buffer based at least in part on determining that the number of elements in the first buffer is less than the first threshold; The system of claim 1 , configured to:

3. The at least one processor configured to set an overflow flag if the number of elements in the first buffer is greater than the first threshold; The at least one processor determining that the number of elements in the first buffer is greater than the first threshold by determining that the overflow flag is set; The system of claim 1 , configured to:

4. The at least one processor further comprises: determining that the number of elements in the first buffer is less than a second threshold that is lower than the first threshold; Remove an element from the second buffer and add the removed element to the first buffer. The system of claim 1 , configured to:

5. the first buffer is a first queue for queuing a plurality of elements to be processed, and the second buffer is a second queue for queuing elements to be added to the first queue; The system of claim 1 .

6. The at least one processor further comprises: determining that the number of elements in the second buffer is greater than zero; Add the obtained element to the second buffer. The system of claim 1 , configured to:

7. The at least one processor further comprises: determining that the second buffer is empty and that the number of elements in the first buffer is less than the first threshold; Add the obtained element to the first buffer. The system of claim 1 , configured to:

8. The at least one processor setting an overflow flag if the number of elements in the first buffer is greater than the first threshold; resetting the overflow flag when it is determined that the second buffer is empty; It is configured as follows: The at least one processor Determining that the second buffer is empty and the number of elements in the first buffer is less than the first threshold by determining that the overflow flag is reset. The system of claim 7 , configured to:

9. the first buffer is a ring buffer; The system of claim 1 .

10. the second buffer is a queue for queuing elements for addition to the first buffer; The system of claim 1 .

11. The at least one processor Implement the number of worker threads to add to the first buffer. It is configured as follows: the first threshold is at most the total number of elements for which the first buffer has capacity minus the number of worker threads; The system of claim 1 .

12. the element is a message, the at least one processor is configured to obtain the element by receiving a message from an external entity via a communication interface. The system of claim 1 .

13. The at least one processor configured to obtain the element by removing the element from a third buffer; The system of claim 1 .

14. each of said elements being a reference to one or more message queues; The system of claim 1 .

15. The element is associated with a message, the system includes a server system including the at least one processor and the memory; the server system is configured to facilitate messaging between a host system and one or more client systems; The system of claim 1 .

16. one or more of the client systems includes a trading device; the host system includes an electronic exchange; The system of claim 15.

17. the server system is configured to establish a plurality of connections between the host system and each of a plurality of client systems; the memory is configured to provide a first buffer and an associated second buffer for each of the plurality of connections; The at least one processor configured to, for at least some of the connections, add an element to the first buffer or the second buffer of each of the connections; The system of claim 15.

18. the messaging between the host system and the one or more client systems is bidirectional; the memory is configured to provide a first buffer and an associated second buffer for each of a first messaging direction and a second messaging direction; The at least one processor configured, for each messaging direction, to add an element to the first buffer or the second buffer, respectively, for that messaging direction; The system of claim 15.

19. the server system includes an order connector for an electronic trading platform; the order connector includes the at least one processor and the memory; The system of claim 15.

20. In memory, a first buffer for storing the number of elements to be processed; and a second buffer for storing elements to be added to the first buffer; providing a by at least one processor, obtaining an element to be processed; determining that the number of elements in the first buffer is greater than a first threshold; and adding the retrieved element to the second buffer; A method comprising:

21. A tangible computer-readable medium containing instructions that, when executed, cause a machine to perform at least the following actions: The operation is In memory, a first buffer for storing elements to be processed; and a second buffer for storing elements to be added to the first buffer; providing a by at least one processor, obtaining an element to be processed; determining that the number of elements in the first buffer is greater than a first threshold; and adding the retrieved element to a second buffer; 10. A tangible computer-readable medium,

Images