A real-time market processing method and system

Abstract

The application provides a real-time market processing method, and relates to the technical field of market data processing. The method comprises the following steps: S1, a multi-line multicast data receiving thread receives market data packets, and stores the market data in a packet buffer according to the receiving sequence of the market data packets; S2, a preprocessing thread splits the market data packets and adds them to an internal buffer CirBuf4Msg queue; S3, a real-time market processing thread acquires the market data from the CirBuf4Msg message queue according to the sequence number, decodes and consumes the market data, and completes the processing of the market data. The receiving, preprocessing, packet loss detection and repair of the market data packets are completed by using independent threads. The uniform message processing logic guarantees the sequence and accuracy of the market. In the extreme network condition, the data repair can be completed through retransmission and snapshot mechanism, and the accuracy of the market data is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of market data processing technology, specifically, to a real-time market data processing method and system. Background Technology

[0002] The Hong Kong Stock Exchange (HKEX) pushes real-time market data via multicast through its data platform OMD. The multicast underlying uses the UDP connectionless communication protocol. To meet the urgent need for faster market data delivery, it is necessary to achieve real-time reception and processing of multicast data, and to have technical means to repair multicast data under extreme network conditions to ensure the accuracy of real-time market data. Summary of the Invention

[0003] To overcome the shortcomings of the prior art, the present invention provides a real-time market data processing method and system to solve the above-mentioned technical problems.

[0004] The technical method adopted by this invention to solve its technical problem is: a real-time market data processing method, the improvement of which includes the following steps: S1, a multi-line multicast data receiving thread receives market data packets and stores the market data in the packet buffer according to the receiving order of the market data packets; S2, a preprocessing thread splits the market data packets and adds them to the internal buffer CirBuf4Msg queue, the CirBuf4Msg queue being a circular buffer for storing market messages; S3, a real-time market data processing thread retrieves market data from the CirBuf4Msg message queue according to the sequence number, decodes it, and consumes it to complete the processing of the market data.

[0005] In the above method, the market data package includes real-time market data packages and snapshot market data packages.

[0006] In the above method, step S1 includes the following steps:

[0007] S11. Start the multi-line multicast data receiving thread. The multi-line multicast data receiving thread obtains the corresponding multicast address according to the pre-set channel market data type and primary / backup line markers.

[0008] S12. After creating a UDP socket on the server, add it to the multicast address and wait for the multicast channel to push market data. UDP is a connectionless communication protocol.

[0009] S13. After receiving a UDP packet on the multicast channel, the multi-line multicast data receiving thread obtains the last available UDP packet buffer node from the packet buffer queue CirBuf inside the multi-line multicast data receiving thread according to the length of the received UDP packet. It then copies the entire UDP packet to the allocated storage space and continues to wait for the next UDP packet on the multicast channel. This process is repeated. CirBuf is a circular buffer for storing UDP packets.

[0010] S14. Store the data in the entire packet buffer in the order the data packets are received, for use by the preprocessing thread.

[0011] In the above method, step S13, obtaining the last available UDP packet buffer node from the packet buffer queue CirBuf inside the multi-line multicast data receiving thread, and copying the entire UDP packet to the allocated storage space, includes the following steps:

[0012] S131. Pre-allocate several buffer nodes and add all nodes to the CirBuf queue. Each buffer queue contains a read pointer readpos and a write pointer writepos. When writing data, the write pointer writepos is incremented sequentially to point to the next writable node. When consuming data, the read pointer readpos is incremented sequentially to point to the next readable node.

[0013] S132. When readpos = writepos, it means the queue is empty. At this time, it waits for new data packets to be read. When (writepos + 1) modulo the queue size equals readpos, it means the queue is full. If a new data packet is received at this time, the buffer queue is automatically expanded, a fixed-size buffer node is allocated, and these newly allocated buffer nodes are added to the tail of the queue.

[0014] S133. Adjust the writepos position to write the UDP packet to the new tail free buffer.

[0015] In the above method, step S2 includes the following steps:

[0016] S21. The preprocessing thread retrieves each UDP packet sequentially from the packet buffer queue CirBuf.

[0017] S22. Associate all messages within the packet with an incrementing sequence number;

[0018] S23. Insert the UDP packet into the specified position of the CirBuf4Msg message queue according to the sequence number.

[0019] In the above method, step S23, inserting UDP packets according to sequence numbers, is based on a line arbitration algorithm, which includes the following steps:

[0020] S231. Compare whether the sequence number of the current message is less than the sequence number of the smallest waiting message in CirBuf4Msg. If it is, discard the current message. If not, find the position in the CirBuf4Msg message queue where the current message should appear based on the sequence number of the current message and the sequence number of the smallest waiting message.

[0021] S232. Check if the position in the queue has been occupied, that is, whether the message with the sequence number has been written by other lines. If it has not been occupied, write the message and set the written flag. If it has been occupied, discard the message.

[0022] S233. When a new message is inserted into the CirBuf4Msg message queue, check whether all messages between the sequence number of the new message and the sequence number of the smallest waiting message have been received. If all messages have been received, notify the real-time market data processing thread through the condition variable and jump to step S3. If not, wait for the arrival of a new data packet to make the sequence numbers continuous, and then notify the real-time market data processing thread through the condition variable and jump to step S3.

[0023] In the above method, step S3 includes the following steps:

[0024] S31. The real-time market data processing thread receives a notification from the preprocessing thread and starts from the smallest sequence number of the waiting message in the CirBuf4Msg message queue to obtain a segment of message data containing consecutive sequence numbers for market data decoding.

[0025] S32. Update the internal cache data structure and serialize the processed result into a protobuf message. Protobuf is a mechanism open-sourced by Google that implements automatic data serialization and deserialization.

[0026] S33. Send the protobuf messages to the Kafka message storage system, process the consecutive messages one by one, modify the value of the minimum waiting sequence number, and provide it to downstream consumers for subscription and use, thus completing the processing of market data. Kafka is a high-throughput distributed publish-subscribe messaging system.

[0027] This invention also provides a real-time market data processing system, including a multi-line multicast data receiving thread, a preprocessing thread, and a real-time market data processing thread.

[0028] The multi-line multicast data receiving thread is used to receive market data packets and store the data in the packet buffer according to the order in which the market data packets are received.

[0029] The preprocessing thread is connected to the multi-line multicast data receiving thread to split market data packets and add them to the internal buffer CirBuf4Msg queue.

[0030] The real-time market data processing thread is connected to the preprocessing thread. It is used to retrieve market data from the CirBuf4Msg message queue according to the sequence number, decode it, consume it, and complete the processing of the market data.

[0031] The system also includes a data integrity detection thread, which is connected to the preprocessing thread and is used to detect whether there is a message gap in CirBuf4Msg and to find consecutive missing sequence numbers from the MissingSet set, which is the set of missing sequence numbers of messages in the preprocessing thread.

[0032] The system also includes a retransmission data thread, which is connected to the data integrity detection thread and the preprocessing thread. After receiving a notification from the data integrity detection thread, the retransmission data thread retrieves messages within a specified sequence number range and updates them to the corresponding message queue CirBuf4Msg to complete data repair.

[0033] The beneficial effects of this invention are: using an independent thread to complete the reception, preprocessing, packet loss detection and repair of market data packets; ensuring the order and accuracy of market data through unified message processing logic; and ensuring the accuracy of market data by completing data repair through retransmission and snapshot mechanisms in extreme network conditions. Attached Figure Description

[0034] Appendix Figure 1 This is a flowchart of a real-time market data processing method according to the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0037] Reference Figure 1 As shown, a real-time market data processing method of the present invention includes the following steps:

[0038] S1. The multi-line multicast data receiving thread receives market data packets and stores the market data in the packet buffer according to the receiving order of the market data packets. The market data packets include real-time market data packets and snapshot market data packets.

[0039] Specifically, it includes the following steps:

[0040] S11. Start the multi-line multicast data receiving thread. The multi-line multicast data receiving thread obtains the corresponding multicast address according to the pre-set channel market data type and primary / backup line markers.

[0041] S12. After creating a UDP socket on the server, add it to the multicast address and wait for the multicast channel to push market data. The market data includes real-time market data and snapshot market data. UDP is a connectionless communication protocol.

[0042] S13. After receiving a UDP packet from the multicast channel, the multi-line multicast data receiving thread retrieves the last available UDP packet buffer node from the packet buffer queue CirBuf inside the multi-line multicast data receiving thread, based on the length of the received UDP packet. The entire UDP packet is copied to the allocated storage space, and then the thread continues to wait for the next UDP packet from the multicast channel. This process is repeated. CirBuf is a circular buffer that stores UDP packets. It is a circular queue with an expandable size. Data is continuously written to the tail of the queue. When the number of unprocessed UDP packets exceeds the current size of the queue, it is automatically expanded and the packets are written to the newly expanded data buffer to ensure that the received UDP packets are not dropped.

[0043] Specifically, the last available UDP packet buffer node is obtained from the packet buffer queue CirBuf within the multi-line multicast data receive thread, and the entire UDP packet is copied to the allocated storage space, including the following steps:

[0044] S131. Pre-allocate several buffer nodes. Since the maximum size of a UDP packet is 1500 bytes, the size of the buffer node can be 1500 bytes. Add all nodes to the CirBuf queue. The buffer queue contains a read pointer readpos and a write pointer writepos. When writing data, the write pointer writepos is incremented sequentially to point to the next writable node. When consuming data, the read pointer readpos is incremented sequentially to point to the next readable node.

[0045] S132. When readpos = writepos, it means the queue is empty. At this time, it waits for new data packets to be read. When (writepos + 1) modulo the queue size equals readpos, it means the queue is full. If a new data packet is received at this time, the buffer queue is automatically expanded, a fixed-size (e.g., 1500 bytes) buffer node is allocated, and these newly allocated buffer nodes are added to the tail of the queue.

[0046] S133. Adjust the writepos position to write the UDP packet to the new tail free buffer.

[0047] S14. Store the data in the entire packet buffer in the order the data packets are received, for use by the preprocessing thread.

[0048] S2. The preprocessing thread splits the market data packets and adds them to the internal buffer CirBuf4Msg queue. The CirBuf4Msg queue is a circular buffer that stores market data messages.

[0049] Specifically, it includes the following steps:

[0050] S21. The preprocessing thread retrieves each UDP packet sequentially from the packet buffer queue CirBuf.

[0051] S22. Associate all messages within the packet with an incrementing sequence number;

[0052] S23. Insert the UDP packet into the specified position of the CirBuf4Msg message queue according to the sequence number;

[0053] Inserting UDP packets based on sequence numbers is based on a line arbitration algorithm to ensure that received messages are ordered by sequence number and to discard duplicate messages. This algorithm includes the following steps:

[0054] S231. Compare whether the sequence number of the current message is less than the sequence number of the smallest waiting message in CirBuf4Msg. If it is, discard the current message. If not, find the position in the CirBuf4Msg message queue where the current message should appear based on the sequence number of the current message and the sequence number of the smallest waiting message.

[0055] S232. Check if the position in the queue has been occupied, that is, whether the message with the sequence number has been written by other lines. If it has not been occupied, write the message and set the written flag. If it has been occupied, discard the message.

[0056] S233. When a new message is inserted into the CirBuf4Msg message queue, check whether all messages between the sequence number of the new message and the sequence number of the smallest waiting message have been received. If all messages have been received, notify the real-time market data processing thread through the condition variable and jump to step S3. If not, it means that this interval is not continuous. In this case, no other operation is required after insertion. Wait for the arrival of subsequent new (out-of-order) data packets to make the packet sequence of the entire interval continuous. Then notify the real-time market data processing thread through the condition variable and jump to step S3.

[0057] S3: The real-time market data processing thread retrieves market data from the CirBuf4Msg message queue according to the sequence number, decodes it, consumes it, and completes the processing of the market data.

[0058] Specifically, it includes the following steps:

[0059] S31. The real-time market data processing thread receives a notification from the preprocessing thread and starts from the smallest sequence number of the waiting message in the CirBuf4Msg message queue to obtain a segment of message data containing consecutive sequence numbers for market data decoding.

[0060] S32. Update the internal cache data structure and serialize the processed result into a protobuf message. Protobuf is a mechanism open-sourced by Google that implements automatic data serialization and deserialization.

[0061] S33. The protobuf message is sent to the Kafka message storage system, where consecutive messages are processed one by one, and the minimum waiting sequence number is modified to be made available for downstream consumers to subscribe to, thus completing the processing of market data. Kafka is a high-throughput distributed publish-subscribe messaging system. If the number of available messages in CirBuf4Msg is empty (i.e., no data is available, or the queue has data but cannot be read temporarily due to gaps in the sequence number), a condition variable is used to wait for a new message to arrive before triggering a queue readable operation. When CirBuf4Msg becomes readable due to the arrival of a new data packet, the real-time market data processing thread obtains the new data packets in sequence and decodes them, repeating this process.

[0062] This invention also provides a real-time market data processing system, including a multi-line multicast data receiving thread, a preprocessing thread, and a real-time market data processing thread.

[0063] The multi-line multicast data receiving thread is used to receive market data packets and store the data in the packet buffer according to the order in which the market data packets are received.

[0064] The preprocessing thread is connected to the multi-line multicast data receiving thread to split market data packets and add them to the internal buffer CirBuf4Msg queue.

[0065] The real-time market data processing thread is connected to the preprocessing thread. It is used to retrieve market data from the CirBuf4Msg message queue according to the sequence number, decode it, consume it, and complete the processing of the market data.

[0066] Furthermore, it also includes a data integrity detection thread and a data retransmission thread. The data integrity detection thread is connected to the preprocessing thread and is used to detect whether there are message gaps in CirBuf4Msg. It finds consecutive missing sequence numbers from the MissingSet set, which is the set of missing sequence numbers of messages in the preprocessing thread. Since some data packets may be lost, when a UDP message is written to CirBuf4Msg, messages with larger sequence numbers will record consecutive missing sequence numbers in the MissingSet set when they are written to the queue. Therefore, the data integrity detection thread needs to check the MissingSet set of the queue, find a consecutive missing sequence number from the set, and notify the data retransmission thread along with the channel ID and other relevant information, so that the latter can complete the data repair.

[0067] The retransmission thread is connected to the data integrity detection thread and also to the preprocessing thread. Upon receiving a notification from the data integrity detection thread, it retrieves messages within a specified sequence number range and updates the corresponding message queue CirBuf4Msg to complete data repair. This thread provides an external interface for the data integrity detection thread to use, storing the corresponding channel's missing sequence number gap in an internal cache. When the missing message queue is not empty, this thread connects to the HKEX RTS data retransmission service via TCP. After login and authentication, it sends the channel ID and sequence number range of the data to be repaired, retrieves messages within the specified sequence number range of that channel through the response, and then updates the message queue CirBuf4Msg corresponding to that channel ID. Once the missing messages are recovered, a queue readable operation may be triggered. At this time, the real-time market data processing thread can continue processing messages in the real-time queue sequentially until it becomes unreadable again.

[0068] Furthermore, the present invention can also read the list of channel IDs to be processed when the program starts by means of a configuration file, and create a thread group for each channel. Each thread group includes a multi-line multicast data receiving thread, a preprocessing thread, a real-time market data processing thread, a data integrity detection thread, and a retransmission data thread, so as to realize real-time market data processing of multiple channels and quickly receive market data of any number of channels from the Hong Kong Stock Exchange.

[0069] This invention uses an independent thread to complete the reception, preprocessing, packet loss detection and repair of market data packets. Through a unified message processing logic, it ensures the order and accuracy of market data. Furthermore, in extreme network conditions, it can complete data repair through retransmission and snapshot mechanisms to ensure the accuracy of market data.

[0070] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A real-time market data processing method, characterized in that: The steps include the following: S1. The multi-line multicast data receiving thread receives market data packets and stores them in the packet buffer according to the receiving order. The market data packets include real-time market data packets and snapshot market data packets; Step S1 includes the following steps: S11. Start the multi-line multicast data receiving thread. The multi-line multicast data receiving thread obtains the corresponding multicast address according to the pre-set channel market data type and primary / backup line markers. S12. After creating a UDP socket on the server, add it to the multicast address and wait for the multicast channel to push market data. UDP is a connectionless communication protocol. S13. After receiving a UDP packet on the multicast channel, the multi-line multicast data receiving thread obtains the last available UDP packet buffer node from the packet buffer queue CirBuf inside the multi-line multicast data receiving thread according to the length of the received UDP packet. It then copies the entire UDP packet to the allocated storage space and continues to wait for the next UDP packet on the multicast channel. This process is repeated. The packet buffer queue CirBuf is a circular buffer that stores UDP packets. S14. Store the data in the entire packet buffer queue CirBuf in the order of packet reception for use by the preprocessing thread; S2. The preprocessing thread splits the market data packets, associates all messages in the market data packets with an incrementing sequence number, and adds them to the internal buffer CirBuf4Msg message queue. The CirBuf4Msg message queue is a circular buffer for storing market data messages. S3: The real-time market data processing thread retrieves market data from the CirBuf4Msg message queue according to the sequence number, decodes it, consumes it, and completes the processing of the market data.

2. The real-time market data processing method as described in claim 1, characterized in that: In step S13, the last available UDP packet buffer node is obtained from the packet buffer queue CirBuf inside the multi-line multicast data receiving thread, and the entire UDP packet is copied to the allocated storage space, including the following steps: S131. Pre-allocate several buffer nodes and add all nodes to the packet buffer queue CirBuf. The packet buffer queue CirBuf contains a read pointer readpos and a write pointer writepos. When writing data, the write pointer writepos is incremented sequentially to point to the next writable node. When consuming data, the read pointer readpos is incremented sequentially to point to the next readable node. S132. When readpos = writepos, it means that the packet buffer queue CirBuf is empty. At this time, it waits for new data packets to be read. When the size of the packet buffer queue CirBuf is equal to readpos (modulo 1), it means that the packet buffer queue CirBuf is full. If a new data packet is received at this time, the packet buffer queue CirBuf is automatically expanded, a fixed-size buffer node is allocated, and these newly allocated buffer nodes are added to the tail of the packet buffer queue CirBuf. S133. Adjust the writepos position to write the UDP packet to the new tail free buffer.

3. The real-time market data processing method as described in claim 2, characterized in that: Step S2 includes the following steps: S21. The preprocessing thread retrieves each UDP packet sequentially from the packet buffer queue CirBuf. S22. Associate all messages within the UDP packet with an incrementing sequence number; S23. Insert the UDP packet into the specified position of the CirBuf4Msg message queue according to the sequence number.

4. The real-time market data processing method as described in claim 3, characterized in that: Step S23, inserting the UDP packet into the specified position of the CirBuf4Msg message queue according to the sequence number, is implemented based on the line arbitration algorithm, which includes the following steps: S231. Compare whether the sequence number of the current message is less than the sequence number of the smallest waiting message in the CirBuf4Msg message queue. If it is, discard the current message. If not, find the position in the CirBuf4Msg message queue where the current message should appear based on the sequence number of the current message and the sequence number of the smallest waiting message. S232. Check if the position in the CirBuf4Msg message queue has been occupied, that is, whether the message with the sequence number has been written by other lines. If it has not been occupied, write the current message and set the written flag. If it has been occupied, discard the current message. S233. When a new message is inserted into the CirBuf4Msg message queue, check whether all messages between the sequence number of the new message and the sequence number of the smallest waiting message have been received. If all messages have been received, notify the real-time market data processing thread through the condition variable and jump to step S3. If not, wait for the arrival of a new data packet to make the sequence numbers continuous, and then notify the real-time market data processing thread through the condition variable and jump to step S3.

5. The real-time market data processing method as described in claim 4, characterized in that: Step S3 includes the following steps: S31. The real-time market data processing thread receives a notification from the preprocessing thread and starts from the smallest sequence number of the waiting message in the CirBuf4Msg message queue to obtain a segment of message data containing consecutive sequence numbers for market data decoding. S32. Update the internal cache data structure and serialize the processed result into a protobuf message. Protobuf is a mechanism open-sourced by Google that implements automatic data serialization and deserialization. S33. Send the protobuf message to the Kafka message storage system, process the consecutive messages one by one, modify the value of the smallest waiting message sequence number, and provide it to downstream consumers for subscription and use to complete the processing of market data. Kafka is a high-throughput distributed publish-subscribe messaging system.

6. A real-time market data processing system, characterized in that: This system is used to implement a real-time market data processing method as described in any one of claims 1-5, including a multi-line multicast data receiving thread, a preprocessing thread, and a real-time market data processing thread. The multi-line multicast data receiving thread is used to receive market data packets and store them in the packet buffer according to the order in which they are received. The preprocessing thread is connected to the multi-line multicast data receiving thread to split market data packets and add them to the internal buffer CirBuf4Msg message queue. The real-time market data processing thread is connected to the preprocessing thread. It is used to retrieve market data from the CirBuf4Msg message queue according to the sequence number, decode it, consume it, and complete the processing of the market data.

7. The real-time market data processing system as described in claim 6, characterized in that: It also includes a data integrity detection thread, which is connected to the preprocessing thread and is used to detect whether there is a message gap in the CirBuf4Msg message queue, and to find consecutive missing sequence numbers from the MissingSet set, which is the set of missing sequence numbers of messages in the preprocessing thread.

8. The real-time market data processing system as described in claim 7, characterized in that: It also includes a retransmission data thread, which is connected to the data integrity detection thread and the preprocessing thread. After receiving a notification from the data integrity detection thread, the retransmission data thread retrieves messages within a specified sequence number range and updates them to the corresponding CirBuf4Msg message queue to complete data repair.

Images