A method and system for processing multi-exchange market information of American stocks

Abstract

This invention provides a method and system for processing market data from multiple US stock exchanges. The method includes the following steps: S1, a timed thread generates asynchronous events; S2, the main thread executes the asynchronous events and subscribes to stock code market data in batches; S3, a market data callback processing thread executes market data message processing logic and puts market data messages into the corresponding message queue; S4, a market data decoding thread parses market data messages, obtains real-time and delayed market data, puts delayed market data into a cache queue, and determines whether the real-time market data is intraday data. If it is intraday data, it is put into the cache queue; if it is pre-market or post-market data, it is put into the market data message queue; S5, a market data merging thread merges the real-time and delayed market data in the cache queue, and puts the merged result into the market data message queue; S6, a market data sending thread converts the messages in the market data message queue into protobuf format and sends them to the Kafka streaming system for downstream consumers.

Description

Technical Field

[0001] This invention relates to the field of market data processing technology, specifically, to a method and system for processing market data from multiple US stock exchanges. Background Technology

[0002] US stock market data access includes delayed quotes, Level 1 advanced quotes, and Level 2 advanced quotes. Level 1 advanced quotes display real-time streaming data from the Nasdaq NB market before and after market hours, while during trading hours it simultaneously receives delayed quotes from several major US stock exchanges (including Nasdaq Composite Delayed Quotes, ARCA / AMEX / NYSE Delayed Quotes, etc., typically with a delay of 15 minutes). For downstream market data subscription services, how to correct the intraday quote data to accurately reflect the US stock market situation has become a pressing issue. This invention provides a solution by merging real-time and delayed quote data to correct intraday quote data. Summary of the Invention

[0003] To overcome the shortcomings of the existing technology, the present invention provides a method and system for processing US stock market data from multiple exchanges, in order to solve the above-mentioned technical problems.

[0004] The technical method adopted by this invention to solve its technical problem is: a method for processing market data from multiple US stock exchanges, the improvement of which includes the following steps: S1, a timed thread generates asynchronous events, including events for exchanges and the US stock market; S2, the main thread executes the asynchronous events and subscribes to stock code market data in batches; S3, a market data callback processing thread executes market data message processing logic, puts market data messages into the corresponding message queue, and the market data messages are divided into chaincode market data and stock code market data according to the code type; S4, a market data decoding thread parses the market data messages, obtains real-time market data and delayed market data, puts the delayed market data into a cache queue, and determines whether the real-time market data is intraday market data. If it is intraday market data, the real-time market data is put into the cache queue; if it is pre-market or post-market market data, the real-time market data is put into the market data message queue, and then jumps to step S6; S5, a market data merging thread merges the real-time market data and delayed market data in the cache queue, and puts the merged result into the market data message queue; S6, a market data sending thread converts the messages in the market data message queue into protobuf format and sends them to the Kafka streaming system for downstream consumers to use.

[0005] In the above method, step S1 includes the following steps:

[0006] S11. After the main thread starts, it reads the configuration file and adds all timed event settings to the timed thread.

[0007] S12. The timer thread periodically checks whether the current time meets the time setting of any timer task. If it does, the event type and exchange identifier for the exchange are written to the event message queue. If it does not meet, the timer thread sleeps for a period of time and then checks again until the time setting of any timer task is met.

[0008] In the above method, step S2 includes the following steps:

[0009] S21. Upon receiving this event, the main thread initiates a chaincode subscription request to the Reuters TREP server, that is, requests to obtain all security codes in this market.

[0010] S22. Batch subscription to stock code market data.

[0011] In the above method, step S3 includes chaincode market data, which includes chaincode snapshot market data and chaincode update market data.

[0012] In the above method, step S4, in which the market data decoding thread parses market data messages to obtain real-time and delayed market data, includes the following steps:

[0013] S401. For each exchange, a corresponding message queue and cache data structure are created internally;

[0014] S402: After the market data receiving thread sends the data to the message queue, it retrieves the message, and the market data decoding thread parses out the security code.

[0015] S403. Based on the security code, retrieve the cached data of the security code from the market data cache, parse each field of the market data message, update the internal cached data, and obtain real-time and delayed market data.

[0016] In the above method, step S4, determining whether the real-time market data is intraday data, is based on the latest time and trading calendar.

[0017] In the above method, step S5 includes the following steps:

[0018] S51, The market data merging thread retrieves messages from the cache queue;

[0019] S52. Check if there is market data with the opposite state in the market data cache. If so, merge the real-time market data and the delayed market data according to specific rules. If not, directly retrieve the market data in the cache queue and merge them.

[0020] S53. Update the latest data for the security code in the market data cache and put the merged result into the market data message queue.

[0021] In the above method, the specific rule in step S52 includes:

[0022] (1) Latest price: Based on the market time, the latest price with the longer market time is selected between real-time market and delayed market.

[0023] (2) Highest price: The highest value of the highest price in real-time market data and delayed market data;

[0024] (3) Lowest price: The lowest value between the real-time market price and the delayed market price;

[0025] (4) Opening price: The opening price of the delayed market shall be used. If there is no opening price in the delayed market, the opening price of the real-time market shall be used.

[0026] (5) Previous closing price: The previous closing price of the delayed market shall be used. If the delayed market does not have a previous closing price, the previous closing price of the real-time market shall be used.

[0027] (6) Closing price: The closing price is based on the delayed market closing price. If the delayed market does not have a closing price, the real-time market closing price will be used.

[0028] In the above method, step S6 includes the following steps:

[0029] S61. The market data sending thread retrieves the original cached messages to be sent one by one from the market data message queue;

[0030] S62. Messages internally converted to protobuf format are sent to the corresponding topic in the Kafka streaming system according to the market data type and market status for downstream consumption. Market data types include market data, tick data, or order book data, and market status includes pre-market, intraday, or post-market.

[0031] This invention also provides a US stock market multi-exchange market data processing system, including a timed thread, a main thread, a market data callback processing thread, a market data decoding thread, a market data merging thread, and a market data sending thread.

[0032] A timed thread is used to generate asynchronous events, including events related to exchanges and the US stock market.

[0033] The main thread is used to execute asynchronous events and subscribe to stock code quotes in batches;

[0034] The market data callback processing thread is used to execute market data message processing logic, putting market data messages into the corresponding message queue. Market data messages are divided into chaincode market data and security code market data according to code type.

[0035] The market data decoding thread is used to parse market data messages, obtain real-time market data and delayed market data. Delayed market data is placed in a cache queue. The thread also determines whether the real-time market data is intraday data. If it is intraday data, the real-time market data is placed in the cache queue. If it is pre-market or post-market data, the real-time market data is placed in the market data message queue and further processed by the market data sending thread.

[0036] The market data merging thread is used to merge real-time and delayed market data in the cache queue, and the merged result is put into the market data message queue.

[0037] The market data sending thread is used to convert messages in the market data message queue into protobuf format and send them to the Kafka streaming system for downstream consumers.

[0038] The beneficial effects of this invention are: it enables the definition of the opening and closing times of each exchange through configuration files, and the automatic subscription and unsubscription of market data; it processes market data from various exchanges through independent threads; and it completes the merging of real-time and delayed market data based on specific rules, ensuring data accuracy. Attached Figure Description

[0039] Appendix Figure 1 This is a logic diagram of a method and system for processing US stock market data across multiple exchanges, as described in this invention.

[0040] Appendix Figure 2 This is a flowchart of a method for processing US stock market data across multiple exchanges according to the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0043] Reference Figure 1 and Figure 2 A method for processing US stock market data across multiple exchanges includes the following steps:

[0044] S1. A timed thread generates asynchronous events, including events for exchanges and the US stock market. Exchange-related events include opening, closing, and trading hours, while US stock market-related events include retrieving the trading calendar and setting daylight saving time (US stocks use Eastern Time). All these events are generated by the timed thread and executed asynchronously by the main thread. This allows for defining event trigger times for multiple exchanges via configuration files, with the timed thread generating asynchronous events based on the configured times and the current time to notify the main thread for execution.

[0045] Specifically, it includes the following steps:

[0046] S11. After the main thread starts, it reads the configuration file and adds all timed event settings to the timed thread.

[0047] S12. The timer thread checks at regular intervals (e.g., every 500ms) whether the current time meets the time setting of any timer task. If it does, the event type and exchange identifier for the exchange are written to the event message queue. If it does not meet, the timer thread sleeps for a period of time (e.g., sleeps for 500ms) and then checks again until the time setting of any timer task is met.

[0048] Here's an example configuration file. The `supported_exchange` configuration item contains a list of exchanges to subscribe to. For each exchange, you'll set `refresh_time` (market data subscription time), `clear_time` (clearing market data sending time), and `close_time` (closing time). This allows you to subscribe to market data starting at the corresponding Eastern Time, receive clearing market data, and cancel the subscription at the close of trading.

[0049] S2: The main thread executes asynchronous events to subscribe to stock code market data in batches;

[0050] Specifically, step S2 includes the following steps:

[0051] S21. Upon receiving this event, the main thread initiates a chaincode subscription request to the Reuters TREP server, that is, requests to obtain all security codes in this market.

[0052] S22. Batch subscription to stock code market data;

[0053] After the main thread starts, it is primarily used to execute asynchronous events. Since the service may experience abnormal restarts, it's necessary to consider how to achieve rapid market data recovery. Normally, a timed thread generates an opening event based on the opening time. Upon receiving this event, the main thread initiates a chaincode subscription request to the Reuters TREP server, requesting all security codes for that market. All subsequent subscription messages are processed asynchronously by the market data callback thread. After a service restart, depending on the time, it may occur after the opening time. In this case, the timed thread cannot be relied upon to generate the opening event. Instead, a replay operation is performed. This involves reading the security code file written by the market data callback thread to obtain the full set of security codes for that exchange, and then subscribing to market data in batches to achieve rapid recovery of market data. This is handled asynchronously by the callback thread.

[0054] S3, the market data callback processing thread executes the market data message processing logic, placing market data messages into the corresponding message queue. Market data messages are categorized into chaincode market data and security code market data based on their code type; further, chaincode market data includes chaincode snapshot market data and chaincode update market data. The processing logic for different types of market data messages is as follows:

[0055] (1) Chaincode snapshot market data

[0056] After the main thread subscribes to the chaincode, it first receives a snapshot of the chaincode's market data. Since a single exchange has a large number of security codes, it may take multiple snapshot messages to obtain all the codes. Once all the security codes are obtained locally, the callback thread begins subscribing to market data in batches of 2000 codes. Because obtaining all the security codes is time-consuming, typically taking several seconds, to reduce market data loss due to subscription delays after service restarts, the market data callback processing thread writes all security codes by exchange to a system file for the main thread to use when resuming subscriptions.

[0057] (2) Chaincode updates market data

[0058] This type of market information is less frequent and is used to push updates when code changes occur after the market opens. For example, in special circumstances, new codes may become available for trading after the market opens. When receiving this type of market information, the code list in each update message needs to be compared with the previously received full code list to confirm whether new codes have been added. For newly added codes, the system should subscribe immediately and update the code file.

[0059] (3) Stock Code Quotes

[0060] The callback thread creates a local copy of the stock code market data message and, depending on the exchange, places the copy into the corresponding message queue for processing by the market data decoding thread.

[0061] S4. The market data decoding thread parses market data messages, obtains real-time and delayed market data, and puts delayed market data into a cache queue (used for data correction, so delayed market data in the intraday state is directly put into the cache queue). At the same time, it determines whether the real-time market data is intraday based on the latest time and trading calendar. If it is intraday market data, it is put into the cache queue and jumps to the next step S5; if it is pre-market or post-market data, it is put into the market data message queue and jumps to step S6.

[0062] The market data decoding thread parses market data messages to obtain real-time and delayed market data, including the following steps:

[0063] S401. For each exchange, a corresponding message queue and cache data structure are created internally;

[0064] S402: After the market data receiving thread sends the data to the message queue, it retrieves the message, and the market data decoding thread parses out the security code.

[0065] S403. Based on the security code, retrieve the cached data of the security code from the market data cache, parse each field of the market data message, update the internal cached data, and obtain real-time and delayed market data.

[0066] S5. The market data merging thread merges the real-time and delayed market data during the intraday period in the cache queue, and puts the merged result into the market data message queue.

[0067] Specifically, step S5 includes the following steps:

[0068] S51: An independent market data merging thread retrieves a message from the cache queue;

[0069] S52. Check if there is market data with the opposite state in the market data cache (if the message in the cache queue is real-time market data, then check the delayed market data for that code in the market data cache; if the message in the cache queue is delayed market data, then check the real-time market data for that security code in the market data cache). If there is, then merge the real-time market data and the delayed market data based on specific rules. If not (for example, there is only real-time market data in the cache queue and no delayed market data in the market data cache, or there is only delayed market data in the cache queue and no real-time market data in the market data cache), then the merge field value is directly taken from the market data in the cache queue and merged.

[0070] The specific rules include:

[0071] (1) Latest price: Based on the market time, the latest price with the longer market time is selected between real-time market and delayed market.

[0072] (2) Highest price: The highest value of the highest price in real-time market data and delayed market data;

[0073] (3) Lowest price: The lowest value between the real-time market price and the delayed market price;

[0074] (4) Opening price: The opening price of the delayed market shall be used. If there is no opening price in the delayed market, the opening price of the real-time market shall be used.

[0075] (5) Previous closing price: The previous closing price of the delayed market shall be used. If the delayed market does not have a previous closing price, the previous closing price of the real-time market shall be used.

[0076] (6) Closing price: The closing price is based on the delayed market data. If there is no closing price for the delayed market data, the closing price for the real-time market data will be used.

[0077] S53. After the merger is completed, update the latest data of the security code in the market data cache and put the merged result into the market data message queue.

[0078] S6. The market data sending thread converts the messages in the market data message queue into protobuf format and sends them to the Kafka streaming system for downstream consumers to use. Protobuf is an open-source mechanism from Google that implements automatic data serialization and deserialization. Kafka is a high-throughput distributed publish-subscribe messaging system.

[0079] Step S6 includes the following steps:

[0080] S61. An independent market data sending thread retrieves the original cached messages to be sent one by one from the market data message queue;

[0081] S62. Messages internally converted to protobuf format are sent to the corresponding topic in the Kafka streaming system according to the market data type and market status for downstream consumption. Market data types include market data, tick data, or order book data, and market status includes pre-market, intraday, or post-market.

[0082] The above methods enable the storage of exchange security code lists in files, allowing for rapid data recovery through batch subscription of market data on the main thread in the event of a service restart. The market data receiving thread reduces reception latency using multiple market data message queues, while multiple market data decoding threads asynchronously parse market data messages, achieving high-concurrency processing. A separate market data merging thread merges real-time and delayed market data messages, improving data accuracy. An independent market data sending thread stores protobuf formatted data in a streaming system for downstream market data consumers. By implementing market data subscription, reception, decoding, merging, and storage through independent threads, decoupled from each other by queues within the service process, overall message processing capabilities are enhanced.

[0083] This invention also provides a US stock market multi-exchange market data processing system, including a timed thread, a main thread, a market data callback processing thread, a market data decoding thread, a market data merging thread, and a market data sending thread.

[0084] A timed thread is used to generate asynchronous events, including events related to exchanges and the US stock market.

[0085] The main thread is used to execute asynchronous events and subscribe to stock code quotes in batches;

[0086] The market data callback processing thread is used to execute market data message processing logic, putting market data messages into the corresponding message queue. Market data messages are divided into chaincode market data and security code market data according to code type.

[0087] The market data decoding thread is used to parse market data messages, obtain real-time market data and delayed market data. Delayed market data is placed in a cache queue. The thread also determines whether the real-time market data is intraday data. If it is intraday data, the real-time market data is placed in the cache queue. If it is pre-market or post-market data, the real-time market data is placed in the market data message queue and further processed by the market data sending thread.

[0088] The market data merging thread is used to merge real-time and delayed market data in the cache queue, and the merged result is put into the market data message queue.

[0089] The market data sending thread is used to convert messages in the market data message queue into protobuf format and send them to the Kafka streaming system for downstream consumers.

[0090] This invention enables the definition of opening and closing times for each exchange through configuration files, automatic subscription and unsubscription of market data; processing of market data from various exchanges through independent threads; and merging of real-time and delayed market data based on specific rules to ensure data accuracy.

[0091] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for processing US stock market data from multiple exchanges, characterized in that: The steps include the following: S1. A timed thread generates asynchronous events, including events related to exchanges and the US stock market. S2: The main thread executes asynchronous events to subscribe to stock code market data in batches; S3, the market data callback processing thread executes the market data message processing logic, puts the market data messages into the corresponding message queue, and the market data messages are divided into chain code market data and security code market data according to the code type; S4. The market data decoding thread parses market data messages, obtains real-time market data and delayed market data, puts delayed market data into the cache queue, and determines whether the real-time market data is intraday data. If it is intraday data, it puts the real-time market data into the cache queue. If it is pre-market or post-market data, it puts the real-time market data into the market data message queue and jumps to step S6. S5. The market data merging thread merges the real-time and delayed market data in the cache queue, and puts the merged result into the market data message queue. Step S5 includes the following steps: S51, The market data merging thread retrieves messages from the cache queue; S52. Check if there is market data with the opposite state in the market data cache. If so, merge the real-time market data and the delayed market data according to specific rules. If not, directly retrieve the market data in the cache queue and merge them. S53. Update the latest data for the security code in the market data cache and put the merged result into the market data message queue; In step S52, the specific rule includes: (1) Latest price: Based on the market time, the latest price with the longer market time is selected between real-time market and delayed market. (2) Highest price: The highest value of the highest price in real-time market data and delayed market data; (3) Lowest price: The lowest value between the real-time market price and the delayed market price; (4) Opening price: The opening price of the delayed market shall be used. If there is no opening price in the delayed market, the opening price of the real-time market shall be used. (5) Previous closing price: The previous closing price of the delayed market shall be used. If the delayed market does not have a previous closing price, the previous closing price of the real-time market shall be used. (6) Closing price: The closing price is based on the delayed market data. If there is no closing price for the delayed market data, the closing price for the real-time market data will be used. S6, the market data sending thread converts the messages in the market data message queue into protobuf format and sends them to the Kafka streaming system for downstream consumers to use.

2. The method for processing US stock market data across multiple exchanges as described in claim 1, characterized in that: Step S1 includes the following steps: S11. After the main thread starts, it reads the configuration file and adds all timed event settings to the timed thread. S12. The timer thread periodically checks whether the current time meets the time setting of any timer task. If it does, the event type and exchange identifier for the exchange are written to the event message queue. If it does not meet, the timer thread sleeps for a period of time and then checks again until the time setting of any timer task is met.

3. The method for processing US stock market data across multiple exchanges as described in claim 2, characterized in that: Step S2 includes the following steps: S21. Upon receiving this event, the main thread initiates a chaincode subscription request to the Reuters TREP server, that is, requests to obtain all security codes in this market. S22. Batch subscription to stock code market data.

4. The method for processing US stock market data from multiple exchanges as described in claim 3, characterized in that: In step S3, the chaincode market data includes chaincode snapshot market data and chaincode update market data.

5. The method for processing US stock market data across multiple exchanges as described in claim 4, characterized in that: In step S4, the market data decoding thread parses market data messages to obtain real-time and delayed market data, including the following steps: S401. For each exchange, a corresponding message queue and cache data structure are created internally; S402: After the market data receiving thread sends the data to the message queue, it retrieves the message, and the market data decoding thread parses out the security code. S403. Based on the security code, retrieve the cached data of the security code from the market data cache, parse each field of the market data message, update the internal cached data, and obtain real-time and delayed market data.

6. The method for processing US stock market data from multiple exchanges as described in claim 5, characterized in that: In step S4, the determination of whether the real-time market data is intraday data is based on the latest time and trading calendar.

7. The method for processing US stock market data from multiple exchanges as described in claim 1, characterized in that: Step S6 includes the following steps: S61. The market data sending thread retrieves the original cached messages to be sent one by one from the market data message queue; S62. Messages internally converted to protobuf format are sent to the corresponding topic in the Kafka streaming system according to the market data type and market status for downstream consumption. Market data types include market data, tick data, or order book data, and market status includes pre-market, intraday, or post-market.

8. A US stock market multi-exchange market data processing system, characterized in that: This includes a timer thread, a main thread, a market data callback processing thread, a market data decoding thread, a market data merging thread, and a market data sending thread. A timed thread is used to generate asynchronous events, including events related to exchanges and the US stock market. The main thread is used to execute asynchronous events and subscribe to stock code quotes in batches; The market data callback processing thread is used to execute market data message processing logic, putting market data messages into the corresponding message queue. Market data messages are divided into chaincode market data and security code market data according to code type. The market data decoding thread is used to parse market data messages, obtain real-time market data and delayed market data. Delayed market data is placed in a cache queue. The thread also determines whether the real-time market data is intraday data. If it is intraday data, the real-time market data is placed in the cache queue. If it is pre-market or post-market data, the real-time market data is placed in the market data message queue and further processed by the market data sending thread. The market data merging thread is used to merge real-time and delayed market data in the cache queue, and the merged result is put into the market data message queue. The steps include the following: S51, The market data merging thread retrieves messages from the cache queue; S52. Check if there is market data with the opposite state in the market data cache. If so, merge the real-time market data and the delayed market data according to specific rules. If not, directly retrieve the market data in the cache queue and merge them. S53. Update the latest data for the security code in the market data cache and put the merged result into the market data message queue; In step S52, the specific rule includes: (1) Latest price: Based on the market time, the latest price with the longer market time is selected between real-time market and delayed market. (2) Highest price: The highest value of the highest price in real-time market data and delayed market data; (3) Lowest price: The lowest value between the real-time market price and the delayed market price; (4) Opening price: The opening price of the delayed market shall be used. If there is no opening price in the delayed market, the opening price of the real-time market shall be used. (5) Previous closing price: The previous closing price of the delayed market shall be used. If the delayed market does not have a previous closing price, the previous closing price of the real-time market shall be used. (6) Closing price: The closing price is based on the delayed market data. If there is no closing price for the delayed market data, the closing price for the real-time market data will be used. The market data sending thread is used to convert messages in the market data message queue into protobuf format and send them to the Kafka streaming system for downstream consumers.

Images