A market replay method and system based on log files

Abstract

The present invention provides a market replay method and system based on log files, and relates to the technical field of market data processing. The method comprises the following steps: S1, a receiving thread receives real-time market information from a market source, and the real-time market information is added to a message queue; S2, a decoding thread performs a transcoding operation on the real-time market information, that is, converts the message format from the data source into an internal message format, and adds the market information to the log queue in the converted message format; S3, a log thread asynchronously receives and processes the market information in the log queue, caches it into an internal data structure, and writes the cached log messages in batches into a full log file; S4, a replay thread reads the log file, finds the market information of the code according to the code prefix to be replayed, and writes the market information of the code into the message queue; S5, a decoding thread decodes the market information of the code in the message queue, and writes it into a message cache system to complete the market replay.

Description

Technical Field

[0001] The present invention relates to the technical field of market data processing, and in particular to a market replay method and system based on log files. Background Art

[0002] Exchanges or third-party market data sources will only send real-time market information after the market opens, and the market decoding system can only perform decoding operations after receiving real-time market information. Since exchanges around the world may be in different time zones, for example, the Nasdaq exchange is located in the Eastern Time Zone of the United States, and intraday market information will not be sent until after 9:30 p.m. Beijing time, once a data processing error occurs in the decoding program, developers usually cannot locate the error in a timely manner, and since the exchange does not support market replay, the error cannot be reproduced through normal processes. Therefore, a market replay solution is needed to meet market needs. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides a market replay method and system based on log files to solve the above technical problems.

[0004] The technical method adopted by the present invention to solve its technical problem is: a market delay method based on a streaming system, and its improvement lies in: comprising the following steps: S1, a receiving thread receives real-time market information from a market source, and the real-time market information is added to a message queue; S2, a decoding thread performs a transcoding operation on the real-time market information, that is, converts the message format from the data source into an internal message format, and adds the market information to a log queue in the converted message format; S3, a log thread asynchronously receives and processes market information in the log queue, caches it into an internal data structure, and writes the cached log messages in batches into a full log file; S4, a replay thread reads the log file, finds the market information of the code according to the code prefix to be replayed, and writes the market information of the code into the message queue; S5, a decoding thread decodes the market information of the code in the message queue, and writes it into a message cache system to complete market information replay.

[0005] In the above method, before step S4, it also includes the step of splitting the full log file into multiple sliced ​​log files by using a log cutting tool.

[0006] In the above method, in step S1, the receiving thread receives the real-time market information from the market source through asynchronous callback.

[0007] In the above method, in step S2, the market message after the message format is converted includes the following information:

[0008] Securities code, message sequence number, exchange ID, delay flag, message type, message generation time and message content.

[0009] In the above method, in step S3, writing the cached log messages in batches to the full log file includes the following steps:

[0010] S301, determining whether the cached log message meets the write condition, the write condition being: the cached log message reaches a specified number, or the current time exceeds a specified interval from the last file write time and the number of cached log messages is greater than 0;

[0011] S302: If the write condition is met, the cached log messages are written in batches to the full log file.

[0012] In the above method, in step S4, the market information of the code is written into the message queue, including the following two strategies:

[0013] Fast playback strategy, that is, each time a record is read, the record is reversed into a market message and the market message is written into the message queue;

[0014] The delayed playback strategy is to compare the market receiving time in the log with the current time. If the current time is less than the market receiving time, it will sleep for a certain period of time, and repeat this cycle until the current time is greater than the market receiving time. Then the log is converted into a market message and written into the message queue.

[0015] The present invention also provides a market replay system based on log files, including a receiving thread, a decoding thread, a log thread and a replay thread.

[0016] The receiving thread is used to receive real-time market information from the market source, and the real-time market information is added to the message queue;

[0017] The decoding thread is used to perform transcoding operation on the real-time market message, that is, converting the message format from the data source into the internal message format, and adding the market message in the converted message format to the log queue;

[0018] The log thread is used to asynchronously receive and process market information in the log queue, cache it in the internal data structure, and write the cached log messages in batches to the full log file;

[0019] The replay thread is used to read the log file, find the market information of the code to be replayed according to the prefix of the code, and write the market information of the code into the message queue;

[0020] The decoding thread is also used to decode the market information of the code in the message queue and write it into the message cache system to complete the market information replay.

[0021] The above system also includes a log cutting tool for splitting a full log file into multiple sliced ​​log files.

[0022] The beneficial effects of the present invention are: realizing market replay based on log files, and supporting two playback strategies: fast playback and delayed playback; and using a log cutting tool to cut the full log file into multiple independent slice log files according to specified rules, which is convenient for fast positioning of data and market replay. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Attached Figure 1 The present invention is a logic diagram of a market replay method based on log files.

[0024] Attached Figure 2 The present invention is a flow chart of a market replay method based on log files. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0027] Reference Figure 1 As shown in the figure, the exchange or third-party market data source sends real real-time market information after the market opens. The market decoding system performs decoding operations after receiving the real-time market information. The specific process is as follows: After the market decoding system subscribes to the code market information from the market source, it receives the real-time market information message through the asynchronous callback of the receiving thread; the real-time market information message is added to the message queue, and the decoding thread decodes the market information in the message queue, performs data analysis, merges the incremental market information into the market cache, and finally converts the merged message into a protobuf message, which is asynchronously written into the message cache system through the sending thread to provide it to downstream use.

[0028] In order to achieve market replay, the decoding thread adds a new market message transcoding operation in addition to the original market message decoding processing logic. Figure 1 and Figure 2As shown, the present invention provides a market replay method based on log files, comprising the following steps:

[0029] S1. The receiving thread receives real-time market information from the market source through asynchronous callback, and the real-time market information is added to the message queue.

[0030] S2, the decoding thread performs a transcoding operation on the real-time market message, that is, converting the message format (e.g., binary message) from the data source into an internal message format (e.g., text message), and adding the market message in the converted message format to the log queue;

[0031] The market message after the message format is converted includes the following key information:

[0032] Securities code, that is, which stock code this log belongs to; message sequence number, that is, the unique identifier of the market message coming from a specific exchange; exchange ID, that is, which exchange the market message is sent from; delay mark, marking whether the market message is real-time or delayed; message type, marking whether the message is a snapshot message or an update message; message generation time, recording the millisecond-level time when the market message is received; message content, that is, a data list containing all message field names, IDs and corresponding values ​​in the message, with multiple fields separated by the '|' symbol.

[0033] S3. After the decoding thread adds each market message to the log queue according to the above message format, the log thread asynchronously receives and processes the market messages in the log queue and caches them in the internal data structure. When the cached log messages meet one of the following writing conditions: the cached log messages reach the specified number (for example, 100), or the current time exceeds the specified interval (for example, 100ms) from the last file write time and the number of cached log messages is greater than 0, the cached log messages are batch-written to the full log file;

[0034] Since the log thread continuously appends the latest log to the end of the log, all the market information for the whole day is written into a single log file. For U.S. stocks, the log file written in this way is about 20G a day. Due to the large amount of data, it is not convenient to trace back all the historical market conditions of a single code, because each time a new code historical market condition is queried, the entire file needs to be read once, the disk access volume is too large, and the query is relatively slow. Therefore, the full log file can be further split into multiple sliced ​​log files through the log cutting tool. Taking U.S. stocks as an example, since U.S. stock codes are mainly represented by English letters, the log cutting tool can split the full log file into 26 log slice files based on the first code prefix, and the names start from AZ. When you need to query the historical market conditions of a specific code, find the corresponding slice file according to the code prefix for search, which greatly reduces the disk file reading volume and speeds up the query.

[0035] The normal process needs to receive market information from the real market source, and to achieve market replay, it is necessary to start an independent replay thread within the market decoding system.

[0036] S4. The replay thread finds the log file according to the prefix of the code to be replayed. To speed up the query, after using the log cutting tool, the sliced ​​log file can be directly found according to the prefix of the code to be replayed, and the content in the slice file is read one by one to filter out all the market conditions of the code, and the market conditions of the code are written into the message queue according to different playback strategies;

[0037] The playback strategy includes the following two strategies: the fast playback strategy, that is, each time a record is read, the record is reversed into a market message and the market message is written into the message queue; the delayed playback strategy, that is, the market reception time in the log is compared with the current time. If the current time is less than the market reception time, it will sleep for a certain time (for example, 10ms), and repeat this cycle until the current time is greater than the market reception time, then the log is converted into a market message and written into the message queue. For the delayed playback strategy, because this method requires obtaining the current time, which is the local time of the machine, only when the local time of the machine exceeds the market reception time, the log will be converted into a market message and written into the message queue. Therefore, it is necessary to adjust the local time on the playback machine where the market decoding system is located to return to the same day as the market message date.

[0038] At this step, the subsequent process is the same as the normal process of receiving real-time market information.

[0039] S5. The decoding thread decodes the market information of the code in the message queue, performs data analysis, merges the incremental market information into the market information cache, and finally converts the merged message into a protobuf message, which is asynchronously written into the message cache system through the sending thread to be provided to the downstream for use, completing the market information replay and achieving accurate reproduction.

[0040] The present invention also provides a market replay system based on log files, including a receiving thread, a decoding thread, a log thread and a replay thread.

[0041] The receiving thread is used to receive real-time market information from the market source, and the real-time market information is added to the message queue;

[0042] The decoding thread is used to perform transcoding operation on the real-time market message, that is, converting the message format from the data source into the internal message format, and adding the market message in the converted message format to the log queue;

[0043] The log thread is used to asynchronously receive and process market information in the log queue, cache it in the internal data structure, and write the cached log messages in batches to the full log file;

[0044] The replay thread is used to read the log file, find the market information of the code to be replayed according to the prefix of the code, and write the market information of the code into the message queue;

[0045] The decoding thread is also used to decode the market information of the code in the message queue and write it into the message cache system to complete the market information replay.

[0046] Furthermore, the market replay system based on log files also includes a log cutting tool for splitting the full log file into multiple sliced ​​log files, so as to facilitate rapid location of data and market replay.

[0047] The present invention realizes market replay based on log files, and supports two playback strategies: fast playback and delayed playback; and uses a log cutting tool to cut the full log file into multiple independent slice log files according to specified rules, which is convenient for fast positioning of data and market replay.

[0048] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A market replay method based on log files, characterized by: The steps include: S1. The receiving thread receives real-time market information from the market source, and the real-time market information is added to the message queue; S2, the decoding thread performs a transcoding operation on the real-time market message, that is, converting the message format from the data source into an internal message format, and adding the market message in the converted message format to the log queue; S3, the log thread asynchronously receives and processes market information in the log queue, caches it in the internal data structure, and writes the cached log messages in batches to the full log file; S4, the replay thread reads the log file, finds the market information of the code to be replayed according to the code prefix, and writes the market information of the code into the message queue; S5. The decoding thread decodes the market information of the code in the message queue and writes it into the message cache system to complete the market information replay; Before step S4, the step of splitting the full log file into multiple sliced ​​log files is also included using a log cutting tool.

2. A log file-based market replay method as claimed in claim 1, characterized in that: In step S1, the receiving thread receives the real-time market information from the market source through asynchronous callback.

3. A log file-based market replay method as claimed in claim 1, characterized in that: In step S2, the market information after the message format is converted includes the following information: Securities code, message sequence number, exchange ID, delay flag, message type, message generation time and message content.

4. A log file-based market replay method as claimed in claim 1, characterized in that: In step S3, the cached log messages are written in batches to the full log file, including the following steps: S301, determining whether the cached log message meets the write condition, the write condition being: the cached log message reaches a specified number, or the current time exceeds a specified interval from the last file write time and the number of cached log messages is greater than 0; S302: If the write condition is met, the cached log messages are written in batches to the full log file.

5. A log file-based market replay method as claimed in claim 1, characterized in that: In step S4, the market information of the code is written into the message queue, including the following two strategies: Fast playback strategy, that is, each time a record is read, the record is reversed into a market message and the market message is written into the message queue; The delayed playback strategy is to compare the market receiving time in the log with the current time. If the current time is less than the market receiving time, it will sleep for a certain period of time, and repeat this cycle until the current time is greater than the market receiving time. Then the log is converted into a market message and written into the message queue.

6. A market replay system based on log files, characterized by: Including receiving thread, decoding thread, logging thread and replay thread, The receiving thread is used to receive real-time market information from the market source, and the real-time market information is added to the message queue; The decoding thread is used to perform transcoding operation on the real-time market message, that is, converting the message format from the data source into the internal message format, and adding the market message in the converted message format to the log queue; The log thread is used to asynchronously receive and process market information in the log queue, cache it in the internal data structure, and write the cached log messages in batches to the full log file; The replay thread is used to read the log file, find the market information of the code to be replayed according to the prefix of the code, and write the market information of the code into the message queue; The decoding thread is also used to decode the market information of the code in the message queue and write it into the message cache system to complete the market information replay; It also includes a log cutting tool for splitting the full log file into multiple sliced ​​log files.

Images