Multi-path market data analysis and optimization method based on FPGA (Field Programmable Gate Array)

Abstract

The invention relates to the technical field of data processing, in particular to an FPGA (Field Programmable Gate Array)-based multi-path market data analysis and optimization method. The method comprises the following steps that S1, market data flow enters an FPGA through a 10GMAC module, and the 10GMAC module is mainly responsible for analyzing interactive sending of the FPGA and external data; s2, the market data flow passes through an XGE module, the XGE module strips a data packet header, and complete service data information is left and sent to a downstream module to be processed; s3, the service information enters a PARSER analysis module, and the PARSER analysis module analyzes and integrates the original market information service information; s4, the analyzed market information data flow enters an OPT module, and the OPT module selects one path of complete data from multiple paths of market information and transmits the data to a subsequent module; and S5, the optimized market information service data enters a UDP module and then is sent to a network by a 10GMAC module. The method is efficient, stable, safe and flexible, and has important significance for improving the performance of a financial transaction system and meeting diversified requirements of transaction users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-path market data analysis and optimization method based on FPGA. BACKGROUND

[0002] The current single-link market analysis and distribution cannot meet the requirements of securities companies and transaction users for system security. There are many links in the whole chain from the generation of market data to the consumption of transaction users, including high-speed private lines from transaction hosts to market gateways, market analysis and forwarding servers, 10G low-latency switches, 10G low-latency network cards, and transaction user strategy servers. Any one of them is highly dependent on complex IT equipment and technology, so there are many uncertain factors, and any slight fluctuation in any link may cause a large amount of real-time market data stream to be lost, which may cause unpredictable harm to transaction business. Traditional software programs will produce large delays when analyzing large amounts of real-time market data streams, especially under the design concept of first-point multi-source optimization and deduplication, the amount of data processed by the system will increase exponentially, which will put more pressure on software processing. In addition, transaction customers may only need to receive part of the market data of securities according to their own needs. Traditional software programs will also consume a lot of time when supporting customer subscription operations, therefore, a multi-path market data analysis and optimization method based on FPGA is designed. SUMMARY

[0003] The purpose of the present application is to provide a multi-path market data analysis and optimization method based on FPGA to solve the problem of security and data processing delay of single-link market analysis and distribution in the background technology.

[0004] To achieve the above purpose, the present application provides a multi-path market data analysis and optimization method based on FPGA, comprising the following steps: S1, the market data stream enters the FPGA through the 10GMAC module, and the 10GMAC module is used for the interaction and transmission of FPGA and external data; S2, the market data stream passes through the XGE module, and the XGE module strips the data packet header to leave the complete business data information for the downstream module processing; S3, the business information enters the PARSER analysis module, and the PARSER analysis module analyzes and integrates the original market business information; S4, the analyzed market data stream is sent to the OPT module, and the OPT module selects one complete data from the multi-path market data for the subsequent module; S5, the preferred market data into the UDP module, the data encapsulated by the UDP module into UDP packet, and then by 10GMAC module to send to the network.

[0005] As a further improvement of the technical solution, the market data can be divided into snapshot data and per-pen data, the snapshot data includes securities basic information, market overview information, price information and bid-ask information, the per-pen data includes transaction time, transaction price, transaction volume, transaction direction, transaction identification, transaction status and transaction participant, the per-pen data records all the orders received by the exchange and the transaction.

[0006] As a further improvement of the technical solution, the core processing step of the OPT module in S4 is: S4.1, the OPT module reads the double-path market data from the PARSER analysis module of A and B respectively, and marks the double-path business data with time stamp TA and TB, and marks the double-path business data with sequence number CNTA and CNTB at each time stamp; S4.2, judge whether TA=TB=TC is true, if true, it means that the business market time stamp of the last time is equal to the time stamp read from A and B, at this time, further judge whether CNTA and CNTB are equal to CNTC, if equal, then send the market of CNTC value, after sending the market, assign the time stamp of sending the market to TC, and assign the CNT of that path to CNTC; S4.3, when TA=TB=TC is not true, judge whether TC is equal to one of the time stamps, if equal, at this time, TB is greater than or less than TC; when TC=TA<TB, it means that B route market updates, at this time, output B route market, otherwise, when TC=TA>TB, it means that B route market time is earlier, at this time, output A route market; S4.4, when TA=TB=TC is not true, judge whether TC is equal to one of the time stamps, if not equal to any of the time stamps, at this time, if TC>TA&&TC>TB, it means that A and B double-path market are all lagging behind the current latest time, at this time, no market is sent, if TB<TC<TA, it means that A route market updates, A route market is sent, if TC<TA<TB, it means that A and B route market are all updated, at this time, A route should be selected to send, to avoid losing business market data in TA and TB time, if TC<TA and TA=TB, it means that the same updated time is read at the same time, at this time, compare CNTA and CNTB, output the data of the smaller one, to avoid losing part of the data at TA=TB time; S4.5 Execute the logic judgments of S4.1~S4.4, send market data to the UDP module, and update the TC and CNTC values. Then, the data reading operation can be performed again to read the new TA, TB, CNTA, and CNTB, and repeat the logic judgments of S4.1~S4.

[0007] As a further improvement to this technical solution, TC is the last timestamp of the market data that has been sent. It is assigned to TC by TA or TB at the time of sending. CNTC is the count value of the next business data to be sent. It is assigned to CNTC by incrementing CNTA or CNTB at the time of sending. The initial values ​​of TA, TB and TC are 0, and the initial value of CNTC is 1.

[0008] As a further improvement to this technical solution, the complete flow of market data is as follows: the market data publishing platform flows to the market data gateway, the market data gateway flows to low-latency switch one, low-latency switch one flows to the parsing end FPGA, the parsing end FPGA flows to low-latency switch two, low-latency switch two flows to the receiving end FPGA, and the receiving end FPGA flows to the client receiving end program.

[0009] As a further improvement to this technical solution, the specific process of the market data release platform flowing to the market data gateway is as follows: after real-time market data is generated on the order matching platform, it is sent out by the market data release platform, and then the market data gateway obtains the market data stream from the market data release platform.

[0010] As a further improvement to this technical solution, the specific process of the market data gateway flowing to the low-latency switch is as follows: the server where the market data gateway is located is directly connected to the low-latency switch through a 10 Gigabit fiber optic cable. After the raw market data obtained from the market data gateway enters the low-latency switch, the low-latency switch will simultaneously and completely output the raw market data obtained from one port to multiple ports for downstream use.

[0011] As a further improvement to this technical solution, the specific process of the low-latency switch leading to the parsing FPGA is as follows: After the raw market data passes through the low-latency switch, it enters the parsing FPGA. The parsing FPGA processes the raw market data into a standard and unified format according to its own logic and outputs a complete market data stream. The parsing FPGA has two parsing FPGAs, a primary and a backup. Each parsing FPGA can simultaneously access two raw market data streams from different sources and maintain a dual-active state where both parsing FPGAs work at the same time.

[0012] As a further improvement to this technical solution, the specific process of the data flow from the parsing FPGA to the low-latency switch 2 is as follows: the market data obtained from the parsing FPGA enters the low-latency switch 2, and the low-latency switch 2 will simultaneously and completely output the raw market data obtained from one port to multiple ports.

[0013] As a further improvement to this technical solution, the specific process of low-latency switch 2 flowing to the receiving FPGA is as follows: market data enters the receiving FPGA through low-latency switch 2, and the receiving FPGA processes the market data into a standard and unified format according to its own logic.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This method employs a "multi-source selection + primary / backup dual-active" mechanism. The parsing FPGA supports simultaneous access to multiple market data sources and selects the best for parsing, while simultaneously keeping the primary and backup parsing FPGAs active. The receiving FPGA supports simultaneous access to dual parsed market data sources, performs deduplication internally, and finally outputs a complete parsed market data source to the end-user strategy program. This method avoids the impact of single-point failures on system operation.

[0015] 2. This method uses FPGA hardware with high concurrency processing capabilities to complete core computing logic such as multi-source optimal parsing, dual-active deduplication, and subscription. Compared with traditional software programs, it can significantly improve the latency caused by data processing, promote transaction efficiency, and improve the effectiveness of the securities market. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the flow of market data within an FPGA, as described in this invention.

[0018] Figure 3 This is a flowchart of the OPT module of the present invention.

[0019] Figure 4 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Please seeFigure 1 As shown, this embodiment provides a method for parsing and optimizing multi-channel market data based on FPGA, including the following steps: 1. A method for parsing and optimizing multi-channel market data based on FPGA, characterized by the following steps: S1. Market data stream enters the FPGA through the 10GMAC module. The 10GMAC module is used for data exchange and transmission between the FPGA and the outside world.

[0022] Market data is divided into snapshot data and tick-by-tick data. Snapshot data includes basic securities information, market overview information, price information, and buy / sell order information. Snapshot data provides a market overview at a specific point in time. Tick-by-tick data includes transaction time, transaction price, transaction volume, transaction direction, transaction identifier, transaction status, and transaction participants. Tick-by-tick data records all orders received by the exchange and the transaction details.

[0023] S2. Market data streams pass through the XGE module, which strips the packet headers, leaving complete business data information, and sends it to downstream modules for processing.

[0024] S3. Business information enters the PARSER parsing module, which then parses and integrates the original market data.

[0025] S4. The parsed market data stream will enter the OPT module, which is the selection module. Since the PARSER module parses multiple market data sources simultaneously, the OPT module needs to select one complete data stream from the multiple market data streams and hand it over to the subsequent modules.

[0026] The core processing steps of the OPT module in S4 are as follows: The S4.1 and OPT modules read dual-channel market data from the A and B dual-channel PARSER parsing modules respectively, and mark each data entry with timestamps TA and TB. At the same time, they mark each data entry with a sequence number CNTA and CNTB at each timestamp. The A and B dual-channel PARSER parsing modules represent two PARSER parsing modules, namely the APARSER parsing module and the BPARSER parsing module. S4.2 Determine whether TA=TB=TC is true. If it is true, it means that the timestamp of the last business market data sent is equal to the timestamp read from channels A and B this time. At this time, further determine whether the read CNTA and CNTB are equal to CNTC. If they are equal, send a market data with the same value as CNTC. After sending the market data, assign the timestamp of the market data sent to TC, and increment CNT of that channel by one and assign it to CNTC. S4.3. When TA = TB = TC does not hold, determine whether TC is equal to one of the timestamps. If it is equal to one of the timestamps, such as TC = TA && TC != TB, at this time, TB may be greater than or less than TC. So when TC = TA < TB, it means that the market data of channel B is updated, and at this time, output the market data of channel B. On the contrary, when TC = TA > TB, it means that the market time of channel B is earlier, and at this time, output the market data of channel A; S4.4. When TA = TB = TC does not hold, determine whether TC is equal to one of the timestamps. If it is not equal to any of the timestamps, at this time, if TC > TA && TC > TB, it means that the market data of both channels A and B lags behind the current latest time, and at this time, no market data is sent. If TB < TC < TA, it means that the market data of channel A is updated, and send the market data of channel A. If TC < TA < TB, it means that the market data of both channels A and B are updated. At this time, channel A should be selected for sending to avoid losing the business market data within the time period of TA and TB. If TC < TA and TA = TB, it means that the market data at the same update moment is read simultaneously. At this time, compare CNTA and CNTB, and output the data of the channel with the smaller CNT to avoid losing some data at the moment of TA = TB; S4.5. Execute the logical judgment of S4.1 to S4.4, send the market data to the UDP module, and at the same time update the values of TC and CNTC. Then, the data reading operation can be executed again to read the new TA, TB, CNTA, and CNTB, and repeat the above logical judgment of S4.1 to S4.4.

[0027] TC is the last timestamp of the market data that has been sent, which is assigned to TC by TA or TB during sending. CNTC is the count value of the next business data expected to be sent, which is assigned to CNTC by adding 1 to CNTA or CNTB during sending. The initial values of TA, TB, and TC are 0, and the initial value of CNTC is 1.

[0028] Through steps S4.1 to S4.5, it can be ensured that in the case of multiple inputs, the parsing FPGA only outputs one complete parsed UDP market data externally. Even if a single market data source fails, it will not affect the normal operation of the parsing FPGA hardware.

[0029] S5. The optimized market business data enters the UDP module, adds the UDP packet, and then is sent to the network by the 10GMAC module.

[0030] The complete flow of the market data is as follows: from the market release platform to the market gateway, from the market gateway to the low-latency switch 1, from the low-latency switch 1 to the parsing end FPGA, from the parsing end FPGA to the low-latency switch 2, from the low-latency switch 2 to the receiving end FPGA, and from the receiving end FPGA to the client receiving end program.

[0031] The specific process of data flow from the market data publishing platform to the market data gateway is as follows: After real-time market data is generated on the order matching platform, it is sent out by the market data publishing platform. Only after obtaining authorization from the exchange can the brokerage firm obtain the market data stream from the market data publishing platform through the market data gateway. The data between the market data publishing platform and the market data gateway is transmitted via a dedicated line, and the exchange is responsible for the accuracy and security of the data transmission.

[0032] The specific process of data flow from the market data gateway to the low-latency switch is as follows: The server where the market data gateway is located is directly connected to the low-latency switch via a 10 Gigabit fiber optic cable. After the raw market data obtained from the market data gateway enters the low-latency switch, the low-latency switch will simultaneously and completely output the raw market data obtained from one port to multiple ports for downstream use.

[0033] The specific process of the low-latency switch leading to the parsing FPGA is as follows: After the raw market data passes through the low-latency switch, it enters the parsing FPGA. The parsing FPGA processes the raw market data into a standard and unified format according to its own logic and outputs a complete market data stream. The parsing FPGA has two parsing FPGAs, a primary and a backup. Each parsing FPGA can simultaneously access raw market data from two different sources and maintain a dual-active state where both parsing FPGAs are working at the same time.

[0034] The specific process of the data flow from the parsing FPGA to the low-latency switch 2 is as follows: the market data obtained from the parsing FPGA enters the low-latency switch 2, and the low-latency switch 2 will simultaneously and completely output the raw market data obtained from one port to multiple ports, so that multiple downstream customer receiving ends can use it at the same time.

[0035] The specific process of low-latency switch 2 flowing to the receiving FPGA is as follows: market data enters the receiving FPGA through low-latency switch 2. The receiving FPGA processes the market data into a standard and unified format according to its own logic. The processing method is the same as that of the parsing FPGA.

[0036] The receiving FPGA receives parsed market data via multicast. The FPGA's two network ports support independent multicast access, each receiving parsed market data from both the primary and backup streams. For the received UDP market data, the receiving FPGA performs deduplication, consistent with the multi-source selection logic of the parsing FPGA. Furthermore, the receiving FPGA supports selecting "all subscription" or "no subscription" based on two dimensions: security type (A-shares, B-shares, funds, bonds, etc.) and market data type (snapshot, tick-by-tick, 50 ticks, etc.). In addition to "all subscription" and "no subscription," it further supports subscribing to specific market data types for individual security targets.

[0037] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An FPGA-based multi-channel market data parsing and optimization method, characterized in that: Comprise the following steps: S1, market data stream through 10GMAC module into FPGA, 10GMAC module is used for the interaction of FPGA and outside data transmission; S2, market data stream through XGE module, by XGE module will be stripped off the data packet header, leave the complete service data information sent to downstream module processing; S3, service information into PARSER analysis module, by PARSER analysis module will be parsed and integrated original market business information; S4, the parsed market data stream into OPT module, OPT module from the multi-channel market to optimize a complete data exchange by subsequent modules; S5, the preferred market business data into UDP module, by UDP module will be encapsulated into UDP message, and then by 10GMAC module sends to the network.

2. The FPGA-based multi-channel market data parsing and optimization method of claim 1, wherein: Market data is divided into snapshot data and each class data, snapshot data includes securities basic information, market information, price information and market information, each class data includes transaction time, transaction price, transaction volume, transaction direction, transaction identification, transaction status and transaction participants, each class of market data records all the transactions received orders and transaction.

3. The FPGA-based multi-channel market data parsing and optimization method of claim 1, wherein: The core processing steps of the S4 OPT module are: S4.1, the OPT module reads the double channel market business data from the PARSER analysis module of A and B respectively, and gives the double channel business data a timestamp mark TA and TB, and gives the double channel business data a sequence number CNTA and CNTB on each timestamp; S4.2, judge TA=TB=TC, if it is true, it means that the business market timestamp of the last time and the current reading from A and B are equal, at this time further judge whether CNTA and CNTB are equal to CNTC, when equal, send a channel market equal to CNTC value, after sending the market, the timestamp of sending the market is assigned to TC, and the CNT of the channel is assigned to CNTC; S4.3, when TA=TB=TC is not true, judge whether TC is equal to one of the time stamps, if it is equal to one of the time stamps, at this time TB is greater than or less than TC;When TC=TA<TB, it means that B route market update, at this time output B route market, otherwise when TC=TA>TB, it means that B route market time is earlier, at this time output A route market. S4.4, when TA=TB=TC is not true, determine whether TC is equal to one of the timestamps, if it is not equal to any timestamp, at this time, if TC>TA&&TC>TB, it means that the A and B double path market are both lagging behind the current latest time, at this time, no market is sent out, if TB<TC<TA, it means that A market is updated, A market is sent out, if TC<TA<TB, it means that A and B market are both updated, at this time, A should be selected for sending, so as to avoid losing business market data in TA and TB time, if TC<TA and TA=TB, it means that the same updated market at the same time is read, at this time, CNTA and CNTB are compared, and the data of the smaller one is output, so as to avoid losing part of the data at TA=TB time; S4.5, execute the logical judgment of S4.1~S4.4, send the market data to the UDP module, and update TC and CNTC value at the same time, that is, the data reading operation can be performed again, TA, TB, CNTA and CNTB are read, and the logical judgment of S4.1~S4.4 is repeated.

4. The FPGA-based multi-way market data parsing and optimization method of claim 3, wherein: TC is the last timestamp of the market sent out, which is assigned to TC by TA or TB at the same time, CNTC is the count value of the next business data expected to be sent out, which is assigned to CNTC after CNTA or CNTB is incremented at the same time, the initial value of TA, TB and TC is 0, and the initial value of CNTC is 1.

5. The FPGA-based multi-channel market data parsing and optimization method of claim 1, wherein: The complete flow of market data is: market publishing platform flows to market gateway, market gateway flows to low latency switch one, low latency switch one flows to analysis end FPGA, analysis end FPGA flows to low latency switch two, low latency switch two flows to receiving end FPGA, and receiving end FPGA flows to customer receiving end program.

6. The FPGA-based multi-way market data parsing and optimization method of claim 5, wherein: The specific process of market publishing platform flowing to market gateway is: after real-time market information is generated in order matching platform, it is sent out by market publishing platform, and then market data stream is obtained from market publishing platform by market gateway.

7. The FPGA-based multi-channel market data parsing and optimization method of claim 5, wherein: The specific process of market gateway flowing to low latency switch one is: the server where the market gateway is located is directly connected with low latency switch one through gigabit optical fiber, after the original market data obtained from market gateway enters low latency switch one, low latency switch one will output the original market data obtained from one port to multiple ports at the same time for downstream use.

8. The FPGA-based multi-way market data parsing and optimization method of claim 5, wherein: The specific process of low latency switch one flowing to analysis end FPGA is: after the original market data passes through low latency switch one, it enters analysis end FPGA, analysis end FPGA processes the original market data into a standard and unified format according to its own logic, and outputs a complete market outside, analysis end FPGA has two main and standby analysis, a single analysis end FPGA simultaneously accesses double path original data market of different sources, and keeps double active state of two analysis end FPGAs working at the same time.

9. The FPGA-based multi-way market data parsing and optimization method of claim 5, wherein: The specific flow of the stream from the analysis end FPGA to the low-latency switch two is that the market data obtained from the analysis end FPGA enters the low-latency switch two, and the low-latency switch two simultaneously and completely outputs the original market data obtained from one port to multiple ports.

10. The FPGA-based multi-way market data parsing and optimization method of claim 5, wherein: The specific flow of the stream from the low-latency switch two to the receiving end FPGA is that the market data enters the receiving end FPGA through the low-latency switch two, and the receiving end FPGA processes the market data into a standard and unified format according to its own logic.

Images