DPU-based market data communication method and system

Abstract

The invention provides a market data communication method and system based on a DPU, the method is applied to a host end connected with a network card, the method comprises the following steps: receiving a data packet sent by the network card in real time, the data in the data packet comprising exchange data and network card FPGA data; the exchange data comprises exchange timestamps and exchange serial numbers, and the number of data packets in a single time period is determined based on the number of exchange serial numbers between adjacent exchange timestamps; the network card FPGA data comprises a network card FPGA decoding timestamp, and a decoding time point of each data packet in a time period is determined based on the network card FPGA decoding timestamp; and determining data density based on the decoding time point of each data packet in one time period and the number of the data packets in the single time period, and determining whether to send a warning instruction to the network card based on the data density, so that the network card determines whether to send the warning instruction to the exchange terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a market data communication method and system based on DPU. Background Art

[0002] Heterogeneous architecture is a technology that integrates different types of processors or hardware into the same system. For example, a general-purpose processor (such as a CPU) is combined with a special-purpose processor (such as a GPU or FPGA). This architecture can select the most suitable processor for processing according to the characteristics of the task, thereby achieving higher processing efficiency and performance.

[0003] In the modern financial market, trading speed and data processing efficiency are the keys to gaining a competitive advantage. With the increase in market volatility and the complexity of trading algorithms, traditional data processing architectures have become difficult to meet the requirements of ultra-low latency. Heterogeneous architecture has become a key area of focus for cutting-edge solutions because it can optimize performance and efficiency according to task characteristics. In a financial trading system, this architecture significantly reduces data processing latency by processing a large amount of data in parallel.

[0004] In an ultra-low latency heterogeneous trading system, efficient data interaction between the host and the device for market data is crucial. However, in the existing interaction between the host and the device, the host often only receives data passively and it is difficult to correct data errors. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a market data communication method based on DPU to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a market data communication method based on DPU. The method is applied to a host side connected to a network card, and the steps of the method include:

[0007] Receiving in real time data packets sent by the network card, where the data in the data packets includes exchange data and network card FPGA data;

[0008] The exchange data includes an exchange timestamp and an exchange sequence number. Determine the number of data packets in a single time period based on the number of exchange sequence numbers between adjacent exchange timestamps;

[0009] The network card FPGA data includes a network card FPGA decoding timestamp. Determine the decoding time point of each data packet in a time period based on the network card FPGA decoding timestamp;

[0010] Determine the data density based on the decoding time points of each data packet in a time period and the number of data packets in a single time period, and determine whether to send a warning instruction to the network card based on the data density, so that the network card determines whether to issue a warning instruction to the exchange side.

[0011] With the above solution, the exchange timestamp is often issued by the exchange, and there are often a large number of data packets between adjacent timestamps. However, in the existing solutions, only the number of data packets between timestamps can be known. In the actual processing process, the sending of a large number of data packets may be concentrated in a small time period within the time period of adjacent timestamps, resulting in too high data density and easy data loss. In this solution, the decoding time points of each data packet in a time period are determined by the network card FPGA decoding the timestamp, and then the data density is determined. When the data density is too high, a warning instruction is sent to the network card so that the network card determines whether to issue a warning instruction to the exchange side to reduce the data density and reduce the possibility of data loss.

[0012] In some embodiments of the present invention, in the step of determining the data density based on the decoding time points of each data packet in a time period and the number of data packets in a single time period, a single time period is divided into multiple sub-time periods, the number of data packets in each sub-time period is determined based on the network card FPGA decoding timestamp of the data packet, and the number of data packets in each sub-time period is compared with a preset first threshold number to determine whether to calculate the data density of the sub-time period.

[0013] In some embodiments of the present invention, in the step of comparing the number of data packets in each sub-time period with the preset first threshold number to determine whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, the data density is calculated based on the time length of the sub-time period and the number of data packets in the sub-time period.

[0014] In some embodiments of the present invention, in the step of determining whether to send a warning instruction to the network card based on the data density, it is determined whether to send a warning instruction to the network card based on the calculated data density of the sub-time period.

[0015] In some embodiments of the present invention, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes the time range of the corresponding sub-time period and the exchange serial number of the data packets transmitted within the sub-time period.

[0016] In some embodiments of the present invention, the data in the data packet further includes contract data and transaction data. The contract data includes a contract name mapping index value and a contract counter. The steps of the method include determining the contract corresponding to the received transaction data based on the contract name mapping index value, and determining the current number of received contracts based on the contract counter.

[0017] In some embodiments of the present invention, in the step of determining the current number of received contracts based on the contract counter, it is determined whether the count value is incremented sequentially based on the contract quantity count value. If the count value is not incremented sequentially, it is determined that a contract is missing, and the contract name mapping index values corresponding to the contracts before and after the missing contract are fed back to the network card so that the network card retransmits the data of the corresponding contract.

[0018] In some embodiments of the present invention, the data in the data packet further includes a filling flag, which is a flag for the network card to fill in the missing subscribed contract numbers in the contract numbers issued by the exchange. It is determined whether the transaction data corresponding to the contract number is filled data based on the filling flag.

[0019] In some embodiments of the present invention, the transaction data includes price data, trading volume data, bid price data, seller data, and position data. The steps of the method further include classifying and storing the price data, trading volume data, bid price data, seller data, and position data in the transaction data.

[0020] The second aspect of the present invention further provides a market data communication system based on DPU. The system includes a computer device, which includes a processor and a memory. Computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0021] The third aspect of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps implemented by the foregoing market data communication method based on DPU.

[0022] The additional advantages, objectives, and features of the present invention will be partially described in the following description, and will become partially obvious to those of ordinary skill in the art after studying the following text, or can be learned from the practice of the present invention. The objectives and other advantages of the present invention can be pointed out and obtained specifically in the description and the drawings.

[0023] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and the above and other objectives achievable with the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0025] Figure 1 It is a schematic diagram of an embodiment of the market data communication method based on DPU of the present invention;

[0026] Figure 2 It is a schematic diagram of another embodiment of the market data communication method based on DPU of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0028] Herein, it should also be noted that in order to avoid obscuring the present invention due to unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0029] In the existing market data processing system, C++ structures are usually used to define and transfer data. These structures contain various trading-related information, such as price, trading volume, and timestamp, etc. When transferring data between programs, the C++ structures are transferred through memory copying or pointer passing. The specific process includes memory copying and pointer passing. Memory copying means that when data needs to be transferred between different modules or threads, a direct memory copying operation is performed. Although this method is simple, in a high-frequency trading system, frequent memory copying will cause a large performance overhead and increase the system latency. Pointer passing means that in order to reduce the overhead of memory copying, data is transferred through pointer passing. Pointer passing can improve the transfer efficiency, but it is necessary to ensure the correct lifecycle of the data and memory management, otherwise it is easy to cause problems such as memory leaks or illegal access.

[0030] As Figure 1 shown, the present invention proposes a market data communication method based on DPU. The method is applied to the host side connected to the network card, and the steps of the method include:

[0031] In the specific implementation process, the host of this solution is connected to the network card through an interface. The network card is used to receive data packets of market data sent by the data source. The network card is provided with an FPGA heterogeneous system, and the host is provided with a CPU for processing data and a memory for storing data.

[0032] Step S100: Receive the data packets sent by the network card in real time. The data in the data packets includes exchange data and network card FPGA data.

[0033] In some embodiments of the present invention, a DPU is provided in the network card of this solution. The DPU integrates an FPGA (Field-Programmable Gate Array) and provides computing power during the process of network card FPGA data.

[0034] In the specific implementation process, after the network card receives the data packets of market data sent by the data source, it processes the data packets and sends the processed data packets to the host.

[0035] Step S200: The exchange data includes an exchange timestamp and an exchange sequence number. Determine the number of data packets in a single time period based on the number of exchange sequence numbers between adjacent exchange timestamps.

[0036] In the specific implementation process, the exchange timestamp is the timestamp when the exchange sends the data packet, and the exchange sequence number is the sequence number of the data packet sent by the exchange.

[0037] In the specific implementation process, each data packet includes transaction data corresponding to multiple contract numbers.

[0038] Step S300: The network card FPGA data includes a network card FPGA decoding timestamp. Determine the decoding time point of each data packet in a time period based on the network card FPGA decoding timestamp.

[0039] In the specific implementation process, the network card FPGA decoding timestamp is the time point when the FPGA heterogeneous system of the network card decodes the data packet sent by the data source.

[0040] In the specific implementation process, the time length between two adjacent exchange timestamps is used as a time period. The accuracy of the network card FPGA decoding timestamp is greater than the accuracy of the exchange timestamp.

[0041] In some embodiments of the present invention, the accuracy of the network card FPGA decoding timestamp is nanoseconds.

[0042] Step S400: Determine the data density based on the decoding time points of each data packet in a time period and the number of data packets in a single time period, and determine whether to send a warning instruction to the network card based on the data density, so that the network card determines whether to send a warning instruction to the exchange side.

[0043] In the specific implementation process, in the actual processing, although the number of data packets may not be large in a time period, the data packets may be concentrated in a small time segment within a time period, resulting in data congestion and prone to data loss. The prior art can only identify the data packets in a time period and is difficult to conduct further identification. This solution determines the data density based on the decoding time points of each data packet in a time period and the number of data packets in a single time period, and determines whether to send a warning instruction to the network card based on the data density. If a warning instruction is sent, it indicates that the data density is too high and there is a risk of data loss; the network card further feeds back based on the warning instruction to ensure the orderliness of subsequent transmissions and reduce the data density.

[0044] In the prior art, the time stamps of market data usually only come from a single device or system. Some only rely on the exchange time stamps carried in the market information, which may lead to inconsistencies and synchronization problems of time information between heterogeneous architecture systems. In a high-frequency trading system, even a time error at the microsecond level may affect the accuracy and reliability of trading results. The lack of a multi-source time stamp system makes the existing market information unable to carry the time information from different devices when being acquired, increasing the complexity and uncertainty of time calibration in the decision-making system.

[0045] Adopting the above solution, in this solution, the exchange time stamps are often issued by the exchange, and there are often a large number of data packets between adjacent time stamps. However, in the existing solutions, only the number of data packets between time stamps can be known. In the actual processing process, the transmission of a large number of data packets may be concentrated in a small time segment within the time period of adjacent time stamps, resulting in too high data density and prone to data loss. This solution determines the decoding time points of each data packet in a time period through the network card FPGA decoding time stamp, and then determines the data density. When the data density is too high, a warning instruction is sent to the network card, so that the network card determines whether to send a warning instruction to the exchange side to reduce the data density and reduce the possibility of data loss.

[0046] Such as Figure 2As shown, in some embodiments of the present invention, in the step of determining the data density based on the decoding time points of each data packet in a time period and the number of data packets in a single time period, it includes step S410 of dividing a single time period into multiple sub-time periods, determining the number of data packets in each sub-time period based on the network card FPGA decoding timestamps of the data packets, comparing the number of data packets in each sub-time period with a preset first threshold number, and determining whether to calculate the data density of the sub-time period.

[0047] In the specific implementation process, if the number of data packets in a sub-time period is greater than the preset first threshold number, it is determined that the density in this sub-time period may be too high and the data density needs to be accurately calculated. If the number of data packets in a sub-time period is not greater than the preset first threshold number, the possibility of excessive density in this sub-time period is relatively small and accurate calculation of the data density is not required.

[0048] Adopting the above scheme, when the number of data packets in a sub-time period is not greater than the preset first threshold number, the possibility of excessive density in this sub-time period is relatively small and accurate calculation of the data density is not required. Then, this scheme does not need to calculate the data density of all sub-time periods, reducing the calculation amount and improving the processing efficiency.

[0049] In some embodiments of the present invention, in the step of comparing the number of data packets in each sub-time period with the preset first threshold number to determine whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, calculate the data density based on the time length of the sub-time period and the number of data packets in this sub-time period.

[0050] In the specific implementation process, the data density can be the number of data packets per nanosecond.

[0051] In some embodiments of the present invention, in the step of determining whether to send a warning instruction to the network card based on the data density, it includes step S420 of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period.

[0052] In the specific implementation process, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, if the calculated data density of the sub-time period is greater than the preset density threshold, send a warning instruction to the network card.

[0053] In some embodiments of the present invention, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes the time range of the corresponding sub-time period and the transaction sequence number of the data packets transmitted within this sub-time period.

[0054] In some embodiments of the present invention, the step of determining whether to send a warning instruction to the network card based on the data density of the calculated sub-time period further includes that if the calculated sub-time periods for which a warning instruction needs to be issued are consecutive sub-time periods, the time ranges of the consecutive multiple sub-time periods and the transaction serial numbers of the data packets transmitted in the consecutive multiple sub-time periods are merged and sent as one warning instruction.

[0055] With the above solution, when the sub-time periods for which a warning instruction needs to be issued are consecutive sub-time periods, this solution merges and sends multiple warning instructions, and can represent multiple warnings through one warning instruction. On the one hand, it improves the warning sending efficiency, and on the other hand, it can represent the importance of the warning through the time length in one warning instruction.

[0056] In some embodiments of the present invention, the data in the data packet further includes contract data and transaction data. The contract data includes a contract name mapping index value and a contract counter. The steps of the method include determining the contract corresponding to the received transaction data based on the contract name mapping index value, and determining the current number of received contracts based on the contract counter.

[0057] In many existing market data structures, there is a lack of sufficient data identification and serial number information, and at the same time, it is impossible to provide information on the inactivity of specific contract transactions to the strategy module, which makes the implementation of time series-based strategies difficult. The status and order information of the data packets are not clear, resulting in low efficiency in the processing and analysis of the data stream, and increasing the risk of data loss and error handling.

[0058] In some embodiments of the present invention, in the step of determining the current number of received contracts based on the contract counter, it is determined whether the counting value is incremented sequentially based on the contract quantity counting value. If the counting value is not incremented sequentially, it is determined that a contract is missing, and the contract name mapping index values corresponding to the contracts before and after the missing contract are fed back to the network card so that the network card can retransmit the data of the corresponding contract.

[0059] In the specific implementation process, according to the number of received contracts determined by the contract counter in this solution, if the counting value is not incremented sequentially, it indicates that data loss may occur during the processing of the network card. This solution feeds back the contract name mapping index values corresponding to the contracts before and after the missing contract to the network card so that the network card can retransmit the data of the corresponding contract.

[0060] In some embodiments of the present invention, the data in the data packet further includes a padding flag, which is a flag for the network card to pad the missing subscribed contract numbers in the contract numbers issued by the exchange. It is determined whether the transaction data corresponding to the contract number is padded data based on the padding flag.

[0061] In the specific implementation process, since the data source will ignore some inactive contract numbers during the process of sending market data, if the contract number is a subscribed contract number, the FPGA heterogeneous system of the network card will complete the missing data, and this solution uses a completion flag to identify the completed contract number.

[0062] Traditional market data structures are not optimized for streaming processing and real-time analysis. The organization and format of data fields may not be conducive to quick access and processing. This design limits the speed and accuracy of trading decisions, especially in trading environments that require rapid response to market changes. At the same time, in the structure design of existing solutions, the access path of data fields is relatively complex, increasing the time for data processing. For example, nested structures or multi-level pointer access will increase the complexity of data access, affecting the real-time performance and response speed of the system.

[0063] The data structures in the prior art do not consider meticulous optimization of memory alignment, resulting in low storage efficiency of data in memory. Unreasonable data alignment will increase the cache miss rate of the CPU, thereby reducing the data access speed and overall system performance, which is particularly important in data-intensive and processing-intensive trading systems.

[0064] In some embodiments of the present invention, the trading data includes price data, trading volume data, bid price data, seller data, and open interest data, and the steps of the method further include classifying and storing the price data, trading volume data, bid price data, seller data, and open interest data in the trading data.

[0065] Adopting the above solution can ensure storage efficiency through classified storage and reduce access complexity.

[0066] The beneficial effects of this solution include:

[0067] 1. This solution realizes precise time synchronization between different devices and data sources by introducing multi-source timestamps, significantly reducing time errors and improving the timeliness and accuracy of trading decisions;

[0068] 2. This solution enhances the integrity monitoring and sequential tracking of data streams through detailed packet status markings and sequence number fields, effectively preventing data loss and processing errors, and improving the reliability of data.

[0069] 3. This solution designs price and trading volume information fields that are convenient for streaming processing, optimizes the data access path, and improves the efficiency and speed of real-time data processing;

[0070] 4. This solution reduces memory waste, improves cache utilization, reduces processing latency, and enhances the overall performance of the system by optimizing the memory alignment and memory access mode of the data structure.

[0071] An embodiment of the present invention further provides a market data communication system based on a DPU. The system includes a computer device, the computer device includes a processor and a memory, computer instructions are stored in the memory, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0072] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps implemented by the foregoing market data communication method based on a DPU. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0073] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried in a carrier wave.

[0074] It should be clear that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0075] In the present invention, features described and / or illustrated for one embodiment can be used in the same or a similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0076] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A market data communication method based on DPU, characterized in that: The method is applied to a host connected to a network card, and the steps of the method include: Receive data packets sent by the network card in real time, where the data in the data packets include exchange data and network card FPGA data; The exchange data includes an exchange timestamp and an exchange sequence number, and the number of data packets in a single time period is determined based on the number of exchange sequence numbers between adjacent exchange timestamps; The network card FPGA data includes a network card FPGA decoding timestamp, and a decoding time point of each data packet in a time period is determined based on the network card FPGA decoding timestamp; The data density is determined based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, and whether to send a warning instruction to the network card is determined based on the data density, so that the network card determines whether to send a warning instruction to the exchange end.

2. The DPU-based market data communication method according to claim 1, characterized in that: In the step of determining the data density based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, the single time period is divided into multiple sub-time periods, the number of data packets in each sub-time period is determined based on the network card FPGA decoding timestamp of the data packet, the number of data packets in each sub-time period is compared with a preset first threshold number, and it is determined whether to calculate the data density of the sub-time period.

3. The DPU-based market data communication method according to claim 2, characterized in that: In the step of comparing the number of data packets in each sub-time period with a preset first threshold number to determine whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, the data density is calculated based on the time length of the sub-time period and the number of data packets in the sub-time period.

4. The DPU-based market data communication method according to claim 3, characterized in that: In the step of determining whether to send a warning instruction to the network card based on the data density, whether to send a warning instruction to the network card is determined based on the calculated data density of the sub-time period.

5. The DPU-based market data communication method according to claim 4, characterized in that: In the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes a time range corresponding to the sub-time period and a switch sequence number of a data packet transmitted in the sub-time period.

6. The DPU-based market data communication method according to any one of claims 1 to 5, characterized in that: The data in the data packet also includes contract data and transaction data, and the contract data includes a contract name mapping index value and a contract counter. The steps of the method include determining the contract corresponding to the received transaction data based on the contract name mapping index value, and determining the number of currently received contracts based on the contract counter.

7. The DPU-based market data communication method according to claim 6, characterized in that: In the step of determining the number of currently received contracts based on the contract counter, whether the count value is sequentially increasing is determined based on the count value of the number of contracts. If the count value is not sequentially increasing, it is determined that a contract is missing, and the contract name mapping index value corresponding to the contracts before and after the missing contract is fed back to the network card, instructing the network card to retransmit the data of the corresponding contract.

8. The DPU-based market data communication method according to claim 1, characterized in that: The data in the data packet also includes a completion flag, which is a flag used by the network card to complete the missing subscribed contract numbers in the contract numbers published by the exchange, and based on the completion flag, it is determined whether the transaction data corresponding to the contract number is completed data.

9. The DPU-based market data communication method according to claim 6, characterized in that: The transaction data includes price data, transaction volume data, bid price data, seller data and position data. The method further includes classifying and storing the price data, transaction volume data, bid price data, seller data and position data in the transaction data.

10. A market data communication system based on DPU, characterized in that: The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.

Images