A DPU-based market data communication method and system

Abstract

The application provides a DPU-based market data communication method and system, which is applied to a host connected with a network card. The method comprises the following steps: receiving data packets sent by the network card in real time, wherein the data in the data packets comprises exchange data and network card FPGA data; the exchange data comprises an exchange timestamp and an exchange serial number, and the number of data packets in a single time period is determined based on the number of exchange serial numbers between adjacent exchange timestamps; the network card FPGA data comprises a network card FPGA decoding timestamp, and the decoding time point of each data packet in a time period is determined based on the network card FPGA decoding timestamp; the data density is determined based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, and whether to send a warning instruction to the network card is determined based on the data density, so that the network card determines whether to send a warning instruction to the exchange end.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a market data communication method and system based on DPU. BACKGROUND

[0002] Heterogeneous architecture is a technology that integrates different types of processors or hardware into the same system, such as combining general-purpose processors (such as CPUs) with specialized processors (such as GPUs or FPGAs). This architecture can select the most suitable processor for processing according to the characteristics of the task, thereby achieving higher processing efficiency and performance.

[0003] In modern financial markets, transaction speed and data processing efficiency are key to gaining a competitive advantage. With the increase in market volatility and the complexity of trading algorithms, traditional data processing architectures have been unable to meet the demand for ultra-low latency. Heterogeneous architecture has become a key focus area for cutting-edge solutions due to its ability to optimize performance and efficiency based on task characteristics. In financial trading systems, this architecture significantly reduces data processing latency by processing large amounts of data in parallel.

[0004] In ultra-low latency heterogeneous trading systems, efficient data interaction between the host and the device for market data is crucial, but in existing interactions between the host and the device, the host often only passively receives data, making it difficult to correct data errors. SUMMARY

[0005] In view of this, the embodiments of the present application provide a market data communication method based on DPU to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a market data communication method based on DPU, which is applied to a host connected to a network card. The steps of the method include:

[0007] Real-time receiving of data packets sent by the network card, the data in the data packet including exchange data and network card FPGA data;

[0008] The exchange data includes exchange timestamps and exchange serial numbers, and the number of data packets in a single time period is determined based on the number of exchange serial numbers between adjacent exchange timestamps;

[0009] The network card FPGA data includes network card FPGA decoding timestamps, and the decoding time point of each data packet in a time period is determined based on the network card FPGA decoding timestamps;

[0010] The data density is determined based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, and whether to send a warning instruction to the network card is determined based on the data density, so that the network card determines whether to send a warning instruction to the exchange end.

[0011] By using the above scheme, the exchange timestamp is usually sent by the exchange, and there are usually a large number of data packets between adjacent timestamps. However, in the existing scheme, only the number of data packets between timestamps can be known, and in actual processing, the transmission of a large number of data packets may be concentrated in a small time period in the time period of adjacent timestamps, causing the data density to be too high and data loss to occur easily. The scheme determines the decoding time point of each data packet in a time period by decoding the timestamp of the network card FPGA, and then determines the data density. When the data density is too high, a warning instruction is sent to the network card, so that the network card determines whether to send a warning instruction to the exchange end to reduce the data density and the possibility of data loss.

[0012] In some embodiments of the present application, in the step of determining the data density based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, the single time period is divided into a plurality of sub-time periods, the number of data packets in each sub-time period is determined based on the network card FPGA decoding timestamp of the data packet, and the number of data packets in each sub-time period is compared with a preset first threshold number to determine whether to calculate the data density of the sub-time period.

[0013] In some embodiments of the present application, in the step of comparing the number of data packets in each sub-time period with a preset first threshold number to determine whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, the data density is calculated based on the time length of the sub-time period and the number of data packets in the sub-time period.

[0014] In some embodiments of the present application, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, it is determined whether to send a warning instruction to the network card based on the calculated data density of the sub-time period.

[0015] In some embodiments of the present application, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes the time range of the corresponding sub-time period and the exchange serial number of the data packet transmitted in the sub-time period.

[0016] In some embodiments of the present application, the data in the data packet further comprises contract data and transaction data, the contract data comprises a contract name mapping index value and a contract counter, and the method further comprises determining a contract corresponding to the received transaction data based on the contract name mapping index value and determining a number of currently received contracts based on the contract counter.

[0017] In some embodiments of the present application, in the step of determining the number of currently received contracts based on the contract counter, it is determined whether the number of contracts is sequentially increased based on the number of contract counter values, and if the number of contracts is not sequentially increased, it is determined that a contract is missing, and the contract name mapping index values corresponding to the contracts before and after the missing contract are fed back to the network card to enable the network card to retransmit the data of the corresponding contract.

[0018] In some embodiments of the present application, the data in the data packet further comprises a completion flag, the completion flag is a flag for the network card to complete the contract number missing in the contract number issued by the transaction, and it is determined whether the transaction data corresponding to the contract number is completion data based on the completion flag.

[0019] In some embodiments of the present application, the transaction data comprises price data, transaction volume data, buy price data, seller data and position volume data, and the method further comprises classifying and storing the price data, transaction volume data, buy price data, seller data and position volume data in the transaction data.

[0020] The second aspect of the present application further provides a DPU-based market data communication system, which comprises a computer device, the computer device comprises a processor and a memory, the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory, and the system implements the steps of the method as described above.

[0021] The third aspect of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the DPU-based market data communication method as described above.

[0022] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will in part be apparent to those of ordinary skill in the art upon examination of the following or can be learned from practice of the application. The objectives and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0023] Those skilled in the art will understand that the objects and advantages of the application can be realized and attained by means of the application especially designed and described herein below. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0025] Figure 1 A schematic diagram of an embodiment of the DPU-based market data communication method of the application;

[0026] Figure 2 A schematic diagram of another embodiment of the DPU-based market data communication method of the application. DETAILED DESCRIPTION

[0027] To make the objects, technical solutions and advantages of the application clearer, the application will be further described below in conjunction with the embodiments and drawings. Herein, the illustrative embodiments of the application and the descriptions thereof are used to explain the application but not to limit the application.

[0028] It should also be noted that, in order not to obscure the application with unnecessary details, only the structures and / or processing steps closely related to the solutions according to the application are shown in the drawings, and other details not closely related to the application are omitted.

[0029] In the existing market data processing system, C++ structures are usually used to define and transmit data. These structures contain various transaction-related information, such as price, transaction volume and timestamp, etc. When transmitting data between programs, C++ structures are transmitted by means of memory copying or pointer transmission. The specific process includes memory copying and pointer transmission. Memory copying refers to the operation of directly copying memory when data needs to be transmitted between different modules or threads. Although this method is simple, frequent memory copying in a high-frequency trading system will result in a large performance overhead and increase the system latency. Pointer transmission refers to the transmission of data through pointers in order to reduce the overhead of memory copying. Pointer transmission can improve transmission efficiency, but it is necessary to ensure the correctness of the data life cycle and memory management, otherwise it is easy to cause memory leakage or illegal access problems.

[0030] As shown in Figure 1 The application provides a DPU-based market data communication method. The method is applied to a host connected to a network card. The steps of the method include:

[0031] In the specific implementation process, the host end of the scheme is connected with a network card through an interface, the network card is used for receiving data packets of market data sent by a data source, the network card is provided with an FPGA heterogeneous system, and the host end is provided with a CPU used for processing data and a memory used for storing data.

[0032] In step S100, data packets sent by the network card are received in real time, and the data in the data packets includes exchange data and network card FPGA data.

[0033] In some embodiments of the application, the network card of the scheme is provided with a DPU, the DPU integrates an FPGA (Field-Programmable Gate Array, Field-Programmable Gate Array), and provides computing power in the process of network card FPGA data.

[0034] In the specific implementation process, after receiving the data packets of market data sent by the data source, the network card processes the data packets and sends the processed data packets to the host end.

[0035] In step S200, the exchange data includes an exchange timestamp and an exchange serial number, and the number of data packets in a single time period is determined based on the number of exchange serial numbers between adjacent exchange timestamps.

[0036] In the specific implementation process, the exchange timestamp is the timestamp of the data packet sent by the exchange, and the exchange serial number is the serial number of the data packet sent by the exchange.

[0037] In the specific implementation process, each data packet includes transaction data corresponding to a plurality of contract numbers.

[0038] In step S300, the network card FPGA data includes a network card FPGA decoding timestamp, and a decoding time point of each data packet in a time period is determined based on the network card FPGA decoding timestamp.

[0039] In the specific implementation process, the network card FPGA decoding timestamp is the time point at which the FPGA heterogeneous system of the network card decodes the data packet sent by the data source.

[0040] In the specific implementation process, the length of time between two adjacent exchange timestamps is taken as a time period, and the precision of the network card FPGA decoding timestamp is higher than that of the exchange timestamp.

[0041] In some embodiments of the application, the precision of the network card FPGA decoding timestamp is nanoseconds.

[0042] Step S400, determining the data density based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, determining whether to send a warning instruction to the network card based on the data density, so that the network card determines whether to send a warning instruction to the exchange end.

[0043] In the specific implementation process, although the number of data packets in a time period may not be large, the data packets may be concentrated in a small time period in a time period, causing data congestion and easy data loss. The prior art can only identify the data packets in a time period, and it is difficult to further identify. The present scheme determines the data density by the decoding time point of each data packet in a time period and the number of data packets in a single time period, and determines whether to send a warning instruction to the network card through the data density. If a warning instruction is sent, it indicates that the data density is too large and there is a risk of data loss. The network card further feeds back to the data source based on the warning instruction to ensure the orderliness of subsequent sending and reduce the data density.

[0044] In the prior art, the time stamp of market data is usually only from a single device or system, and some only rely on the exchange time stamp carried in the market information, which may cause inconsistency and synchronization problems of time information between heterogeneous architecture systems. In high-frequency trading systems, even a microsecond-level time error may affect the accuracy and reliability of the trading results. The lack of a multi-source time stamp system makes the existing market information unable to carry time information from different devices when it is acquired, increasing the complexity and uncertainty of the time calibration of the decision system.

[0045] With the above scheme, in the present scheme, the exchange time stamp is usually sent by the exchange, and there are usually a large number of data packets between adjacent time stamps. However, in the existing scheme, only the number of data packets between time stamps can be known, and in the actual processing process, the sending of a large number of data packets may be concentrated in a small time period in the time period of adjacent time stamps, causing high data density and easy data loss. The present scheme determines the decoding time point of each data packet in a time period by the network card FPGA decoding time stamp, and further determines the data density. When the data density is too high, a warning instruction is sent to the network card, so that the network card determines whether to send a warning instruction to the exchange end to reduce the data density and reduce the possibility of data loss.

[0046] As Figure 2As shown, in some embodiments of the present application, in the step of determining the data density based on the decoding time point of each data packet in a time period and the number of data packets in a single time period, the step S410 is included, that is, dividing the single time period into a plurality of sub-time periods, determining the number of data packets in each sub-time period based on the network card FPGA decoding timestamp of the data packets, comparing the number of data packets in each sub-time period with the preset first threshold number, and determining whether to calculate the data density of the sub-time period.

[0047] In the specific implementation process, if the number of data packets in the sub-time period is greater than the preset first threshold number, it is determined that the sub-time period may have a condition of excessive density, and the data density needs to be accurately calculated. If the number of data packets in the sub-time period is not greater than the preset first threshold number, the sub-time period has a smaller possibility of excessive density, and the data density does not need to be accurately calculated.

[0048] By using the above scheme, when the number of data packets in the sub-time period is not greater than the preset first threshold number, the sub-time period has a smaller possibility of excessive density, and the data density does not need to be accurately calculated. Therefore, the present scheme does not need to calculate the data density of all sub-time periods, thereby reducing the calculation amount and improving the processing efficiency.

[0049] In some embodiments of the present application, in the step of comparing the number of data packets in each sub-time period with the preset first threshold number and determining whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, the data density is calculated based on the time length of the sub-time period and the number of data packets in the sub-time period.

[0050] In the specific implementation process, the data density can be the number of data packets per nanosecond.

[0051] In some embodiments of the present application, in the step of determining whether to send a warning instruction to the network card based on the data density, the step S420 is included, that is, determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period.

[0052] In the specific implementation process, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, if the calculated data density of the sub-time period is greater than a preset density threshold, a warning instruction is sent to the network card.

[0053] In some embodiments of the present application, in the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes the time range of the corresponding sub-time period and the transaction serial number of the data packets transmitted in the sub-time period.

[0054] In some embodiments of the present application, the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period further comprises, if the sub-time period for which the warning instruction needs to be sent is a continuous sub-time period, merging the time range of the continuous sub-time periods and the transaction sequence number of the data packets transmitted in the continuous sub-time periods as a warning instruction.

[0055] With the above scheme, when the sub-time period for which the warning instruction needs to be sent is a continuous sub-time period, the scheme sends multiple warning instructions in a merged manner, which can indicate multiple warnings through one warning instruction, thereby improving the efficiency of sending warning instructions and indicating the importance of the warning through the length of the warning instruction.

[0056] In some embodiments of the present application, the data in the data packet further comprises contract data and transaction data, the contract data comprises a contract name mapping index value and a contract counter, and the method comprises the steps of determining the contract corresponding to the received transaction data based on the contract name mapping index value and determining the number of currently received contracts based on the contract counter.

[0057] In many existing market data structures, there is a lack of sufficient data identification and sequence number information, and specific contract transaction inactivity information cannot be provided to the strategy module, which makes it difficult to implement time series type strategies. The state and order information of the data packet is not clear, which leads to low efficiency in processing and analyzing the data stream and increases the risk of data loss and error processing.

[0058] In some embodiments of the present application, in the step of determining the number of currently received contracts based on the contract counter, it is determined whether the count value of the number of contracts is sequentially increasing based on the count value, and if the count value is not sequentially increasing, it is determined that a contract is missing, and the contract name mapping index values corresponding to the contracts before and after the missing contract are fed back to the network card to enable the network card to retransmit the data of the corresponding contract.

[0059] In the specific implementation process, the number of received contracts determined by the contract counter, if the count value is not sequentially increasing, indicates that data loss may have occurred in the processing of the network card, and the present application feeds back the contract name mapping index values corresponding to the contracts before and after the missing contract to the network card to enable the network card to retransmit the data of the corresponding contract.

[0060] In some embodiments of the present application, the data in the data packet further comprises a padding flag, the padding flag is a flag for the network card to pad the missing contract number of the contract number issued by the exchange, and it is determined whether the transaction data corresponding to the contract number is padding data based on the padding flag.

[0061] In the implementation process, since the data source ignores part of the inactive contract number in the process of sending market data, if the contract number is the subscribed contract number, the FPGA heterogeneous system of the network card will complete the missing data. The present scheme identifies the completed contract number through the completion flag.

[0062] The traditional market data structure is not optimized for stream processing and real-time analysis. The organization and format of data fields may not be conducive to fast access and processing. This design limits the speed and accuracy of trading decisions, especially in trading environments that require quick responses to market changes. At the same time, in the structure design of the existing scheme, the access path of the data field is complex, which increases the data processing time. For example, nested structures or multi-layer pointer access increase the complexity of data access, affecting the real-time performance and response speed of the system.

[0063] The data structure in the prior art does not consider the detailed optimization of memory alignment, resulting in low storage efficiency of data in memory. Unreasonable data alignment increases the CPU cache miss rate, thereby reducing data access speed and overall system performance, which is particularly important in data-intensive and processing-intensive trading systems.

[0064] In some embodiments of the present application, the transaction data includes price data, transaction volume data, buy price data, seller data, and position volume data, and the method further comprises classifying and storing the price data, transaction volume data, buy price data, seller data, and position volume data in the transaction data.

[0065] By using the above scheme, the storage efficiency can be ensured by classified storage, and the access complexity is reduced.

[0066] The beneficial effects of the present scheme include:

[0067] 1. The present scheme introduces multiple source timestamps to achieve accurate time synchronization between different devices and data sources, significantly reducing time errors and improving the timeliness and accuracy of trading decisions.

[0068] 2. The present scheme enhances data stream integrity monitoring and sequence tracking through detailed data packet state markers and sequence number fields, effectively preventing data loss and processing errors, and improving data reliability.

[0069] 3. The present scheme designs price and transaction volume information fields suitable for stream processing, optimizes data access paths, and improves real-time data processing efficiency and speed.

[0070] 4. The present scheme optimizes memory alignment and memory access mode of the data structure, reduces memory waste, improves cache utilization, reduces processing delay, and improves overall system performance.

[0071] The embodiment of the present application also provides a DPU-based market data communication system, which comprises a computer device, the computer device comprises a processor and a memory, the memory stores computer instructions, and the processor is used for executing the computer instructions stored in the memory, and the system realizes the steps of the method as described above.

[0072] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the DPU-based market data communication method as described above. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0073] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The actual implementation depends on the specific application and design constraints imposed on the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link.

[0074] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0075] In the present application, the features described and / or exemplified for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of the features of other embodiments.

[0076] The above merely illustrates the preferred embodiments of the present application, and is not used to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of protection of the present application.

Claims

1. A DPU-based market data communication method, characterized in that, The method is applied to a host connected with a network card, and steps of the method include: Real-time receiving of a data packet sent by the network card, data in the data packet including exchange data and network card FPGA data; The exchange data including an exchange timestamp and an exchange serial number, and a number of data packets in a single time period being determined based on a number of exchange serial numbers between adjacent exchange timestamps; The network card FPGA data including a network card FPGA decoding timestamp, and a decoding time point of each data packet in a single time period being determined based on the network card FPGA decoding timestamp; Based on the decoding time point of each data packet in a single time period and the number of data packets in the single time period, data density being determined, and based on the data density, whether to send a warning instruction to the network card being determined, so that the network card determines whether to send a warning instruction to an exchange end.

2. The DPU-based market data communication method of claim 1, wherein, In the step of determining the data density based on the decoding time point of each data packet in a single time period and the number of data packets in the single time period, the single time period is divided into a plurality of sub-time periods, the number of data packets in each sub-time period is determined based on the network card FPGA decoding timestamp of the data packet, and whether to calculate the data density of the sub-time period is determined by comparing the number of data packets in each sub-time period with a preset first threshold number. 3.The DPU-based market data communication method of claim 2, wherein, In the step of comparing the number of data packets in each sub-time period with the preset first threshold number and determining whether to calculate the data density of the sub-time period, if the number of data packets in any sub-time period is greater than the first threshold, the data density is calculated based on the time length of the sub-time period and the number of data packets in the sub-time period.

4. The DPU-based market data communication method of claim 3, wherein, In the step of determining whether to send a warning instruction to the network card based on the data density, whether to send a warning instruction to the network card is determined based on the calculated data density of the sub-time period.

5. The DPU-based market data communication method according to claim 4, wherein, In the step of determining whether to send a warning instruction to the network card based on the calculated data density of the sub-time period, the warning instruction includes a time range corresponding to the sub-time period and exchange serial numbers of data packets transmitted in the sub-time period.

6. The DPU-based market data communication method according to any one of claims 1 to 5, characterized in that, The data in the data packet further includes contract data and transaction data, the contract data including a contract name mapping index value and a contract counter, and steps of the method include determining a contract corresponding to received transaction data based on the contract name mapping index value and determining a number of currently received contracts based on the contract counter.

7. The DPU-based market data communication method of claim 6, wherein, In the step of determining the number of currently received contracts based on the contract counter, whether the number of contracts is sequentially increased is determined based on a number count value of the contracts, if the number count value is not sequentially increased, it is determined that a contract is missing, contract name mapping index values corresponding to contracts before and after the missing contract are fed back to the network card, and the network card is instructed to perform data retransmission of the corresponding contract.

8. The DPU-based market data communication method of claim 1, wherein, The data in the data packet further includes a padding flag, the padding flag being a flag for padding a missing contract number in a contract number published by an exchange by the network card, and whether transaction data corresponding to the contract number is padding data is determined based on the padding flag. 9.The DPU-based market data communication method of claim 6, wherein, The transaction data includes price data, volume data, bid data, seller data and position size data, and the method further includes classifying and storing the price data, volume data, bid data, seller data and position size data in the transaction data.

10. A DPU-based market data communication system, characterized by, The system comprises a computer device comprising a processor and a memory, the memory having computer instructions stored therein, the processor being configured to execute the computer instructions stored in the memory, and the system implements the steps as claimed in any one of claims 1-9 when the computer instructions are executed by the processor.

Images