A binary-based data stream conversion method

Abstract

The application relates to the technical field of data conversion, in particular to a BINARY-based data stream conversion method, which comprises the following steps: (1) generating FID and related parameters; (2) receiving data; (3) generating head and tail fields of the data format; (4) converting and storing the data; and (5) outputting the data. The method converts the data stream of each field of the upstream end which is originally output in a fixed bit number into a BINARY data format output data stream, so that the data stream after hardware decoding can conform to the BINARY data format output, the downstream end can directly receive and decode the data without modification, and the compatibility is higher. The method can improve the compatibility and standardization of the entire data link.

Description

Technical Field

[0001] The present invention relates to the technical field of data conversion, and in particular to a BINARY-based data stream conversion method. Background Art

[0002] The BINARY data format is one of the Shanghai Stock Exchange Market Gateway data interface specifications, namely the Shanghai Stock Exchange Market Gateway BINARY Data Interface Specification; it also defines the mechanism and related data exchange format for transmitting market data through the BINARY protocol to the Shanghai Stock Exchange Market Gateway MDGW.

[0003] All BINARY numeric fields use big-endian encoding. Each BINARY message consists of a header, body, and trailer, with a maximum message length of 8 KB. However, the header, body, and trailer in a BINARY message have inconsistent data types. The data sent by upstream hardware decoding corresponds to a fixed length of 16 Bytes (128 bits) for each BINARY field. Directly sending this fixed-length data stream presents two problems:

[0004] First, the format of data received by the downstream end needs to be modified, which is not conducive to system compatibility;

[0005] Secondly, it will cause data length redundancy, thereby reducing data transmission efficiency and increasing data reception delay.

[0006] Therefore, a new technical solution is urgently needed to solve this problem. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned problems of the prior art and provide a BINARY-based data stream conversion method to solve the problems existing in the traditional technical solution, such as the need to modify the format of the data received by the downstream end, which is not conducive to the compatibility of the system; and the random number problem that easily causes data length redundancy, thereby reducing the efficiency of data transmission and increasing the delay of data reception.

[0008] The above objectives are achieved through the following technical solutions:

[0009] A method for converting a data stream based on BINARY includes the following steps:

[0010] Step (1) Generation of FID and Related Parameters: The FID module generates an FID for each field according to the length of each field in the BINARY data format and the bit width of the BRAM storage module, and generates three parameters corresponding to each other. These three parameters are used to store the original data stream in the BINARY data format in the BRAM storage module;

[0011] Step (2) Data reception: The data input module receives the FAST data stream sent from the upstream FAST hardware decoding system, and sends the received FAST data stream to the data format module and the data conversion and storage module for subsequent processing; the data format module can also receive the STEP data stream sent from the upstream STEP hardware decoding system;

[0012] Step (3) Generation of Header and Tail Fields of Data Format: The data format module fills the BINARY header field according to the STEP data stream, completes the header length field by calculating the number of data in the FAST data stream, and calculates the checksum of the data in the FAST data stream in an 8-bit cumulative manner to complete the generation of the tail field;

[0013] Step (4) Data conversion and storage: The FAST data stream in step (3) is matched one-to-one with the FID in step (1) through the data conversion and storage module, and then converted into the data of each field in the BINARY data format according to the stored address (Addr_offset), position (Byte_ptr) and number of bytes (Byte_cnt) of the field corresponding to the FID, and generates We (BRAM read and write signal), Addr_offset (BRAM write address), and Din (8B) (8B data);

[0014] Step (5) Data output After the entire FAST data stream is completely stored, the data output module outputs the data in the BRAM storage module in sequence according to the address based on the AXIS4 handshake mechanism.

[0015] Furthermore, in step (1), the FID generation module generates a unique field ID for each field in the format according to the BINARY data format rules, which is recorded as FID; the FID conversion module generates the storage address, position and number of bytes of each field in the BARM storage module in a one-to-one correspondence based on the FID and the position and bit width of each field in the BINARY data format.

[0016] Furthermore, the handshake protocol for receiving the FAST data stream in step (2) is AXIS4, and the handshake is performed based on the high level of the valid and ready signals to achieve the operation of receiving data. The value of the last signal is judged to determine whether a single FAST data packet has ended; the number of each FAST data stream is calculated, and then the value of BodyLength is calculated based on this.

[0017] Furthermore, the data format module in step (3) includes a format header generation module and a format tail generation module. The format header generation module generates a header field in BINARY format according to the BINARY interface specification. The header field includes a MsgType field, a SendingTime field and a MsgSeqNum field, wherein the MsgType field is a fixed value M102, the SendingTime field and the MsgSeqNum field are obtained from the STEP data stream, and BodyLength is set to represent the message body length, which does not include the message header and tail. The format tail generation module generates a tail field in BINARY format according to the BINARY interface specification. According to the BINARY interface specification, the value of the tail field (32 bits) is the checksum (8 bits). The checksum is calculated not by adding each field, but by dividing each field into 8 bits, i.e., one byte, and then accumulating them in sequence. When last is pulled high, the last data of the FAST data stream arrives, and the accumulated value of the checksum is assigned to the tail field.

[0018] Furthermore, the data conversion and storage module in step (4) includes a data stream conversion module and a BARM storage module. The data stream conversion module extracts the fields required for the BINARY data format based on the generated FID and the address (Addr_offset), position (Byte_ptr) and number of bytes (Byte_cnt) of the field corresponding to each other, and combines the template adopted by the FAST data stream to convert the data header data, tail data and data obtained from FAST into a BINARY data format data stream so that the data can be subsequently stored in BRAM; the BARM storage module stores the header field, tail field and BINARY data format data stream obtained from FAST output by the data stream conversion module in sequence according to the BINARY data format in the BARM storage module.

[0019] Furthermore, the step (5) is specifically as follows: outputting the BINARY data format data stream stored in the BRAM storage module through the data output module, and outputting it in sequence according to the address order of 64 bits using the AXIS4 protocol.

[0020] Furthermore, since the last data of the BINARY data format data stream is less than 64 bits, the keep signal of the AXIS4 protocol is modified to 8'h0000_0111, and the last signal is output at the same time, indicating that this is the last bit of the data stream.

[0021] Furthermore, the data input module, the data format module, the FID module, the data conversion and storage module, and the data output module are arranged in FPGA or AISC.

[0022] Beneficial effects

[0023] The present invention provides a BINARY-based data stream conversion method that can be applied as an independent module between an upstream STEP / FAST hardware decoding system and a downstream MDS software system with two different protocols, so that the two systems with different protocols can achieve normal communication without any modification. By converting the data stream in which each field at the upstream end is output with an original fixed number of bits into a BINARY data format output data stream, the data stream after hardware decoding can be output in the BINARY data format, and the downstream end can directly receive the decoded data without modification, thereby achieving higher compatibility. The present method can improve the compatibility and standardization of the entire data link. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of a BINARY-based data stream conversion method according to the present invention;

[0025] Figure 2 A corresponding diagram between each field and FID in the BINARY data format of the BINARY-based data stream conversion method described in the present invention;

[0026] Figure 3 The present invention provides a schematic diagram of the storage position of each field in the BINARY data format in the BRAM storage module of the BINARY-based data stream conversion method. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and examples. The described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0028] like Figure 1 As shown, a BINARY-based data stream conversion method converts the data stream in which each field of the upstream end is originally output with a fixed number of bits into a BINARY data format output data stream, so that the data stream after hardware decoding can be output in the BINARY data format, including the following steps:

[0029] Step (1) Generation of FID and Related Parameters: The FID module generates an FID for each field according to the length of each field in the BINARY data format and the bit width of the BRAM storage module, and generates three parameters corresponding to each other. These three parameters are used to store the original data stream in the BINARY data format in the BRAM storage module;

[0030] Step (2) Data reception: The data input module receives the FAST data stream sent from the upstream FAST hardware decoding system, and sends the received FAST data stream to the data format module and the data conversion and storage module for subsequent processing; the data format module can also receive the STEP data stream sent from the upstream STEP hardware decoding system;

[0031] Step (3) Generation of Header and Tail Fields of Data Format: The data format module fills the BINARY header field according to the STEP data stream, completes the header length field by calculating the number of data in the FAST data stream, and calculates the checksum of the data in the FAST data stream in an 8-bit cumulative manner to complete the generation of the tail field;

[0032] Step (4) Data conversion and storage: The FAST data stream in step (3) is matched one-to-one with the FID in step (1) through the data conversion and storage module, and then converted into the data of each field in the BINARY data format according to the stored address (Addr_offset), position (Byte_ptr) and number of bytes (Byte_cnt) of the field corresponding to the FID, and generates We (BRAM read and write signal), Addr_offset (BRAM write address), and Din (8B) (8B data);

[0033] Step (5) Data output After the entire FAST data stream is completely stored, the data output module outputs the data in the BRAM storage module in sequence according to the address based on the AXIS4 handshake mechanism.

[0034] In step (1) of this embodiment, the FID generation module generates a unique field ID for each field in the format according to the BINARY data format rules, which is recorded as FID; the FID conversion module generates the storage address, position and number of bytes of each field in the BARM storage module in a one-to-one correspondence based on the FID and the position and bit width of each field in the BINARY data format.

[0035] Specifically, such as Figure 2 and 3As shown, at the beginning, when FID=1, that is, FID1, the storage address (Addr_offset), position (Byte_ptr) and number of bytes of the field (Byte_cnt) are Addr_offset1=8'd0, Byte_ptr1=3'd0, Byte_cnt1=4;

[0036] When FID=2, i.e. FID2, the storage address (Addr_offset), position (Byte_ptr), and number of bytes of the field (Byte_cnt) are Addr_offset2=8'd0, Byte_ptr2=3'd4, Byte_cnt2=8 respectively;

[0037] And so on, the mapping of the entire BINARY format in BRAM storage is completed.

[0038] The handshake protocol for receiving the FAST data stream in step (2) of this embodiment is AXIS4. The handshake is performed based on the high level of the valid and ready signals to achieve the operation of receiving data. The value of the last signal is judged to determine whether a single FAST data packet has ended; the number of each FAST data stream is calculated, and then the value of BodyLength is calculated based on this.

[0039] The data format module in step (3) of this embodiment includes a format header generation module and a format tail generation module. The format header generation module generates a header field in BINARY format according to the BINARY interface specification. The header field includes a MsgType field, a SendingTime field, and a MsgSeqNum field, wherein the MsgType field is a fixed value M102, the SendingTime field and the MsgSeqNum field are obtained from the STEP data stream, and BodyLength is set to represent the message body length, which does not include the message header and tail. The format tail generation module generates a tail field in BINARY format according to the BINARY interface specification. According to the BINARY interface specification, the value of the tail field (32 bits) is the checksum (8 bits). The checksum is calculated not by adding each field, but by dividing each field into 8 bits, i.e., one byte, and then accumulating them in sequence. When last is pulled high, the last data of the FAST data stream arrives, and the accumulated value of the checksum is assigned to the tail field.

[0040] Specifically, the calculation of the head data length BINARY_LEN - each time AXIS4 handshake, a 128-bit data FAST_DATA[127:0] is received, indicating that there is a fast field input, according to the length Len of the field in the BINARY format definition i (unit: byte - Btye), at this time BINARY_LEN increases Len i , and finally until the entire fast data stream ends, that is, the last signal is pulled high

[0041] The overall formula can be: BINARY LEN =∑ i(last＝1) Len i

[0042] The following is an example of 8-bit cumulative code (using 8-bit cumulative checksum is to be compatible with different data lengths, whether the field is several bytes or not, the checksum can be met)

[0043] The following is a code segment for calculating the checksum:

[0044]

[0045] The calculation of the checksum CHECK_SUM is consistent with the calculation of the head data length BINARY_LEN above - each time AXIS4 handshake, a 128-bit data FAST_DATA i [127:0] is received, indicating that there is a fast field input, according to the length Len of the field in the BINARY format definition i (unit: byte - Byte), then extract the low Len i bit data FAST_DATA i [Len i -1:0] from the current data FAST_DATA i [127:0], at this time the CHECK_SUM i of the current field is accumulated by 8 bits, and finally until the entire fast data stream ends, that is, the last signal is pulled high CHECK_SUM i is accumulated:

[0046] Where L i =Len i / 8;

[0047] Finally, CHECK_SUM =∑ i(last＝1) CHECK_SUM i .

[0048] The data conversion and storage module in step (4) of this embodiment includes a data stream conversion module and a BARM storage module. The data stream conversion module extracts the fields required for the BINARY data format based on the generated FID and the address (Addr_offset), position (Byte_ptr) and number of bytes (Byte_cnt) of the field corresponding to each other, and combines the template adopted by the FAST data stream to convert the data header data, tail data and data obtained from FAST into a BINARY data format data stream so that the data can be subsequently stored in BRAM; the BARM storage module stores the header field, tail field and BINARY data format data stream obtained from FAST output by the data stream conversion module in sequence according to the BINARY data format in the BARM storage module.

[0049] Specifically, each time AXIS4 handshakes, a 128-bit data FAST_DATA is received i [127:0] indicates that there is a fast field input, and the FID value FID in step (1) is obtained according to the position of the field in the BINARY format i , according to the BINARY format definition of the length of the field Len ； (Unit is byte--Byte), then from this data FAST_DATA i [127:0] Extract the low Len i Bit data FAST_DATA i [Len i -1:0], according to FID i You can find the address, location and number of bytes of this field in the BARM memory, and then convert FAST_DATA i [Len i -1:0] is stored in the BARM storage module.

[0050] The step (5) of this embodiment is specifically: outputting the BINARY data format data stream stored in the BRAM storage module through the data output module, and outputting it in sequence according to the address order of 64 bits using the AXIS4 protocol.

[0051] Since the last data of the BINARY data format data stream is less than 64 bits, the keep signal of the AXIS4 protocol is modified to 8'h00000111, and the last signal is output at the same time, indicating that this is the last bit of the data stream.

[0052] In this embodiment, the data input module, the data format module, the FID module, the data conversion and storage module, and the data output module are arranged in FPGA or AISC.

[0053] The above description is only for explaining the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for converting data streams based on BINARY, characterized in that: The steps include: Step (1) Generation of FID and related parameters: The FID module generates an FID for each field according to the length of each field in the BINARY data format and the bit width of the BRAM storage module, and generates three parameters corresponding to each other. These three parameters are used to store the original data stream in the BINARY data format in the BRAM storage module; Step (2) receiving data, receiving the FAST data stream sent by the upstream FAST hardware decoding system through the data input module, and sending the received FAST data stream to the data format module and the data conversion and storage module for subsequent processing; the data format module can also receive the STEP data stream sent by the upstream STEP hardware decoding system; Step (3) Generating the header and tail fields of the data format: the data format module fills the BINARY header field according to the STEP data stream, completes the header length field by calculating the number of data in the FAST data stream, and calculates the checksum of the data in the FAST data stream in an 8-bit cumulative manner to complete the generation of the tail field; Step (4) data conversion and storage: the FAST data stream in step (3) is matched one-to-one with the FID in step (1) through the data conversion and storage module, and then converted into the data of each field in the BINARY data format according to the storage address, position and number of bytes of the field corresponding to the FID, and a BRAM read and write signal, a BRAM write address and 8B data are generated; Step (5) data output: after the entire FAST data stream is completely stored, the data output module outputs the data in the BRAM storage module in sequence according to the address based on the AXIS4 handshake mechanism; In step (1), the FID generation module generates a unique field ID for each field in the format according to the BINARY data format rules, which is recorded as FID; the FID conversion module generates the storage address, position and number of bytes of each field in the BARM storage module in a one-to-one correspondence based on the FID and the position and bit width of each field in the BINARY data format.

2. A method for converting data streams based on BINARY according to claim 1, characterized in that: The handshake protocol for receiving the FAST data stream described in step (2) is AXIS4. The handshake is performed based on the high level of the valid and ready signals to achieve the operation of receiving data. The value of the last signal is used to determine whether a single FAST data packet has ended; the number of each FAST data stream is calculated, and then the value of BodyLength is calculated based on this.

3. A method for converting data streams based on BINARY according to claim 2, characterized in that: The data format module in step (3) includes a format header generation module and a format tail generation module. The format header generation module generates a header field in BINARY format according to the BINARY interface specification. The header field includes a MsgType field, a SendingTime field and a MsgSeqNum field, wherein the MsgType field is a fixed value M102, the SendingTime field and the MsgSeqNum field are obtained from the STEP data stream, and BodyLength is set to represent the message body length, which does not include the message header and tail. The format tail generation module generates a tail field in BINARY format according to the BINARY interface specification. According to the BINARY interface specification, the value of the tail field is the checksum. The checksum is calculated not by adding each field, but by dividing each field into 8 bits, i.e., one byte, and then accumulating them in sequence. When last is pulled high, the last data of the FAST data stream arrives, and the accumulated value of the checksum is assigned to the tail field.

4. The method for converting a binary data stream according to claim 1, wherein: The data conversion and storage module in step (4) includes a data stream conversion module and a BRAM storage module. The data stream conversion module extracts the fields required for the BINARY data format based on the generated FID and the address, position and number of bytes of the field corresponding to the generated FID, and combines the template used by the FAST data stream to convert the data header data, tail data and data obtained from FAST into a BINARY data format data stream so that the data can be subsequently stored in the BRAM; The header field and the tail field output by the data stream conversion module and the BINARY data format data stream obtained from FAST are stored in the BARM storage module in sequence according to the BINARY data format through the BARM storage module.

5. The method for converting a data stream based on BINARY according to claim 4, wherein: The step (5) is specifically as follows: outputting the BINARY data format data stream stored in the BRAM storage module through the data output module, and outputting it in sequence according to the address order of 64 bits using the AXIS4 protocol.

6. A method for converting a data stream based on BINARY according to claim 5, characterized in that: Since the last data of the BINARY data format data stream is less than 64 bits, the keep signal of the AXIS4 protocol is modified to 8'h0000_0111, and the last signal is output at the same time, indicating that this is the last bit of the data stream.

7. The method for converting a data stream based on BINARY according to claim 1, wherein: The data input module, the data format module, the FID module, the data conversion and storage module, and the data output module are arranged in FPGA or AISC.

Images