A method for generating market dbf files based on reflection mechanism

Abstract

The present invention provides a method for generating a market dbf file based on a reflection mechanism, which relates to the technical field of market data processing. The method includes the following steps: S1. Start the main thread and read the metadata in the configuration file. The metadata includes the configuration of each field in the dbf and the list of market message topics in the kafka cluster; S2. The main thread connects to the kafka cluster, subscribes to the data of multiple market topics according to the list, and receives the real-time market messages pushed by the kafka cluster; S3. Parse the real-time market messages into corresponding pb market messages, obtain the values of each dbf field, and update the internal dbf cache record; S4. Start a timed thread to refresh the dbf cache record to a disk file and generate a dbf file, which realizes supporting the adjustment of the field order, as well as adding and / or deleting fields, obtaining the data of each field through the pb reflection mechanism, formatting it to the record buffer, refreshing the dbf cache record to the disk file, and generating a dbf file.

Description

Technical Field

[0001] The present invention relates to the technical field of market data processing. Specifically, the present invention relates to a method for generating a market dbf file based on the reflection mechanism. Background Art

[0002] In the stock market, generating a dbf file is mainly based on the real-time market information and order book information written into the kafka streaming system by the market decoding service system. Data such as security codes, opening prices, previous closing prices, order book buying and selling prices and quantities are written into the specified dbf file. The trading counter reads the dbf file and displays each field on the trading interface for traders to refer to when placing orders.

[0003] The existing methods for generating dbf files need to perform targeted parsing and processing for the market message formats of each exchange, and the order between the fields in the dbf file must be determined in advance. Once there are changes in the field order or new fields are added and / or deleted, the code of the dbf file generation service needs to be modified, recompiled and deployed. It is not flexible enough to meet trading requirements and affects market trading. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a method for generating a market dbf file based on the reflection mechanism to solve the above technical problems.

[0005] The technical method adopted by the present invention to solve its technical problems is: a method for generating a market dbf file based on the reflection mechanism, which is improved in that it includes the following steps: S1. Start the main thread and read the metadata in the configuration file. The metadata includes the configuration of each field in the dbf and the list of market message topics in the kafka cluster; S2. The main thread connects to the kafka cluster, subscribes to the data of multiple market topics according to the list, and receives the real-time market messages pushed by the kafka cluster; S3. Parse the real-time market messages into corresponding pb market messages, obtain the values of each dbf field, and update the internal dbf cache record. Pb is protobuf; S4. Start a timing thread to refresh the dbf cache record to a disk file and generate a dbf file.

[0006] In the above method, step S3 includes the following steps:

[0007] S31. Query and obtain the pb message name written into the kafka cluster according to the list;

[0008] S32. Dynamically generate an empty message object through the reflection mechanism of pb, parse each piece of kafka message data, and obtain the latest market pb message object;

[0009] S33. Obtain the value of the key_field_name field from the market pb message object to get the security code of this market message. key_field_name is the name of the security code field in the pb message. Obtain the corresponding dbf message record from the message record cache through this security code. When the security code exists in the message record cache, return the dbf message record. When the security code does not exist in the message record cache, create a new dbf message record;

[0010] S34. Process each field in dbf_fields in a loop. dbf_fields are the dbf fields. Obtain the data mapped by each field in the pb message. When the data is of string type, directly copy it to the corresponding offset in the dbf buffer according to the specified length. When the data is of integer type, obtain the real floating-point data according to price_base, format it as a string, and output it to the dbf buffer. price_base is the price base.

[0011] In the above method, step S4 includes the following steps:

[0012] S41. The timing thread accesses the external interface before the market opens to obtain the trading calendar of this market, and gets the trading days and trading time periods of this market;

[0013] S42. Delete all the remaining data of the previous day in the dbf cache records, and record the latest trading date;

[0014] S43. Regularly write all the dbf records in the dbf cache records to a disk file in the dbf file format to generate a dbf file.

[0015] In the above method, step S43 includes the following steps:

[0016] S431. Write the file header information to the disk file;

[0017] S432. Use the read record metadata to write the description information of each record to the disk file;

[0018] S433. Write each dbf data record to the disk file to obtain the final dbf file.

[0019] The beneficial effects of the present invention are as follows: By defining all field metadata of the dbf file through a configuration file, it supports adjusting the field order, as well as adding and / or deleting fields; By defining the mapping relationship between the market quotation topics to be subscribed and pb messages in the form of a configuration file, the pb reflection mechanism is used to dynamically parse any market quotation message of a topic; Each dbf field is associated with the corresponding field in the pb message protocol, and the data of each field is obtained through the pb reflection mechanism and formatted into a record buffer; Through an independent timing thread, the record metadata, dbf field description information, and real market quotation data are written into the dbf file according to the field order defined in the configuration file to obtain the final dbf file. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Appendix Figure 1 It is a flowchart of a method for generating a market dbf file based on the reflection mechanism of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The present invention will be further described below in conjunction with the drawings and embodiments.

[0022] The concept, specific structure and technical effects generated by the present invention will be clearly and completely described below in conjunction with the embodiments and drawings to fully understand the purpose, features and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention. In addition, all connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the more optimal connection structure that can be formed by adding or reducing connection accessories according to the specific implementation situation. Each technical feature in the present invention can be combined interactively without conflicting with each other.

[0023] Refer to Figure 1 As shown, a method for generating a market dbf file based on the reflection mechanism of the present invention includes the following steps:

[0024] S1. Start the main thread and read the metadata in the configuration file. The metadata includes the configuration of each field in the dbf and the list of market quotation message topics of the kafka cluster. Dbf is the dBase Table Format File, a database file storage method with a specific file format, mainly including a file header + record metadata + real record data. Kafka is a high-throughput distributed publish-subscribe message system;

[0025] S2. The main thread connects to the kafka cluster, subscribes to the data of multiple market quotation topics according to the list, and receives the real-time market quotation messages pushed by the kafka cluster in a callback manner;

[0026] S3. Parse the real-time market message into the corresponding pb market message, obtain the value of each dbf field, and update the internal dbf cache record;

[0027] Specifically, step S3 includes the following steps:

[0028] S31. According to the list, query the name of the pb message written into the kafka cluster. Pb is protobuf, which is a mechanism for automatic serialization and deserialization of data open-sourced by Google;

[0029] S32. Dynamically generate an empty message object through the reflection mechanism of pb, parse each piece of kafka message data, and obtain the latest market pb message object;

[0030] S33. Obtain the value of the key_field_name field from the market pb message object to get the security code of this market message. Obtain the corresponding dbf message record from the message record cache through this security code. When the security code exists in the message record cache, return the dbf message record. When the security code does not exist in the message record cache, create a new dbf message record;

[0031] key_field_name is the name of the security code field in the pb message, which can be understood as the keyword field name in the protobuf message. After deserializing the data consumed from the real-time market kafka topic, there is usually a field named id.code, which records which security code this market message belongs to. The dbf service will obtain the name of this security code and use it as the key of the map index to find the message structure of this code in memory, and then update the message structure with the protobuf message content;

[0032] S34. Each dbf message record is essentially a buffer. Its total record length can be obtained by accumulating the lengths of all fields, and the offset of each field in this buffer can be obtained by accumulating the previous fields. Therefore, each field in dbf_fields (i.e., dbf fields) can be processed cyclically to obtain the data mapped to it in the pb message. When the data is of string type, it is directly copied to the corresponding offset in the dbf buffer according to the specified length, that is, the data value in the protobuf message is taken out and assigned to the cache structure in memory. For example, when receiving the market quote of US stock AAPL, the code AAPL is copied to the CODE column in string format, and the market time time is copied to the DEALTIME column in integer format; when the data is of integer type, the real floating-point data is obtained according to price_base, then formatted as a string and output to the dbf buffer. price_base is the price base, mainly used to solve the floating-point precision problem. Assuming the value of price_base is 10000, then the floating-point number 1.2345 will get the value 10000 * 1.2345 = 12345. Here, formatting means that when the data is of integer type, it is represented by 4 bytes or 8 bytes in memory, but it needs to be shown as a string in dbf, so the integer needs to be formatted as a string and finally output to the dbf file;

[0033] There is a mapping relationship between the fields in the dbf file and the fields in the protobuf message, that is, the fields in the dbf file correspond to a certain field in the protobuf message. For example, the dbf file has a column of latest price field, which corresponds to the price field in the protobuf protocol; the dbf file has a column of the price of the first-level order book field, which corresponds to position[0].price in the protobuf protocol;

[0034] Through the above process, we can receive messages from multiple kafka topics and write the corresponding market data into the dbf cache record.

[0035] S4. Start a timed thread to refresh the dbf cache record to a disk file and generate a dbf file;

[0036] Specifically, step S4 includes the following steps

[0037] S41. The timed thread accesses an external interface to obtain the trading calendar of the market before the daily opening (for example, the Hong Kong stock market selects 8 o'clock, and the US stock market selects 15:00 Beijing time) to obtain the trading days and trading time periods of the market;

[0038] S42. After obtaining the real trading calendar, the timing thread executes a loop every second. If the current time is close to the opening time, all the remaining data of the previous day in the dbf cache records is deleted to prepare for receiving data of the new day, and the latest trading date is recorded.

[0039] S43. The timing thread regularly (for example, every 3 seconds) writes all the dbf records in the dbf cache records to a disk file in the dbf file format to generate a dbf file. According to the trading calendar configuration, the market conditions do not change after the market closes, so there is no need to write anymore. Specifically, the step S43 includes the following steps: S431. Write the file header information to the disk file; S432. Use the read record metadata to write the description information of each record to the disk file. Specifically, for the record, it is what the name of each column in the dbf file is, what the data type is, floating-point precision, etc., which are all specified in the configuration file. The description information is the metadata of the data column; S433. Write each dbf data record to the disk file to obtain the final dbf file, realizing the use of the pb reflection mechanism to parse the value of any message field according to the message type name, and obtaining the names of the pb message types associated with multiple kafka topics to be read through the configuration file, and writing all the field data to the corresponding buffer positions of each dbf data record in the configured field order, and regularly writing all the records to the dbf file.

[0040] The present invention defines all field metadata of the dbf file through a configuration file, supports adjusting the field order, as well as adding and / or deleting fields; defines the mapping relationship between the market conditions topics to be subscribed and the pb messages through the configuration file, and uses the pb reflection mechanism to dynamically parse any topic market conditions message; each dbf field is associated with the corresponding field in the pb message protocol, and the data of each field is obtained through the pb reflection mechanism and formatted into the record buffer; the record metadata, dbf field description information, and real market conditions data are written to the dbf file in the field order defined in the configuration file through an independent timing thread to obtain the final dbf file.

[0041] The above is a specific description of the preferred embodiment of the present invention, but the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for generating a market dbf file based on the reflection mechanism, characterized in that: It includes the following steps: S1. Start the main thread, read the metadata in the configuration file. The metadata includes the configuration of each field in the dbf and the list of kafka cluster market message topics; S2. The main thread connects to the kafka cluster, subscribes to the data of multiple market topics according to the list, and receives the real-time market messages pushed by the kafka cluster; S3. Parse the real-time market messages into corresponding pb market messages, obtain the values of each dbf field, and update the internal dbf cache record. Pb is protobuf; Step S3 includes the following steps: S31. Query and obtain the pb message name written to the kafka cluster according to the list; S32. Dynamically generate an empty message object through the reflection mechanism of pb, parse each piece of kafka message data, and obtain the latest market pb message object; S33. Obtain the value of the key_field_name field from the market pb message object to get the security code of the corresponding market message. key_field_name is the name of the security code field in the pb message. Obtain the corresponding dbf message record from the message record cache through this security code. When the security code exists in the message record cache, return the dbf message record. When the security code does not exist in the message record cache, create a new dbf message record; S34. Process each field in dbf_fields in a loop. dbf_fields are the dbf fields. Obtain the data mapped by each field in the pb message. When the mapped data is of string type, directly copy it to the corresponding offset in the dbf buffer according to the specified length. When the mapped data is of integer type, obtain the real floating-point data according to price_base, format it as a string, and output it to the dbf buffer. price_base is the price base; S4. Start a timing thread to refresh the dbf cache record to the disk file and generate a dbf file.

2. The method for generating a market dbf file based on the reflection mechanism according to claim 1, characterized in that: Step S4 includes the following steps: S41. The timing thread accesses the external interface before the market opens to obtain the trading calendar of the market, and obtains the trading days and trading time periods of the market; S42. Delete all the remaining data of the previous day in the dbf cache record and record the latest trading date; S43. Regularly write all the dbf records in the dbf cache record to the disk file in the dbf file format to generate a dbf file.

3. The method for generating a market dbf file based on the reflection mechanism according to claim 2, wherein: Step S43 includes the following steps: S431. Write the file header information to the disk file; S432. Write the description information of each record to the disk file using the read record metadata; S433. Write each dbf data record to the disk file to obtain the final dbf file.

Images