Data processing method, system and apparatus

Abstract

The embodiment of the present specification provides a data processing method, system and device, wherein the data processing method comprises: receiving market snapshot data; extracting order data and price data from the market snapshot data, performing step-by-step compression on the order data according to order quantity to obtain target order data; writing the target order data into an order file to obtain order structured data, and writing the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein the price structured data comprises address information of order information corresponding to each price level in the order structured data.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to data processing methods, systems and apparatus. Background Technology

[0002] In the financial sector, 1000-level market data, compared to standard market data (which typically displays buy 1 to buy 5 and sell 1 to sell 5, five levels each), is a more detailed market data service. It can display order information for all price levels from the limit down to the limit up, up to 1000 levels. The snapshot generated by 1000-level market data can store order information for all price levels and record all order queue information for each price level, thus supporting the needs of high-frequency trading, market analysis, and regulatory compliance. In practical applications, due to the large data volume of a single 1000-level market data transaction, the snapshot contains many levels and has a large depth. Furthermore, the 1000-level market data snapshot service typically requires building a snapshot every few seconds, which means that saving it to disk using traditional databases requires a large amount of disk space, significantly increasing storage costs. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0005] Receive market snapshot data;

[0006] Extract order data and price data from the market snapshot data, and compress the order data step by step according to the order volume to obtain the target order data;

[0007] The target order data is written to an order file to obtain structured order data, and the price data is written to a price file to obtain structured price data, which serves as a storage operation for the market snapshot data;

[0008] The price structured data includes the address information of the order information corresponding to each price tier in the order structured data.

[0009] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a market data snapshot service node, a market data service node, and a storage node, comprising:

[0010] The market data snapshot service node is used to acquire market data, generate market data snapshot data based on the market data, and send the market data snapshot data to the market data service node.

[0011] The market data service node is used to extract order data and price data from the market snapshot data, compress the order data step by step according to the order volume to obtain target order data, write the target order data into an order file to obtain order structured data, and write the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein, the price structured data includes the address information of the order information corresponding to each price level in the order structured data, and the order file and the price file are stored in the storage node.

[0012] According to a third aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0013] The receiving module is configured to receive market snapshot data;

[0014] The extraction module is configured to extract order data and price data from the market snapshot data, and compress the order data step by step according to the order volume to obtain the target order data.

[0015] The writing module is configured to write the target order data into an order file to obtain order structured data, and to write the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein, the price structured data includes the address information of the order information corresponding to each price level in the order structured data.

[0016] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0017] Memory and processor;

[0018] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0019] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0020] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0021] The data processing method provided in this embodiment aims to reduce storage costs and save storage space. Before receiving market snapshot data and writing it to disk, considering that the main source of high storage space consumption is order data in the market snapshot data, order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain order structured data, and write the price data into the price file to obtain price structured data. This is used as a storage operation for market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0023] Figure 2 This is a schematic diagram of the architecture of a data processing method provided in one embodiment of this specification;

[0024] Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0025] Figure 4 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0026] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0027] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0028] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0029] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0031] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0033] Google Protocol Buffers (Protobuf) is a language-independent, platform-independent, and scalable serialization mechanism for structured data, designed for efficient data storage and communication. It uses a compact binary format, making it 3-10 times smaller than XML / JSON and 5-100 times faster at serialization. For example, integer types use Varint encoding, requiring only 1 byte to store small values.

[0034] Varint (Variable-length Integer) is an encoding method for efficiently storing and transmitting integers. Its core idea is to dynamically adjust the byte length to represent small values ​​with as few bytes as possible, while supporting the expansion of large values.

[0035] This specification provides a data processing method, and also relates to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0036] In practical applications, the order queue depth for the same price level in a thousand-level market data is typically 100. Taking a thousand-level market data of any stock with 200 levels as an example, if a thousand-level market data snapshot is built every 3 seconds, the required storage space is as follows: Each thousand-level market data snapshot uses an int32 type to store the transaction volume information, requiring 2*200*100*4=160000 bytes. The daily trading time is 14400 seconds, so each stock requires 14400 / 3=4800 thousand-level market data snapshots per day. Statistically, it can be determined that the thousand-level market data snapshots for this stock require 768,000,000 bytes. If a database such as MySQL is used to write each thousand-level market data snapshot to the database, it requires a huge amount of disk space. Therefore, an effective solution is urgently needed to solve the above problem.

[0037] The data processing method provided in this embodiment aims to reduce storage costs and save storage space. Before receiving market snapshot data and writing it to disk, considering that the main source of high storage space consumption is order data in the market snapshot data, order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain order structured data, and write the price data into the price file to obtain price structured data. This is used as a storage operation for market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space.

[0038] See Figure 1 , Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0039] Step S102: Receive market snapshot data.

[0040] Specifically, market snapshot data refers to the thousand-level market snapshots received by the market service node based on a certain stock. It can record fields such as the trading status or market conditions at a specific point in time, closing price, opening price, price at each level, order volume at each level, and time. It is used in scenarios such as market analysis and compliance supervision. The constructed market snapshot data is stored in a specified disk space.

[0041] It should be noted that the market snapshot data is generated by the upstream node of the market data service node. In practice, market snapshot data can be constructed based on market data when the upstream node triggers a market snapshot storage request. Specifically, the market snapshot storage request refers to the request triggered by the market snapshot service node after receiving market data from the exchange gateway. This request can be triggered at set intervals according to actual needs, such as every 3 seconds, to construct the market snapshot data corresponding to each piece of market data. Correspondingly, the market data is the thousand-level market data, specifically generated from the Level 2 tick-by-tick market data issued by the exchange gateway. It includes detailed price level information and order volume information for n levels of buy and sell orders, where n can be set according to business needs. In other words, the exchange gateway issues Level 2 tick-by-tick market data to the market snapshot service node, and after the market snapshot service node generates the market snapshot data (thousand-level market snapshot), it sends it to the market data service node for subsequent persistent storage.

[0042] In practical applications, the construction of market snapshot data can be completed by the market snapshot service node. It can generate market snapshot data for each stock through a preset algorithm and send it to subsequent market service nodes at regular intervals. This enables the market service nodes to persist the market snapshot data for use in market analysis, compliance and supervision, and other stages.

[0043] In practical implementation, when constructing market snapshot data based on market data, Level 2 tick-by-tick market data can be received in real time, including detailed information such as buy orders, sell orders, and trades. The data content can include key fields such as timestamp, price, order volume, and trade direction (buy / sell). The received data can then be cleaned to remove duplicate, erroneous, or invalid records, while simultaneously verifying data integrity and accuracy to ensure data quality meets requirements. Based on this, the data can be categorized into buy order queues and sell order queues according to trade direction (buy / sell). The data in each queue is then sorted by price; typically, the buy queue is sorted from highest to lowest price, and the sell queue from lowest to highest price.

[0044] Furthermore, a data structure (such as an array, linked list, or dictionary) is created to store order book information, including buy order queues and sell order queues. Each queue needs to store up to one thousand orders, including price and quantity. The cleaned and sorted data is then populated into the initial order book, forming thousands of order queues for both buy and sell orders. When a snapshot storage request for market data is triggered, the buy and sell order queue information in the current order book can be encapsulated into a snapshot data structure. This allows the market snapshot data to record key information such as timestamps, buy order queues (sorted by price from high to low), and sell order queues (sorted by price from low to high). This information can then be persisted later.

[0045] Step S104: Extract order data and price data from the market snapshot data, and compress the order data step by step according to the order volume to obtain the target order data.

[0046] Specifically, after obtaining the market snapshot data for any stock, in order to reduce the storage space occupied by the market snapshot data, it can be compressed. In the market snapshot data, since the order data usually uses the int type, it requires 32 bits and 4 bytes of space. However, the actual single order information does not require 4 bytes of space to store. Therefore, in order to save this space, the order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order amount of each order. After compression, the fully compressed order data for the corresponding stock can be obtained, that is, the target order data. Subsequently, the compressed order data and price data are stored to achieve the purpose of reducing storage space.

[0047] Specifically, the order data refers to the order queue information in the market snapshot data, including but not limited to the queue depth information and order volume information corresponding to each price level. Correspondingly, the price data refers to the number of price levels and the price information corresponding to each price level. For example, in a thousand-level market snapshot of stock A, the order data includes the queue depth of 100 levels (the number of orders for each price level) corresponding to each of the 500 buy and sell levels, and the order volume corresponding to each order. For example, at a price of 19.98, the order volumes for the 100 orders are 255, 100, 200…3200, etc. The price data is the price corresponding to the 500 buy and 500 sell levels of stock A, such as buy 1-19.8, buy 2-19.7, buy 3-19.6…; sell 1-14.5, sell 2-14.6, sell 3-14.7…

[0048] Accordingly, step-by-step compression specifically refers to dividing each order information into different levels based on the actual field value of each order quantity. Different compression methods are then applied to each level to reduce space usage. For example, if there are order quantities of 5000, 18000, and 30000 for the price range of 1-19.8, when compressing each order information, 5000 can be classified as the first level, 18000 as the second level, and 30000 as the third level. Different compression strategies are then applied to each level to reduce space usage. In practice, the order quantity range corresponding to the compression level can be set according to actual needs. For example, 0-10000 orders can be set as the first compression level, 10000-20000 as the second compression level, and above 20000 as the third compression level. This embodiment does not impose any limitations on this.

[0049] Furthermore, to achieve dynamic compression while reducing storage space usage, a data serialization protocol can be used. In this embodiment, the step of compressing the delegated data level by level according to the delegated amount to obtain the target delegated data includes:

[0050] Extract multiple order volume information corresponding to each price tier from the order data, and divide the multiple order volume information into at least two levels of information compression intervals; compress the order volume information contained in each level of information compression interval according to a preset data serialization protocol, and determine the target order data based on the compression results; wherein, each level of information compression interval corresponds to a different information compression strategy.

[0051] Specifically, the order information refers to the order volume corresponding to each price tier. Based on the order volume, multiple order information entries can be divided into at least two levels of information compression intervals. These compression intervals are defined by order volumes of varying sizes, and each level corresponds to a different compression strategy. This allows order information of different lengths to be compressed in different ways, dynamically reducing space occupancy and saving storage resources. Correspondingly, the preset data serialization protocol specifically refers to the Google Protocol Buffers protocol, used to serialize and compress the order data.

[0052] In practice, when using the Google Protocol Buffers protocol to compress the order volume information contained in different levels of information compression ranges, if the order volume information is in the range of 0-10000 lots, 1-2 bytes of Varint encoding can be selected. For a few large orders that may exceed 16393 lots, 3 bytes of Varint encoding can be selected. For a very small number of extremely large orders, 4-5 bytes of Varint encoding can be selected to ensure that the average encoding length is controlled between 1.2-1.8 bytes / order volume information. Compared with 4 bytes, this effectively reduces the space occupation rate.

[0053] For example, the order volume range of 0-127 has a value range (binary) of 0x00-0x7F and a Varint byte count of 1; the order volume range of 128-16383 has a value range (binary) of 0x80 0x01-0xFF 0x7F and a Varint byte count of 2; the order volume range of 16384-2097151 has a value range (binary) of 0x80 0x80 0x01-0xFF 0xFF 0x7F and a Varint byte count of 3; and the order volume range of 2097152-268435455 has a value range (binary) of 0x80 0x80 0x80 0x01-... and a Varint byte count of 4. By dividing the order volume in this way, it can be ensured that order volume information at different levels is compressed by different strategies, and the encoding speed is faster.

[0054] Based on this, when compressing order volume information at different levels in the order data, multiple order volume information corresponding to each price tier can be extracted from the order data first. Then, according to the actual distribution of the order volume information, the multiple order volume information can be divided into at least two levels of information compression intervals. In actual scenarios, since most of the order volume information will be distributed in the lower-level information compression intervals, and the length of the order volume information in the lower-level information compression intervals is shorter, the average encoding length of the lower-level information compression intervals can be smaller. That is, the level of the information compression interval is directly proportional to the encoding length corresponding to each information compression interval. Then, the order volume information contained in each level of information compression interval can be compressed separately according to the preset data serialization protocol to determine the target order data based on the compression results. This facilitates the subsequent persistent storage of target order data with lower storage space occupancy.

[0055] Furthermore, the persistence of market snapshot data is achieved by caching the data in memory before the order book is written. In this embodiment, the step of extracting order data and price data from the market snapshot data, and compressing the order data step by step according to the order volume to obtain the target order data includes:

[0056] The market snapshot data is parsed and cached in the target memory; order data and price data are extracted from the market snapshot data cached in the target memory, and the order data is compressed step by step according to the order volume to obtain the target order data.

[0057] Specifically, target memory refers to the memory resources called upon by the market data service node during data processing operations.

[0058] In practice, after the exchange gateway receives the market data, it needs to use it to construct market snapshot data, thereby triggering the subsequent market snapshot storage operation. This storage operation can be completed by the persistence module. The persistence module is a functional module deployed by the market service node to perform persistence processing on the market snapshot data. It is used to compress the order data in the market snapshot data and write the target order data and price data to the disk to complete the storage operation of the market snapshot data.

[0059] In other words, after receiving Level 2 tick-by-tick market data from the exchange gateway, the system can generate market data based on the Level 2 tick-by-tick data using a preset algorithm, and then construct market snapshot data. Afterward, the market snapshot data can be persisted. During the data persistence phase, the market service node can first parse the market snapshot data and cache the parsed market snapshot data in the target memory, thus enabling the persistence module to process the market snapshot data in memory. The persistence module extracts order data and price data from the market snapshot data cached in the target memory, and then compresses the order data step by step according to the order volume to obtain target order data with a smaller space footprint. The price data and the compressed order data can then be persisted to disk to complete the storage of the market snapshot data, while reducing storage space usage.

[0060] See Figure 2 The diagram illustrates the architecture of the data processing method. After the exchange gateway sends Level 2 tick-by-tick market data associated with stock A, the market snapshot service node can calculate and generate a thousand-level market snapshot corresponding to stock A using a preset algorithm. This snapshot can then be sent to the market service node at a preset 3-second interval. Upon receiving the thousand-level market snapshot for stock A from the market snapshot service node, the market service node can use its information parsing module to parse the snapshot. Based on this parsing, the thousand-level market data is cached in memory by the data caching module. Subsequently, the persistence module compresses the order data within the thousand-level market snapshot data and stores it on the target disk. This completes the storage of the thousand-level market snapshot for stock A, facilitating compliance supervision and market analysis.

[0061] In summary, by employing different compression strategies for different information compression ranges, the purpose of compressing the entrusted information can be achieved. Furthermore, different encoding lengths can be selected based on the length of the entrusted information, thereby effectively reducing the space occupancy rate.

[0062] Step S106 involves writing the target order data into an order file to obtain order structured data, and writing the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein, the price structured data includes the address information of the order information corresponding to each price level in the order structured data.

[0063] Specifically, the order file refers to the file used to store order data in the storage space, such as the queue.dat file; the price file refers to the file used to store price data in the storage space, such as the level.dat file; where the storage space is the disk that stores the market snapshot data, and the storage space can be implemented with any type of disk, such as solid-state drives, hard disks, etc., and this embodiment does not make any limitation.

[0064] In other words, the data processing method provided in this embodiment, considering the need for compression of order data, splits order data and price data into two parts and stores them separately in different files within the storage space. After the order data is stored in the order file, it forms structured order data associated with the market snapshot data; after the price data is stored in the price file, it forms structured price data associated with the market snapshot data. Furthermore, the structured order data includes queue depth information and order volume information corresponding to each price tier; the structured price data includes price information and address information corresponding to each price tier. Simultaneously, the price information corresponding to each price tier is stored according to a multi-branch tree structure.

[0065] It should be noted that since both the order structured data and the price structured data are part of the currently stored market snapshot data, in order to establish the relationship between the two, after the order structured data is generated, the address information of the order volume information corresponding to each price level in the order structured data can be determined, and this address information can be written into the price structured data. This establishes the relationship between the price level and the order volume information, so that when reading the market snapshot, the two separately stored parts of data can be read based on this relationship and merged into the market snapshot data for use.

[0066] In practice, the structure of the entrusted structured data is as follows:

[0067] FileOffset queueAddr; / / The file offset in queue.dat, which is -1 if not written.

[0068] struct queueData

[0069] {

[0070] int queueCnt; / / Queue depth

[0071] void * queueValue; / / An array (int32) at this address stores the number of pending orders. [queueCnt]

[0072] FileOffset queueAddr; / / The file offset in queue.dat, which is -1 if not written.

[0073] }

[0074] In other words, the structured data for the order is the order queue information structure, which stores the current price level, queue depth, and order quantity array. Its format in the queue.dat file is shown in Table 1 below:

[0075] Table 1

[0076]

[0077] Accordingly, the structure of the price structured data is as follows:

[0078] / / The key is the price tier; the value is the order queue information for that tier.

[0079] std::map<float, struct queueData> quotaLevelData[2]; / / Buy and sell directions

[0080] The price tier information can be stored using a std::map red-black tree data structure, with the price (float) type as the index key and the corresponding value being the order queue information under that price; its storage format in the level.dat file is shown in Table 2 below:

[0081] Table 2

[0082]

[0083] In each price tier, the queueAddr field represents the address of the order quantity information corresponding to that price tier in the queue.dat file.

[0084] Furthermore, taking a snapshot of 200 price levels as an example, compared to the previous snapshot, the most frequently changing data in the current snapshot is the queue near the transaction price, while the data for most price levels remains unchanged. This is because the price level queue stores customer buy or sell orders, which only change after being executed or cancelled. Price levels near the transaction price will change due to execution, while price levels far from the transaction price will not change unless cancelled. Therefore, even with most price level data remaining unchanged, full storage still results in significant disk usage. Thus, before persisting the snapshot data, by comparing snapshot data from two adjacent moments, only the changed data can be stored, further reducing storage space consumption. In this embodiment, the target memory caches the previous moment's market snapshot data.

[0085] The steps of writing the target order data into an order file to obtain structured order data and writing the price data into a price file to obtain structured price data include:

[0086] The market snapshot data is traversed, and the traversal results are used to detect whether there is the same price information in the market snapshot data as in the previous market snapshot data.

[0087] If not, the target order data is written into the order file to obtain the order structured data, and the address information of the order information corresponding to each price tier in the order structured data is recorded. The price data and the address information are then written into the price file to obtain the price structured data.

[0088] Specifically, the previous snapshot data refers to the snapshot data that has been persisted before the current snapshot data to be persisted, and is adjacent to the current snapshot data to be persisted.

[0089] In practical implementation, when persisting market snapshot data, considering that multiple snapshots are recorded for each stock daily, if the market data remains unchanged or changes only slightly, adjacent snapshots may contain identical or similar data, leading to some duplicate data in the storage space. Therefore, this embodiment separates order data and price data when storing market snapshot data. Since each snapshot generation requires specific snapshot data storage operations, to further save storage space, a comparison operation between adjacent snapshots can be performed on the snapshot data before persistence. Different comparison results can then be used to select different methods for separate market snapshot data storage.

[0090] Based on this, before persisting the market snapshot data, the persistence module can be used to traverse the market snapshot data. At this time, the traversal results can be used to check whether there is the same price information in the market snapshot data as in the previous moment's market snapshot data. If not, the traversal operation determines that the price information corresponding to each price level in the market snapshot data generated at the current moment has changed relative to the previous moment, and there is no identical price information. Therefore, it is further explained that the market snapshot data at the current moment needs to be stored completely. Thus, the target order data can be written to the order file to obtain the order structured data. At the same time, the address information of the order information corresponding to each price level in the order structured data can be recorded. After that, the price data and address information can be written to the price file to obtain the price structured data, thereby completing the persistence operation of the market snapshot data at the current moment, that is, completing the storage of the market snapshot data.

[0091] In practice, when comparing the current market snapshot data with the previous market snapshot data, the current market snapshot data (quotaLevelData) and the previous market snapshot data (prequotaLevelData) can be cached in memory simultaneously according to the identification information of each stock (such as stock code). Furthermore, the market snapshot data in memory is structured as std::map.<float, struct queueData> The persistence module iterates through the current market snapshot data quotaLevelData. The map stores the number of elements quotaLevelData.size() is the number of price levels levelCnt, and queueData is the order queue information corresponding to each price level.

[0092] Based on the traversal results, the system first checks if the corresponding price level exists in the previous timeframe's snapshot data `prequotaLevelData`, using the price information for any price tier. If the price tier does not exist in the previous timeframe's snapshot data, the current price tier data is considered new and needs to be written to the order file for storage. Therefore, the queue depth information and all order volume information recorded in `queueData` can be written to the `queue.dat` file for persistent storage of the order data corresponding to that price tier. Simultaneously, the address information after the current order data is written to the file can be recorded and stored in `quotaLevelData.queueData.queueAddr`. At this point, the price data `levelCnt` corresponding to the current price tier and the address `queueAddr` of the order data in the `queue.dat` file can be written to the `level.dat` file, serving as storage for the new price tier corresponding to the snapshot data in the current timeframe's snapshot data. Meanwhile, to facilitate subsequent storage operations, the current market snapshot data quotaaLevelData can be updated to prequotaLevelData, so that when the next market snapshot data is received, it can be compared with the current market snapshot data, thereby completing the storage operation of the next market snapshot data.

[0093] Furthermore, if the result of the traversal test confirms whether there is price information identical to that in the previous snapshot data, it indicates that there are market conditions with the same price level as the previous time. Therefore, it can be further determined whether the order information for market conditions with the same price level is identical. Specifically:

[0094] The system detects whether the order information corresponding to the same price information in the market snapshot data and the previous market snapshot data is the same; if they are not the same, it executes the steps of writing the target order data into the order file to obtain the order structured data, recording the address information of the order information corresponding to each price level in the order structured data, and writing the price data and the address information into the price file to obtain the price structured data.

[0095] Based on this, if the traversal results detect that the price information in the market snapshot data is the same as that in the previous market snapshot data, it indicates that the price information corresponding to the price level being compared at this time is not a new price relative to the previous time. Therefore, it is possible to further compare whether the order information corresponding to the same price level is the same, that is: whether the order volume of the same price level has not changed at the current time relative to the previous time. If not, it indicates that the order volume has changed. In this case, the order volume information corresponding to the price information at the previous time cannot be used as the order volume corresponding to the price information at the current time. Therefore, the steps of writing the target order data into the order file to obtain the order structured data, recording the address information of the order information corresponding to each price level in the order structured data, and writing the price data and the address information into the price file to obtain the price structured data can be repeated.

[0096] In other words, if the previous snapshot data contains the same price level as the current snapshot data, the order queue data corresponding to that price level can be further compared to see if they are the same. Specifically, the order volume corresponding to the same price level in the previous and current snapshots is the same. If they are different, it indicates a change in order volume, and therefore the write operation for the new snapshot data can be repeated. Specifically, the queue depth information and all order volume information recorded in `queueData` are written to the `queue.dat` file for persistent storage of the order data corresponding to that price level. Simultaneously, the address information after the current order data is written to the file can be recorded and stored in `quotaLevelData.queueData.queueAddr`. At this point, the price data `levelCnt` corresponding to the current price level and the address `queueAddr` of the order data in the `queue.dat` file can be written to the `level.dat` file as storage of the snapshot data corresponding to the new price level in the current snapshot data. Meanwhile, to facilitate subsequent storage operations, the current market snapshot data quotaaLevelData can be updated to prequotaLevelData, so that when the next market snapshot data is received, it can be compared with the current market snapshot data, thereby completing the storage operation of the next market snapshot data.

[0097] Furthermore, if the judgment result for whether the order information corresponding to the same price information in the current market snapshot data and the previous market snapshot data is the same is the same, it indicates that there are market conditions with the same price level and the same order information between the current time and the previous time. Therefore, in order to save storage space, only the price data can be stored to represent its corresponding snapshot at the current time, while the order data can be left unwritten and the order data storage information from the previous time can be directly reused, thereby further saving storage space. Specifically:

[0098] In the order file, the target address information corresponding to the same price information in the previous market snapshot data is read; the target address information and the price data are written into the price file to obtain price structured data, which is used as a storage operation on the market snapshot data.

[0099] Therefore, if the order information corresponding to the same price information in the current market snapshot data and the previous market snapshot data is identical, it indicates that there are price tiers with the same price and corresponding order volume at the current and previous times. If the data is still stored using the same method as new data, the same snapshot data will be stored for both times, resulting in wasted storage space. Therefore, to further save storage space, only the price data can be stored, while the order volume corresponding to that price tier can use the address information of the order information stored at the previous time. In other words, for price tiers with the same price and order volume, when storing data, the address information corresponding to the order information already stored at that price tier at the previous time can be added to the price structured data. This avoids storing the current time's order information again, thus saving storage space.

[0100] Therefore, during storage operations, the target address information corresponding to the same price information in the previous snapshot data can be read from the order file. This target address information can then be stored as the price level corresponding to the same price information in the address information recorded in the price file. At this point, the target address information and price data can be written to the price file to obtain structured price data, which is then used for storing the snapshot data.

[0101] In other words, if the order queue data in the previous snapshot data prequotaLevelData is the same as the data for that price level, it means that the price and order volume corresponding to that price level have not changed between the previous and current times. For example, orders far from the transaction price are generally unchanged in two adjacent snapshots. Therefore, it is only necessary to write the price level information levelCnt and the file address of the queue data file prequotaLevelData.queueData.queueAddr for each price level into the level.dat file, while the queue depth data and all pending order volume data in queueData do not need to be written into queue.dat.

[0102] In practice, since the market snapshot data stores the order volume corresponding to multiple price levels, the above comparison operation can be performed for each price level in the current market snapshot data. For prices that do not match successfully, the price data and order data are stored in the normal separate storage method. For prices that match successfully, the address of the order information corresponding to the price information at the previous moment in the order file can be selected for storage, thereby saving the order volume storage operation of one price level and further reducing the storage space occupied.

[0103] The data processing method provided in this embodiment aims to reduce storage costs and save storage space. Before receiving market snapshot data and writing it to disk, considering that the main source of high storage space consumption is order data in the market snapshot data, order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain order structured data, and write the price data into the price file to obtain price structured data. This is used as a storage operation for the market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space.

[0104] The following is in conjunction with the appendix Figure 3 Taking the application of the data processing method provided in this specification in a stock snapshot data storage scenario as an example, the data processing method will be further explained. Figure 3A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0105] Step S302: Receive market snapshot data.

[0106] Step S304: parse the market snapshot data and cache the parsed market snapshot data in the target memory.

[0107] Step S306: Extract order data and price data from the market snapshot data in the target memory cache.

[0108] Step S308: Extract multiple order volume information corresponding to each price tier from the order data, and divide the multiple order volume information into information compression intervals of at least two levels.

[0109] Step S310: Compress the entrusted information contained in each level of information compression interval according to the preset data serialization protocol, and determine the target entrusted data based on the compression result.

[0110] Step S312: Iterate through the market snapshot data and check whether there is the same price information in the market snapshot data as in the previous snapshot data. If not, proceed to step S314; if yes, proceed to step S316.

[0111] Step S314: Write the target order data into the order file to obtain the order structured data, and record the address information of the order information corresponding to each price level in the order structured data. Write the price data and address information into the price file to obtain the price structured data, which is used as a storage operation for the market snapshot data.

[0112] Step S316: Check whether the order information corresponding to the same price information in the market snapshot data and the previous market snapshot data is the same. If they are different, proceed to step S314; if they are the same, proceed to step S318.

[0113] Step S318: In the order file, read the target address information of the order information corresponding to the same price information in the previous market snapshot data; write the target address information and price data into the price file to obtain the price structured data, which is used as the storage operation of the market snapshot data.

[0114] The structured data for the entrustment includes queue depth information and entrust volume information corresponding to each price tier; the structured data for the price includes price information and address information corresponding to each price tier, wherein the price information corresponding to each price tier is stored in a multi-branch tree structure.

[0115] The data processing method provided in this embodiment aims to reduce storage costs and save storage space. Before receiving market snapshot data and writing it to disk, considering that the main source of high storage space consumption is order data in the market snapshot data, order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain order structured data, and write the price data into the price file to obtain price structured data. This is used as a storage operation for the market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space.

[0116] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 4 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 4 As shown, the data processing system 400 includes a market data snapshot service node 410, a market data service node 420, and a storage node 430, comprising:

[0117] The market data snapshot service node 410 is used to acquire market data, generate market data snapshot data based on the market data, and send the market data snapshot data to the market service node.

[0118] The market data service node 420 is used to extract order data and price data from the market data snapshot data, compress the order data step by step according to the order volume to obtain target order data, write the target order data into an order file to obtain order structured data, and write the price data into a price file to obtain price structured data, as a storage operation on the market data snapshot data; wherein, the price structured data includes the address information of the order information corresponding to each price level in the order structured data, and the order file and the price file are stored in the storage node 430.

[0119] In an optional embodiment, the step of compressing the entrustment data step by step according to the entrustment volume to obtain the target entrustment data includes:

[0120] Extract multiple order volume information corresponding to each price tier from the order data, and divide the multiple order volume information into at least two levels of information compression intervals; compress the order volume information contained in each level of information compression interval according to a preset data serialization protocol, and determine the target order data based on the compression results; wherein, each level of information compression interval corresponds to a different information compression strategy.

[0121] In one optional embodiment, the structured data of the entrustment includes queue depth information and entrustment volume information corresponding to each price tier; the structured data of the price includes price information and address information corresponding to each price tier, wherein the price information corresponding to each price tier is stored in a multi-branch tree structure.

[0122] In an optional embodiment, the step of extracting order data and price data from the market snapshot data, and compressing the order data step by step according to the order volume to obtain target order data, includes:

[0123] The market snapshot data is parsed and cached in the target memory; order data and price data are extracted from the market snapshot data cached in the target memory, and the order data is compressed step by step according to the order volume to obtain the target order data.

[0124] In one optional embodiment, the target memory caches the market snapshot data from the previous moment;

[0125] The steps of writing the target order data into an order file to obtain structured order data and writing the price data into a price file to obtain structured price data include: traversing the market snapshot data, detecting whether there is price information in the market snapshot data that is the same as that in the market snapshot data at the previous moment based on the traversal result; if not, writing the target order data into an order file to obtain structured order data, recording the address information of the order information corresponding to each price level in the structured order data, and writing the price data and the address information into a price file to obtain structured price data.

[0126] In an optional embodiment, if the determination result of whether there is the same price information in the market snapshot data as in the previous moment's market snapshot data is yes, the method further includes:

[0127] The system detects whether the order information corresponding to the same price information in the market snapshot data and the previous market snapshot data is the same; if they are not the same, it executes the steps of writing the target order data into the order file to obtain the order structured data, recording the address information of the order information corresponding to each price level in the order structured data, and writing the price data and the address information into the price file to obtain the price structured data.

[0128] In an optional embodiment, if the determination result of whether the order information corresponding to the same price information in the market snapshot data and the previous time snapshot data are the same is the same, it further includes:

[0129] In the order file, the target address information corresponding to the same price information in the previous market snapshot data is read; the target address information and the price data are written into the price file to obtain price structured data, which is used as a storage operation on the market snapshot data.

[0130] The data processing system provided in this embodiment, in order to reduce storage costs and save storage space, considers that the main source of high storage space consumption is the order data in the market snapshot data before receiving and writing the data to disk. Therefore, it can extract order data and price data from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain the order structured data, and write the price data into the price file to obtain the price structured data. This is the storage operation of the market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space.

[0131] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0132] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0133] Receiver module 502 is configured to receive market snapshot data;

[0134] The extraction module 504 is configured to extract order data and price data from the market snapshot data, and compress the order data step by step according to the order volume to obtain the target order data.

[0135] The writing module 506 is configured to write the target order data into an order file to obtain order structured data, and to write the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein, the price structured data includes the address information of the order information corresponding to each price level in the order structured data.

[0136] In an optional embodiment, the step of compressing the entrustment data step by step according to the entrustment volume to obtain the target entrustment data includes:

[0137] Extract multiple order volume information corresponding to each price tier from the order data, and divide the multiple order volume information into at least two levels of information compression intervals; compress the order volume information contained in each level of information compression interval according to a preset data serialization protocol, and determine the target order data based on the compression results; wherein, each level of information compression interval corresponds to a different information compression strategy.

[0138] In one optional embodiment, the structured data of the entrustment includes queue depth information and entrustment volume information corresponding to each price tier; the structured data of the price includes price information and address information corresponding to each price tier, wherein the price information corresponding to each price tier is stored in a multi-branch tree structure.

[0139] In an optional embodiment, the step of extracting order data and price data from the market snapshot data, and compressing the order data step by step according to the order volume to obtain target order data, includes:

[0140] The market snapshot data is parsed and cached in the target memory; order data and price data are extracted from the market snapshot data cached in the target memory, and the order data is compressed step by step according to the order volume to obtain the target order data.

[0141] In one optional embodiment, the target memory caches the market snapshot data from the previous moment;

[0142] The steps of writing the target order data into an order file to obtain structured order data and writing the price data into a price file to obtain structured price data include: traversing the market snapshot data, detecting whether there is price information in the market snapshot data that is the same as that in the market snapshot data at the previous moment based on the traversal result; if not, writing the target order data into an order file to obtain structured order data, recording the address information of the order information corresponding to each price level in the structured order data, and writing the price data and the address information into a price file to obtain structured price data.

[0143] In an optional embodiment, if the determination result of whether there is the same price information in the market snapshot data as in the previous moment's market snapshot data is yes, the method further includes:

[0144] The system detects whether the order information corresponding to the same price information in the market snapshot data and the previous market snapshot data is the same; if they are not the same, it executes the steps of writing the target order data into the order file to obtain the order structured data, recording the address information of the order information corresponding to each price level in the order structured data, and writing the price data and the address information into the price file to obtain the price structured data.

[0145] In an optional embodiment, if the determination result of whether the order information corresponding to the same price information in the market snapshot data and the previous time snapshot data are the same is the same, it further includes:

[0146] In the order file, the target address information corresponding to the same price information in the previous market snapshot data is read; the target address information and the price data are written into the price file to obtain price structured data, which is used as a storage operation on the market snapshot data.

[0147] The data processing apparatus provided in this embodiment, in order to reduce storage costs and save storage space, before receiving market snapshot data and writing the data to disk, considers that the main source of high storage space consumption is the order data in the market snapshot data. Therefore, order data and price data can be extracted from the market snapshot data. At this time, the order data can be compressed step by step according to the order volume, so as to reduce the storage space occupied by the order information without loss, thereby obtaining target order data with smaller storage space. Then, a data separation storage method is designed to write the target order data into the order file to obtain order structured data, and write the price data into the price file to obtain price structured data. This is the storage operation of the market snapshot data. The price structured data includes the address information of the order information corresponding to each price level in the order structured data, thereby establishing a mapping relationship between them. When storing market snapshot data, the order data and price data can be separated. The order data is compressed to reduce the space occupancy rate, thereby achieving the purpose of persistent market snapshot data and saving disk space.

[0148] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0149] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0150] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0151] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0152] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0153] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0154] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0155] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0156] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0157] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0158] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0159] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0160] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0161] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0162] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0163] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A data processing method, characterized in that, include: Receive market snapshot data; The order data and price data are extracted from the market snapshot data. The order data is then compressed step-by-step according to the order volume to obtain the target order data. The step-by-step compression of the order data according to the order volume to obtain the target order data includes: extracting multiple order volume information corresponding to each price tier from the order data; dividing the multiple order volume information into at least two levels of information compression intervals; compressing the order volume information contained in each level of information compression interval according to a preset data serialization protocol; and determining the target order data based on the compression results. Each level of information compression interval corresponds to a different information compression strategy, and the information compression interval is a range divided by order volumes of different spans. The target order data is written to an order file to obtain structured order data, and the price data is written to a price file to obtain structured price data, which serves as a storage operation for the market snapshot data; The entrustment file and the price file are stored in the same storage space, and the price structured data includes the address information of the entrustment information corresponding to each price tier in the entrustment structured data.

2. The data processing method according to claim 1, characterized in that, The structured data for the orders includes queue depth information and order volume information corresponding to each price tier; The structured price data includes price information corresponding to each price tier and the address information, wherein the price information corresponding to each price tier is stored in a multi-branch tree structure.

3. The data processing method according to claim 1, characterized in that, The step involves extracting order data and price data from the market snapshot data, and compressing the order data step by step according to the order volume to obtain the target order data, including: The market snapshot data is parsed, and the parsed market snapshot data is cached in the target memory; Extract order data and price data from the market snapshot data in the target memory cache, and perform stepwise compression of the order data according to the order volume to obtain the target order data.

4. The data processing method according to claim 3, characterized in that, The target memory caches the previous moment's market snapshot data; The steps of writing the target order data into an order file to obtain structured order data and writing the price data into a price file to obtain structured price data include: The market snapshot data is traversed, and the traversal results are used to detect whether there is the same price information in the market snapshot data as in the previous market snapshot data. If not, the target order data is written into the order file to obtain the order structured data, and the address information of the order information corresponding to each price tier in the order structured data is recorded. The price data and the address information are then written into the price file to obtain the price structured data.

5. The data processing method according to claim 4, characterized in that, If the determination result of whether there is the same price information in the market snapshot data as in the previous moment's market snapshot data is yes, the method further includes: Detect whether the order information corresponding to the same price information in the market snapshot data and the previous market snapshot data is the same; If they are not the same, execute the steps of writing the target order data into the order file to obtain the order structured data, recording the address information of the order information corresponding to each price tier in the order structured data, writing the price data and the address information into the price file to obtain the price structured data.

6. The data processing method according to claim 5, characterized in that, If the determination result of whether the order information corresponding to the same price information in the market snapshot data and the previous time snapshot data is the same is the same, the method further includes: In the order file, the target address information corresponding to the same price information in the previous time snapshot market data is read; The target address information and the price data are written into a price file to obtain structured price data, which is then used as storage operation for the market snapshot data.

7. A data processing system, characterized in that, This includes market data snapshot service nodes, market data service nodes, and storage nodes, including: The market data snapshot service node is used to acquire market data, generate market data snapshot data based on the market data, and send the market data snapshot data to the market data service node. The market data service node is used to extract order data and price data from the market snapshot data, and to compress the order data step by step according to the order volume to obtain target order data. The step-by-step compression of the order data to obtain target order data includes: extracting multiple order volume information corresponding to each price tier from the order data, dividing the multiple order volume information into at least two levels of information compression intervals; compressing the order volume information contained in each level of information compression interval according to a preset data serialization protocol, and determining the target order data based on the compression results; wherein each level of information compression interval corresponds to a different information compression strategy, and the information compression interval is an interval divided by order volumes of different spans; writing the target order data into an order file to obtain order structured data, and writing the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein the order file and the price file are stored in the same storage space, the price structured data includes the address information of the order information corresponding to each price tier in the order structured data, and the order file and the price file are stored in the storage node.

8. A data processing apparatus, characterized in that, include: The receiving module is configured to receive market snapshot data; The extraction module is configured to extract order data and price data from the market snapshot data, and compress the order data step-by-step according to the order volume to obtain target order data. The step-by-step compression of the order data according to the order volume to obtain target order data includes: extracting multiple order volume information corresponding to each price level from the order data; dividing the multiple order volume information into at least two levels of information compression intervals; compressing the order volume information contained in each level of information compression interval according to a preset data serialization protocol; and determining the target order data based on the compression results. Each level of information compression interval corresponds to a different information compression strategy, and the information compression interval is an interval divided by order volumes of different spans. The writing module is configured to write the target order data into an order file to obtain order structured data, and to write the price data into a price file to obtain price structured data, as a storage operation on the market snapshot data; wherein the order file and the price file are stored in the same storage space, and the price structured data includes the address information of the order information corresponding to each price level in the order structured data.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

11. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images