Order book streaming inversion method, device and equipment based on Spark-Streaming and medium
By using the Spark-Streaming streaming inversion method, combined with Kafka and Spark-Streaming for partitioning and statistical calculations of order book data, the problem of decreased computational performance in batch processing is solved, achieving efficient order book inversion and improving the system's processing capacity.
Patent Information
- Application Number
- CN202511163379.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-19
- Publication Date
- 2025-11-21
AI Technical Summary
Existing technologies for order book inversion suffer from the problem that batch processing requires managing the temporal continuity between batches, leading to a decrease in computational performance as data increases.
A streaming inversion method based on Spark-Streaming is adopted. Data is partitioned through the distributed message distribution system Kafka, and Spark-Streaming is used for basic data inversion and statistical data calculation. This releases the basic data of the order book, avoids single-batch caching, and achieves time continuity management between batches.
It improved the performance of the data processing system, enabling it to handle tens of thousands of transaction data per second, ensuring computing performance and real-time performance.
Smart Images
Figure CN120994711A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, device, and medium for order book streaming inversion based on Spark-Streaming. Background Technology
[0002] Order book inversion mainly involves using order placement, cancellation, and transaction data within a trading period to reconstruct the state of the order book at each moment. Essentially, it is a time series processing process. However, due to the massive amount of data received and processed in real time, it is a real-time big data problem. In solving similar problems, SQL-like batch processing processes need to manually divide the temporally continuous data into different batches. In this approach, the batch processing logic needs to manage the temporal continuity between batches. Another approach is not to divide the data and perform full calculations each time, but the computational performance of this approach will decrease as the data increases.
[0003] Therefore, there is a need to find an order book inversion method that can automatically perform batch processing while ensuring computational performance. Summary of the Invention
[0004] In view of the above situation, this application provides a Spark-Streaming-based order book streaming inversion method, apparatus, device and medium, which aims to solve the above problems or at least partially solve the above problems.
[0005] Firstly, this application provides a Spark-Streaming-based order book streaming inversion method, including:
[0006] Obtain the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka;
[0007] The delegated transaction data in the Kafka distributed message distribution system is partitioned;
[0008] Spark-Streaming is used to perform basic data inversion on the order book data of each partition to generate basic order book data for each partition. Statistical data is calculated on the basic order book data of each partition to obtain the order book statistical data for each partition. Finally, the order book data and the basic order book data of each partition are released.
[0009] The order book statistics for each partition are written into the distributed message distribution system Kafka.
[0010] For example, partitioning the delegated transaction data in the distributed message distribution system Kafka includes:
[0011] Obtain multiple contract identifiers from the entrusted transaction data;
[0012] The entrusted transaction data is partitioned in the distributed message distribution system Kafka based on multiple contract identifiers.
[0013] For example, partitioning the delegated transaction data in the distributed message distribution system Kafka includes:
[0014] The order transaction data is read from the distributed message distribution system Kafka using Spark Streaming;
[0015] Obtain multiple contract identifiers from the entrusted transaction data;
[0016] The entrusted transaction data is partitioned using Spark Streaming based on multiple contract identifiers.
[0017] For example, partitioning the delegated transaction data in the distributed message distribution system Kafka includes:
[0018] Construct a composite primary key, which includes the transaction date and the contract identifier;
[0019] The delegated transaction data in the Kafka distributed message distribution system is partitioned according to the composite primary key.
[0020] For example, the method further includes: performing basic data inversion and statistical data calculation on the entrusted transaction data in batches using Spark-Streaming.
[0021] For example, the method further includes storing the order book statistics data written to each partition in the distributed message distribution system Kafka into a statistics database.
[0022] For example, the method further includes:
[0023] The order book base data for each partition is generated by performing basic data inversion on the order transaction data of each partition using another Spark-Streaming method, and the order book base data of each partition is stored in the base library through the distributed message distribution system Kafka.
[0024] Secondly, this application provides a Spark-Streaming-based order book streaming inversion apparatus, comprising:
[0025] The acquisition module is used to acquire the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka;
[0026] The partitioning module is used to partition the delegated transaction data in the distributed message distribution system Kafka;
[0027] The inversion statistics module is used to perform basic data inversion on the order transaction data of each partition using Spark-Streaming to generate basic order book data for each partition, perform statistical data calculation on the basic order book data of each partition to obtain the order book statistical data of each partition, and release the order transaction data and the basic order book data of each partition.
[0028] The write module is used to write the order book statistics data of each partition into the distributed message distribution system Kafka.
[0029] Thirdly, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the Spark-Streaming-based order book streaming inversion method as described in the first aspect.
[0030] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the Spark-Streaming-based order book streaming inversion method described in the first aspect.
[0031] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:
[0032] This application automatically implements time continuity management and near real-time big data computing between batches by performing order book inversion using the Spark-Streaming distributed stream computing engine. It obtains the order book statistics of each partition by performing statistical data calculation on the basic order book data of each partition, and releases the entrusted transaction data and the basic order book data of each partition. The basic order book data is not cached within a single batch, which greatly improves the performance of the entire data processing system and can handle tens of thousands of transaction streams per second. Attached Figure Description
[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0034] Figure 1 This is a schematic diagram of an application environment for the Spark-Streaming-based order book streaming inversion method in one embodiment of the present invention;
[0035] Figure 2 This is a flowchart illustrating an order book streaming inversion method based on Spark-Streaming in one embodiment of the present invention;
[0036] Figure 3 This is another flowchart illustrating the order book streaming inversion method based on Spark-Streaming in one embodiment of the present invention;
[0037] Figure 4 This is another flowchart illustrating the order book streaming inversion method based on Spark-Streaming in one embodiment of the present invention;
[0038] Figure 5 This is a schematic diagram of a Spark-Streaming-based order book streaming inversion device according to an embodiment of the present invention;
[0039] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;
[0040] Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."
[0043] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0044] As mentioned earlier, for real-time big data processing problems such as order book inversion, one approach is to solve it through a SQL-like batch processing procedure, which requires managing the time continuity between batches. Another approach is to perform full computation without splitting the data, but the computational performance decreases as the data increases. Both of these approaches have limitations. To address this technical problem, this application provides an order book streaming inversion method based on Spark-Streaming.
[0045] The order book streaming inversion method based on Spark-Streaming provided in this invention can be applied to applications such as... Figure 1 In this application environment, the device communicates with the server via a network. The server can obtain the entrusted transaction data from the device and import it into the distributed message distribution system Kafka. The server then partitions the entrusted transaction data in Kafka. Using Spark Streaming, the server performs basic data inversion on the entrusted transaction data of each partition to generate basic order book data for each partition. Statistical data is then calculated on the basic order book data of each partition to obtain statistical data for the order book of each partition. Finally, the entrusted transaction data and basic order book data of each partition are released. The statistical data for the order book of each partition is then written into the distributed message distribution system Kafka. This application automatically implements time continuity management between batches by performing order book inversion using the Spark Streaming streaming engine. By calculating statistical data on the basic order book data of each partition to obtain statistical data for the order book of each partition and releasing the entrusted transaction data and basic order book data of each partition, and by not caching basic order book data within a single batch, the performance of the entire data processing system is greatly improved, enabling it to handle tens of thousands of transactions per second.
[0046] The device side can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server side can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.
[0047] Please see Figure 2 , Figure 3 , Figure 4 As shown, Figure 4 A flowchart illustrating the order book streaming inversion method based on Spark-Streaming provided in this embodiment of the invention includes the following steps:
[0048] S10: Obtain the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka.
[0049] For example, order data includes order placement, cancellation, and execution data during the trading period.
[0050] For example, Kafka is an open-source messaging system developed by the Apache Software Foundation. It is a distributed, publish / subscribe-based message queue that provides data buffering, distributed data transmission, and streaming data processing capabilities for big data systems. It helps enterprises quickly process massive amounts of data and supports multiple programming languages and real-time data stream processing. In this application, Kafka primarily connects a financial transaction system as the producer and Spark Streaming as the consumer. By sending the entrusted transaction data from the financial transaction system to Kafka, the data can be simultaneously transmitted to multiple Spark Streaming tasks, enabling parallel data processing and distributed computing. Kafka allows the establishment of a persistent message queue between producers and consumers. In this queue, all messages are retained for a period of time so that consumers can retrieve them at any time. By sending entrusted transaction data to Kafka, this feature of Kafka can be fully utilized to cache and buffer data, thereby reducing the pressure on Spark Streaming and improving the system's reliability and fault tolerance.
[0051] S20: Partition the entrusted transaction data in the distributed message distribution system Kafka.
[0052] For example, partitioning is the basis for subsequent parallel inversion of multiple contracts. The partitioning process can be performed in the distributed message distribution system Kafka, and Spark Streaming does not need to partition again during reading. In one implementation, step S20, partitioning the entrusted transaction data in the distributed message distribution system Kafka, includes:
[0053] S21: Obtain multiple contract identifiers from the entrusted transaction data;
[0054] S22: Partition the entrusted transaction data in the distributed message distribution system Kafka according to the multiple contract identifiers.
[0055] For example, the partitioning process can also be performed by storing the partitioned data in the distributed message distribution system Kafka as a single partition, and then repartitioning it after Spark Streaming reads the order transaction data. In one implementation, step S20, which involves partitioning the order transaction data in the distributed message distribution system Kafka, includes:
[0056] S21′: Read the entrusted transaction data from the distributed message distribution system Kafka via Spark-Streaming;
[0057] S22′: Obtain multiple contract identifiers from the entrusted transaction data;
[0058] S23′: Partition the entrusted transaction data according to multiple contract identifiers using Spark-Streaming.
[0059] For example, in step S20 of this application, a composite primary key is constructed, which includes the transaction date and the contract identifier; the order transaction data in the distributed message distribution system Kafka is partitioned according to the composite primary key. During the order book inversion calculation, considering the business characteristics of financial futures industry product contracts, the order book inversion of a single contract on a single trading day can only be calculated sequentially and cannot be parallelized. Parallelization can only be performed on multiple contracts. Therefore, this application constructs a composite primary key (tradingday, instrumentid, contract ID) as the primary key for distributed computing to achieve distributed computing across multiple contracts. The contract identifier in this application is the contract number or contract ID.
[0060] like Figure 2 , Figure 3 As shown, in this application, steps S21' to S23' are used. The partitioning process is performed by storing the data in a single partition in the distributed message distribution system Kafka. Spark-Streaming then repartitions the data after reading the entrusted transaction data. Each partition represents the entrusted transaction data corresponding to a contract identifier. Figure 3 As shown, contract identifiers include IF1912, TF191, IH191, etc., and each contract identifier corresponds to a partition.
[0061] S30: Use Spark-Streaming to perform basic data inversion on the order transaction data of each partition to generate basic order book data for each partition, perform statistical data calculation on the basic order book data of each partition to obtain the order book statistical data of each partition, and release the order transaction data and the basic order book data of each partition.
[0062] For example, the real-time order book inversion task is a big data processing problem. The basic order book data generated by the inversion of a single contract can easily exceed memory limits. Therefore, this method combines Spark-Streaming's micro-batch architecture to automatically perform sharding processing, that is, dividing the order stream data into different batches according to time, and processing the next batch only after the previous batch is completed. This can effectively solve the problem of real-time big data processing with limited memory. In other words, Spark-Streaming performs basic data inversion and statistical data calculation on the order transaction data in batches.
[0063] For example, Spark-Streaming automatically shards the entrusted transaction data obtained from the distributed message distribution system Kafka, and then partitions it using a composite primary key. Each partition is then used for basic data inversion and statistical data calculation.
[0064] For example, most of the computational workload of the inversion lies in the inversion of basic data and the calculation of statistical data. For each contract, this calculation process requires maintaining an order book data structure in memory and calculating hundreds of indicators. The data of multiple contracts will have a skewed distribution, with a larger amount of data in the main contract and a smaller amount of data in the non-main contract. In a single batch, the inversion of the main contract is not finished when the inversion of the non-main contract ends. Therefore, the inversion speed of the order transaction data of a single batch is determined according to the amount of data in the main contract.
[0065] S40: Write the single-book statistical data of each partition into the distributed message distribution system Kafka.
[0066] For example, such as Figure 2 and Figure 3As shown, in the data writing stage, the order book statistics are relatively small in scale. The order book statistics for each partition are written to the distributed message distribution system Kafka in parallel, and then stored in the statistics database. Since the order book basic data is nearly a hundred times the size of the order transaction data, taking a data processing speed of thousands of transactions per second as an example, the system needs to write 100,000 basic records per second. Comparative tests revealed that, excluding the write operation, caching a large amount of order book basic data in memory within a single batch significantly slows down the overall system performance. Therefore, in this application, when inverting the order transaction data using Spark-Streaming, the order book basic data is not cached within a single batch. After the order book basic data is inverted, statistical data is directly calculated, and the resulting order book statistics are written to the distributed message distribution system Kafka. Afterwards, the order transaction data and order book basic data are immediately released. This solution can handle transaction flows of tens of thousands of transactions per second because the statistical data is directly provided to the public and cannot tolerate large time delays (minutes).
[0067] For example, such as Figure 2 and Figure 3 As shown, since order book basic data is required when querying data, the method described in this application further includes: generating order book basic data for each partition by performing basic data inversion on the entrusted transaction data of each partition through another Spark-Streaming, and storing the order book basic data of each partition in the basic library through the distributed message distribution system Kafka.
[0068] Two independent Spark Streaming methods are used to retrieve order book transaction data from the distributed message distribution system Kafka. Simultaneously, sharding, partitioning, and basic data inversion are performed. One Spark Streaming method directly calculates statistical data from the obtained order book basic data and stores it in a statistics database via Kafka. The other Spark Streaming method directly stores the obtained order book basic data in a basic database via Kafka. This improves the response performance of order book statistical data without affecting subsequent queries to the order book basic data.
[0069] As can be seen, in the above scheme, this application automatically realizes the time continuity management between batches by performing order book inversion through the Spark-Streaming streaming computing engine. By performing statistical data calculation on the basic order book data of each partition, the statistical data of each partition's order book is obtained, and the entrusted transaction data and the basic order book data of each partition are released. The basic order book data is not cached within a single batch, which greatly improves the performance of the entire data processing system and can handle tens of thousands of transaction streams per second.
[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0071] In one embodiment, a Spark-Streaming-based order book streaming inversion device is provided, which corresponds one-to-one with the Spark-Streaming-based order book streaming inversion method described in the above embodiments. For example... Figure 5 As shown, this Spark-Streaming-based order book streaming inversion device includes an acquisition module 101, a partitioning module 102, an inversion statistics module 103, and a writing module 104. Detailed descriptions of each functional module are as follows:
[0072] The acquisition module 101 is used to acquire the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka;
[0073] Partitioning module 102 is used to partition the entrusted transaction data in the distributed message distribution system Kafka;
[0074] The inversion statistics module 103 is used to perform basic data inversion on the order transaction data of each partition through Spark-Streaming to generate basic order book data for each partition, perform statistical data calculation on the basic order book data of each partition to obtain the order book statistical data of each partition, and release the order transaction data and the basic order book data of each partition.
[0075] The write module 104 is used to write the order book statistics data of each partition into the distributed message distribution system Kafka.
[0076] Specifically, the partitioning module 102 is also used to obtain multiple contract identifiers in the entrusted transaction data;
[0077] The entrusted transaction data is partitioned in the distributed message distribution system Kafka based on multiple contract identifiers.
[0078] Specifically, partitioning module 102 is also used to read the entrusted transaction data from the distributed message distribution system Kafka via Spark-Streaming;
[0079] Obtain multiple contract identifiers from the entrusted transaction data;
[0080] The entrusted transaction data is partitioned using Spark Streaming based on multiple contract identifiers.
[0081] Specifically, the partitioning module 102 is also used to construct a composite primary key, which includes the transaction date and the contract identifier;
[0082] The delegated transaction data in the Kafka distributed message distribution system is partitioned according to the composite primary key.
[0083] Specifically, the inversion statistics module 103 is also used to perform basic data inversion and statistical data calculation on the entrusted transaction data in batches using Spark-Streaming.
[0084] Specifically, the Spark-Streaming-based order book streaming inversion device also includes a storage module for storing the order book statistics data written to each partition of the distributed message distribution system Kafka into a statistics library.
[0085] In another Spark-Streaming application, the Spark-Streaming-based order book streaming inversion device includes the partitioning module 102, the inversion module, the writing module 104, and the storage module mentioned above. The partitioning module 102 has the same function as described above.
[0086] The inversion module is used to generate the basic order book data for each partition by performing basic data inversion on the entrusted transaction data of each partition using Spark-Streaming.
[0087] The storage module is used to store the basic order book data of each partition into the base library through the distributed message distribution system Kafka.
[0088] This invention automatically implements time continuity management between batches by performing order book inversion using the Spark-Streaming streaming computing engine. It obtains the order book statistics for each partition by performing statistical data calculation on the basic order book data of each partition, and releases the entrusted transaction data and the basic order book data of each partition. The basic order book data is not cached within a single batch, which greatly improves the performance of the entire data processing system and can handle tens of thousands of transaction streams per second.
[0089] Specific limitations regarding the Spark-Streaming-based order book streaming inversion apparatus can be found in the limitations of the Spark-Streaming-based order book streaming inversion method described above, and will not be repeated here. Each module in the aforementioned Spark-Streaming-based order book streaming inversion apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.
[0090] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external devices via a network connection. When executed by the processor, the computer program implements the functions or steps of a Spark-Streaming-based order book streaming inversion method on the server side.
[0091] In one embodiment, a computer device is provided, which may be a device terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a Spark-Streaming-based order book streaming inversion method on the device side.
[0092] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:
[0093] Obtain the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka;
[0094] The delegated transaction data in the Kafka distributed message distribution system is partitioned;
[0095] Spark-Streaming is used to perform basic data inversion on the order book data of each partition to generate basic order book data for each partition. Statistical data is calculated on the basic order book data of each partition to obtain the order book statistical data for each partition. Finally, the order book data and the basic order book data of each partition are released.
[0096] The order book statistics for each partition are written into the distributed message distribution system Kafka.
[0097] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0098] Obtain the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka;
[0099] The delegated transaction data in the Kafka distributed message distribution system is partitioned;
[0100] Spark-Streaming is used to perform basic data inversion on the order book data of each partition to generate basic order book data for each partition. Statistical data is calculated on the basic order book data of each partition to obtain the order book statistical data for each partition. Finally, the order book data and the basic order book data of each partition are released.
[0101] The order book statistics for each partition are written into the distributed message distribution system Kafka.
[0102] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and device side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0105] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A streaming inversion method for order books based on Spark-Streaming, characterized in that, include: Obtain the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka; The delegated transaction data in the Kafka distributed message distribution system is partitioned; Spark-Streaming is used to perform basic data inversion on the order book data of each partition to generate basic order book data for each partition. Statistical data is calculated on the basic order book data of each partition to obtain the order book statistical data for each partition. Finally, the order book data and the basic order book data of each partition are released. The order book statistics for each partition are written into the distributed message distribution system Kafka.
2. The order book streaming inversion method based on Spark-Streaming according to claim 1, characterized in that, The partitioning of the delegated transaction data in the Kafka distributed message distribution system includes: Obtain multiple contract identifiers from the entrusted transaction data; The entrusted transaction data is partitioned in the distributed message distribution system Kafka based on multiple contract identifiers.
3. The order book streaming inversion method based on Spark-Streaming according to claim 1, characterized in that, The partitioning of the delegated transaction data in the Kafka distributed message distribution system includes: The order transaction data is read from the distributed message distribution system Kafka using Spark Streaming; Obtain multiple contract identifiers from the entrusted transaction data; The entrusted transaction data is partitioned using Spark Streaming based on multiple contract identifiers.
4. The order book streaming inversion method based on Spark-Streaming according to any one of claims 1 to 3, characterized in that, The partitioning of the delegated transaction data in the Kafka distributed message distribution system includes: Construct a composite primary key, which includes the transaction date and the contract identifier; The delegated transaction data in the Kafka distributed message distribution system is partitioned according to the composite primary key.
5. The order book streaming inversion method based on Spark-Streaming according to claim 1, characterized in that, The method also includes: performing basic data inversion and statistical data calculation on the entrusted transaction data in batches using Spark-Streaming.
6. The order book streaming inversion method based on Spark-Streaming according to claim 1, characterized in that, The method further includes storing the order book statistics data written to each partition of the distributed message distribution system Kafka into a statistics database.
7. The order book streaming inversion method based on Spark-Streaming according to claim 1, characterized in that, The method further includes: The order book base data for each partition is generated by performing basic data inversion on the order transaction data of each partition using another Spark-Streaming method, and the order book base data of each partition is stored in the base library through the distributed message distribution system Kafka.
8. A Spark-Streaming-based order book streaming inversion device, characterized in that, include: The acquisition module is used to acquire the entrusted transaction data and import the entrusted transaction data into the distributed message distribution system Kafka; The partitioning module is used to partition the delegated transaction data in the distributed message distribution system Kafka; The inversion statistics module is used to perform basic data inversion on the order transaction data of each partition using Spark-Streaming to generate basic order book data for each partition, perform statistical data calculation on the basic order book data of each partition to obtain the order book statistical data of each partition, and release the order transaction data and the basic order book data of each partition. The write module is used to write the order book statistics data of each partition into the distributed message distribution system Kafka.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the order book streaming inversion method based on Spark-Streaming as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the order book streaming inversion method based on Spark-Streaming as described in any one of claims 1 to 7.