Plug-in micro-core architecture and financial market data processing method
By using a pluggable microkernel architecture to convert heterogeneous financial market data into standard data, the stability and efficiency issues of data access and processing in financial transactions are solved, enabling the generation of low-latency, high-stability investment signals and improving the data processing capabilities of trading operations.
Patent Information
- Application Number
- CN202510942829.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-28
Smart Images
Figure CN120848976A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of financial data processing technology, and in particular to a pluggable microkernel architecture and a method for processing financial market data. Background Technology
[0002] In financial trading scenarios, financial market data is characterized by real-time nature, diversity, and complexity. On the one hand, the structures of publicly available financial market data from different exchanges and / or different types of trading products vary greatly. On the other hand, different securities firms have different data development standards. Even for the same type of contract on the same exchange, the market data disclosed by different securities firms are completely different, resulting in a large number of heterogeneous financial market data results.
[0003] To address the aforementioned issues, a common solution is to establish a unified internal standard financial market data structure for different trading instruments, design corresponding programs to transcode the acquired heterogeneous data, and output it in a unified standard format. The next-level core components then compare, calculate, and filter market data from different sources, ultimately inputting different strategies to generate trading signals for participation in the financial market. However, the financial market constantly generates massive amounts of market data, and new trading instruments, exchanges, brokerage firms, and newly developed data models are constantly emerging. If a new transcoding program is developed for each new brokerage firm, exchange, or trading instrument, it becomes problematic. Firstly, each new transcoding program requires a separate development equipment, resulting in high resource and manpower costs. Secondly, too many parallel programs increase the load on the core components' data access, integration, and filtering. Over time, this leads to frequent data loss, duplication, anomalies, and delays, resulting in poor stability and availability, making it difficult to generate reliable and effective trading signals. Summary of the Invention
[0004] To address the aforementioned issues, the first aspect of this application provides a pluggable microkernel architecture, comprising: at least one market data plugin, each of the market data plugins corresponding to a market data source, wherein the market data plugin converts non-standard market data obtained from the market data source into standard market data;
[0005] The plugin registry is used to enter metadata information for all the aforementioned market data plugins.
[0006] The plug-in management module controls the connection to the market data plug-in, dynamically loads or uninstalls the corresponding market data plug-in according to the plug-in registry based on the instructions, or controls the lifecycle of the corresponding market data plug-in.
[0007] The market data optimization module connects to the market data plugins, receives standard market data from all sources of the market data plugins, and uses an optimization algorithm to filter out the best data.
[0008] The market data matching module is connected to the market data optimization module and uses a matching algorithm to process the optimized data to produce multi-level market data that can be used to generate investment signals; and
[0009] The communication module is connected to the market data plugin, the market data optimization module, and the market data matching module, and distributes the multi-level market data to the downstream server.
[0010] In some alternative embodiments, the market data plugins are deployed in a containerized manner and thus isolated from each other.
[0011] In some optional embodiments, the market data plugin is a COM component, which includes a startup interface, a shutdown interface, a data acquisition callback interface, and an exception reporting interface.
[0012] In some optional embodiments, the communication module is provided with a communication framework, and the market data plugin, the market data optimization module and the market data matching module are located at different levels of the communication framework and are decoupled from each other.
[0013] In some optional embodiments, the communication framework provides a dedicated circular buffer for each market data plugin, and the standard market data is stored in the circular buffer for retrieval by the market data optimization module.
[0014] In some optional embodiments, the circular buffer is a lock-free circular queue, which automatically adjusts its capacity according to the inbound rate of the standard market data.
[0015] The second aspect of this application provides a financial market data processing method, which employs a plug-in microkernel architecture according to any one of the above claims to convert heterogeneous non-standard market data from different market data sources into standard market data and output multi-level market data that can be used to generate investment signals. The method specifically includes the following steps:
[0016] S1: Create a new market data plugin for the newly accessed market data source, and synchronously enter the metadata information of the market data plugin into the plugin registry;
[0017] S2: The plug-in management module dynamically loads the market data plug-in and adds it to the pluggable microkernel architecture; S3: The market data plug-in is started to convert the non-standard market data from the market data source into the standard market data and output it to the communication module;
[0018] S4: The market data selection module obtains standard market data from multiple different market data plugin sources from the communication module, dynamically filters the selected data using an optimization algorithm, and outputs the selected data to the communication module; and
[0019] S5: The market data matching module receives the preferred data from the communication module, generates multiple market data using a matching algorithm, and outputs the multiple market data to the communication module.
[0020] In some optional implementations, the process of using an optimization algorithm to filter preferred data further includes the following steps:
[0021] S41: Confirm business requirements and associate the corresponding market data plugin;
[0022] S42: The communication module establishes the circular buffer for each associated market data plugin, and the market data plugin updates the tail pointer through atomic operations to write the standard market data into the circular buffer;
[0023] S43: The market data selection module obtains the standard market data in the circular buffer through the polling head pointer; and
[0024] S44: The market data optimization module uses an optimization algorithm to dynamically filter the standard market data received from different market data plugins, and the optimization algorithm includes the ant colony algorithm.
[0025] In some optional implementation methods, the optimization algorithm is an ant colony algorithm, and the screening of the optimization data further includes the following steps:
[0026] S441: Abstract the standard market data from different market data plugins into different independent paths, and initialize the initial pheromone concentration for each path;
[0027] S442: Calculate the data latency and data stability index for each path, and convert them into pheromone increments;
[0028] S443: Establish a pheromone-time simulation function, calculate the probability distribution of the magnitude of the second derivative of the simulation function at the next time point, and predict the optimal path; and
[0029] S444: Select the corresponding standard market data as the preferred data for the optimal path.
[0030] In some optional implementation methods, the generation of multi-level market data using the matching algorithm further includes the following steps:
[0031] S51: Classify the preferred data and establish a communication channel for each type of preferred data; S52: Each communication channel output corresponds to a specific thread. The preferred data is input one by one into the corresponding communication channel and assigned to the corresponding thread for independent matching;
[0032] S53: In a single thread, each piece of preferred data is further categorized according to the delegate type, and the preferred data is packaged accordingly; and
[0033] S54: Merge the preferred data of the same order type in different threads and output the multi-level market data.
[0034] This application has at least the following technical effects:
[0035] 1) The first aspect of this application provides a pluggable microkernel architecture. Through pluggable design, it connects to specific market data structures, maximizing the access efficiency of market data sources and ensuring good scalability and highly customizable development capabilities of market data sources. The pluggable management module can dynamically load the market plugins during the operation of this pluggable microkernel architecture without restarting, flexibly managing the lifecycle of market plugins, realizing fast and multi-adaptable market access flow, and avoiding intrusion into core paths such as subsequent market data selection and market data matching. The market data selection module performs path filtering and can output low-latency, high-stability selected data. The market data matching module merges the selected data to generate multiple levels of market data, which are distributed to downstream servers. This can be used to generate investment signals and reduce the end-to-end total latency of financial market data, that is, the time interval from the issuance of individual market data by securities firms, exchanges, and other market data sources to the generation of multiple levels of market data. According to calculations, it can significantly shorten the total latency, for example, to 20%-50% of the conventional value, and provide faster and more timely basic market data for the subsequent workflow of the trading business chain.
[0036] 2) The second aspect of this application provides a financial market data processing method. Based on the above-mentioned plug-in microkernel architecture, the financial market data processing method in this application has the characteristics of rich market data sources, high data processing efficiency, low latency, high stability and strong reliability of data processing results. Attached Figure Description
[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 This is a schematic diagram of a pluggable microkernel architecture provided in this application;
[0039] Figure 2 This is a flowchart of a financial market data processing method provided in this application. Detailed Implementation
[0040] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0041] It should be noted that the terms "comprising" and "having" and any variations thereof in this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes or methods.
[0042] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The term "exemplary" as used herein means "serving as an example, embodiment, or illustration." Any embodiment illustrated herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.
[0043] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented even without certain specific details. In some instances, methods and means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.
[0044] See Figure 1 The first aspect of this application provides a pluggable microkernel architecture, including:
[0045] At least one market data plugin, each of which is connected to a market data source, and the market data plugin converts non-standard market data obtained from the market data source into standard market data;
[0046] The plugin registry is used to enter metadata information for all the aforementioned market data plugins.
[0047] The plug-in management module controls the connection to the market data plug-in, dynamically loads or uninstalls the corresponding market data plug-in according to the plug-in registry based on the instructions, or controls the lifecycle of the corresponding market data plug-in.
[0048] The market data optimization module connects to the market data plugins, receives standard market data from all sources of the market data plugins, and uses an optimization algorithm to filter out the best data.
[0049] The market data matching module is connected to the market data optimization module and uses a matching algorithm to process the optimized data to produce multi-level market data that can be used to generate investment signals; and
[0050] The communication module is connected to the market data plugin, the market data optimization module, and the market data matching module, and distributes the multi-level market data to the downstream server.
[0051] By employing a plug-in design to interface with specific market data structures, the system maximizes the efficiency of market data source access, ensuring excellent scalability and highly customizable development capabilities. The plug-in management module dynamically loads market data plug-ins during the runtime of this plug-in microkernel architecture without requiring a restart, flexibly managing the lifecycle of plug-in market data and enabling fast, multi-adaptable market data access flows while avoiding intrusion into subsequent core paths such as market data selection and matching. The market data selection module performs path filtering, outputting low-latency, highly stable selected data. The market data matching module merges the selected data to generate multiple levels of market data, which are then distributed to downstream servers. This can be used to generate investment signals, reducing the end-to-end total latency of financial market data—the time interval between the issuance of individual market data from brokerages, exchanges, and other market data sources and the generation of multiple levels of market data. This allows for faster and more timely provision of basic market data to subsequent workflows in the trading business chain. In practical implementation, the average total latency using the plug-in microkernel architecture provided in this application can be controlled within 80-200μs, compared to the average total latency of 400μs for quantitative private equity funds, reducing the total latency to 20%-50% of the average.
[0052] Furthermore, the metadata information of the market data plugin includes: identity ID, brokerage and exchange source information, contract type, COM interface list, configuration parameters, etc. For example, when adding a new brokerage interface, the inventors design specific transcoding logic for the new brokerage's market data and form a market data plugin. After the market data plugin is integrated into this plugin-based microkernel architecture, input commands control the plugin management module to dynamically load the market data plugin to complete the integration, significantly shortening the cycle of introducing new market data sources.
[0053] It is understood that the standard market data can be in the form of data packets, and each standard market data packet may include: tick-by-tick orders from the market information source, market snapshots, order queues, and other information. The format of the standard market data is uniformly based on an internal protocol, such as Protobuf encoding, and may include core fields such as timestamps, prices, and trading volumes.
[0054] Further, see Figure 2According to the pluggable microkernel architecture, a financial market data processing method is used to convert heterogeneous non-standard market data from different market data sources into standard market data and output multi-level market data that can be used to generate investment signals. The method specifically includes the following steps:
[0055] S1: Create a new market data plugin for the newly accessed market data source, and synchronously enter the metadata information of the market data plugin into the plugin registry;
[0056] S2: The plug-in management module dynamically loads the market data plug-in and adds it to the pluggable microkernel architecture; S3: The market data plug-in is started to convert the non-standard market data from the market data source into the standard market data and output it to the communication module;
[0057] S4: The market data selection module obtains standard market data from multiple different market data plugin sources from the communication module, dynamically filters the selected data using an optimization algorithm, and outputs the selected data to the communication module; and
[0058] S5: The market data matching module receives the preferred data from the communication module, generates multiple market data using a matching algorithm, and outputs the multiple market data to the communication module.
[0059] By adopting the above-mentioned financial market data processing methods, we can expand the data source channels for market data, improve data processing efficiency, and obtain data processing results with characteristics such as low latency, high stability, and high reliability, which is conducive to forming timely and high-quality investment signals.
[0060] In some optional embodiments, the market data plugins are deployed in a containerized manner to isolate them from each other. In practice, the market data plugins are deployed within Docker containers and run independently of each other, thus preventing interference, isolating faults to enhance stability, optimizing resource allocation to improve performance, and facilitating maintenance and updates.
[0061] In some optional embodiments, the market data plugin is a COM component, which includes a startup interface, a shutdown interface, a data acquisition callback interface, and an exception reporting interface. Specifically, the startup interface is used to initialize the market data plugin's operating environment and establish a connection with the market data source; the shutdown interface is used to safely release the market data plugin's resources and disconnect external connections; the data acquisition callback interface is used to receive non-standard market data and perform standardized transcoding; and the exception reporting interface is used to report plugin operating status anomalies in real time. Furthermore, by using the dependency inversion and control inversion principles, the market data plugin is abstracted as an interface rather than a specific plugin, enabling precise control over the dependencies between each market data plugin and the market data optimization module, ensuring the scalability of the market data plugin.
[0062] In some optional embodiments, the communication module is provided with a communication framework, and the market data plugin, the market data optimization module and the market data matching module are located at different levels of the communication framework and are decoupled from each other.
[0063] For example, the communication framework is the Disruptor framework, the market data plugin is connected to the bottom layer of the Disruptor framework, the market data optimization module is connected to the middle layer, and the market data matching module is connected to the top layer.
[0064] In some optional embodiments, the communication framework provides a dedicated circular buffer for each market data plugin, and the standard market data is stored in the circular buffer for retrieval by the market data optimization module. It is understood that the circular buffer is used to connect the bottom and middle layers of the Disruptor framework, enabling bidirectional information flow between the market data plugin and the market data optimization module.
[0065] Furthermore, the data interaction method between the market data plugin and the market data optimization module is as follows:
[0066] S41: Confirm business requirements and associate the corresponding market data plugin;
[0067] S42: The communication module establishes a circular buffer for each associated market data plugin; the market data plugin updates the tail pointer through atomic operations and writes the standard market data to the circular buffer; and
[0068] S43: The market data selection module obtains the standard market data in the circular buffer by polling the head pointer.
[0069] In some optional embodiments, the circular buffer is a lock-free circular queue, whose capacity is automatically adjusted according to the standard market data enqueue rate. Specifically, when the standard market data arrival rate is high, the queue capacity is increased. Dynamic expansion reduces the chance of the queue being full, avoiding frequent waiting for enqueues by the market data plugin thread, and even data loss, thereby improving the overall throughput of the architecture, reducing latency, and ensuring the timely execution of subsequent transactions. Conversely, when the standard market data arrival rate is low, the queue capacity can be reduced. Shrinking the queue reduces memory usage, optimizes resource utilization, and indirectly improves the overall response speed of the architecture.
[0070] In some optional implementations, the process of using an optimization algorithm to filter preferred data further includes the following steps:
[0071] S44: The market data optimization module uses an optimization algorithm to dynamically filter the standard market data received from different market data plugins, and the optimization algorithm includes the ant colony algorithm.
[0072] Furthermore, the process of filtering the preferred data using the ant colony algorithm also includes the following steps:
[0073] S441: Abstract the standard market data from different market data plugins into different independent paths, and initialize the initial pheromone concentration for each path;
[0074] S442: Calculate the data latency and data stability index for each path, and convert them into pheromone increments;
[0075] S443: Establish a pheromone-time simulation function, calculate the probability distribution of the magnitude of the second derivative of the simulation function at the next time point, and predict the optimal path; and
[0076] S444: Select the corresponding standard market data as the preferred data for the optimal path.
[0077] The sign of the second derivative of the pheromone-time function reflects the changing trend of pheromone concentration. A positive second derivative indicates that the rate of increase in pheromone concentration is accelerating, or the rate of decrease is slowing down. An accelerating rate of increase means the path's attractiveness is rapidly increasing; a slowing rate of decrease means the decreasing trend in the path's attractiveness is slowing down. Conversely, a negative second derivative indicates that the rate of increase in pheromone concentration is slowing down, or the rate of decrease is accelerating. A slowing rate of increase means the increasing attractiveness of the path is weakening; an accelerating rate of decrease means the attractiveness of the path is rapidly decreasing.
[0078] By analyzing the probability distribution of changes in the magnitude of the second derivative of the pheromone-time function, the probability of the optimal path turning point can be predicted, thus enabling the prediction of the optimal path. Specifically: if the second derivative of the pheromone-time function for a certain path is positive and continuously increasing, it indicates that the pheromone concentration of this path may soon rise rapidly, attracting more "ants" and generating more subsequent market data. Conversely, if the second derivative is negative and its absolute value increases, it may indicate that the attractiveness of this path will soon weaken, requiring advance prediction and adjustment of the path selection strategy.
[0079] In some optional implementation methods, the generation of multi-level market data using the matching algorithm further includes the following steps:
[0080] S51: Classify the preferred data and establish a communication channel for each type of preferred data; S52: Each communication channel output corresponds to a specific thread. The preferred data is input one by one into the corresponding communication channel and assigned to the corresponding thread for independent matching;
[0081] S53: In a single thread, each piece of preferred data is further categorized according to the delegate type, and the preferred data is packaged accordingly; and
[0082] S54: Merge the preferred data of the same order type in different threads and output the multi-level market data.
[0083] Specifically, the order types include: market order, limit order, stop-loss order, take-profit order, stop-loss limit order, etc. The multi-level market data can be a three-dimensional data matrix composed of multi-dimensional data such as order book depth information, tick transaction details information, and market status indicator information, which can be used to directly generate quantitative investment signals to guide financial operations.
[0084] In the description of this specification, references to terms such as "some embodiments," "for example," or "example" indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this embodiment. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0085] Although this embodiment has been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made without departing from the principles and spirit of this embodiment, the scope of which is defined by the claims and their equivalents.
Claims
1. A pluggable microkernel architecture, characterized in that, include: At least one market data plugin, each of which is connected to a market data source, and the market data plugin converts non-standard market data obtained from the market data source into standard market data; The plugin registry is used to enter metadata information for all the aforementioned market data plugins. The plug-in management module controls the connection to the market data plug-in, dynamically loads or uninstalls the corresponding market data plug-in according to the plug-in registry based on the instructions, or controls the lifecycle of the corresponding market data plug-in. The market data optimization module connects to the market data plugins, receives standard market data from all sources of the market data plugins, and uses an optimization algorithm to filter out the best data. The market data matching module is connected to the market data optimization module and uses a matching algorithm to process the optimized data to produce multi-level market data that can be used to generate investment signals. as well as The communication module is connected to the market data plugin, the market data optimization module, and the market data matching module, and distributes the multi-level market data to the downstream server.
2. The pluggable microkernel architecture according to claim 1, characterized in that, The market data plugins are isolated from each other through containerized deployment.
3. The pluggable microkernel architecture according to claim 1, characterized in that, The market data plugin is a COM component, which includes a startup interface, a shutdown interface, a data acquisition callback interface, and an exception reporting interface.
4. The pluggable microkernel architecture according to claim 1, characterized in that, The communication module is provided with a communication framework, and the market data plugin, the market data optimization module and the market data matching module are located at different levels of the communication framework and are decoupled from each other.
5. The pluggable microkernel architecture according to claim 4, characterized in that, The communication framework provides a dedicated circular buffer for each market data plugin, and the standard market data is stored in the circular buffer for retrieval by the market data optimization module.
6. The pluggable microkernel architecture according to claim 5, characterized in that, The circular buffer is a lock-free circular queue, and the lock-free circular queue automatically adjusts its capacity according to the inbound rate of the standard market data.
7. A method for processing financial market data, characterized in that, Using the pluggable microkernel architecture according to any one of claims 1-6, heterogeneous non-standard market data from different market data sources is transcoded into the standard market data and outputs multi-level market data that can be used to generate investment signals. The method specifically includes the following steps: S1: Create a new market data plugin for the newly accessed market data source, and synchronously enter the metadata information of the market data plugin into the plugin registry; S2: The plug-in management module dynamically loads the market data plug-in and adds it to the plug-in microkernel architecture; S3: Activate the market data plugin to convert the non-standard market data from the market data source into the standard market data, and output it to the communication module; S4: The market data selection module obtains standard market data from multiple different market data plugin sources from the communication module, dynamically filters the selected data using an optimization algorithm, and outputs the selected data to the communication module; and S5: The market data matching module receives the preferred data from the communication module, generates multiple market data using a matching algorithm, and outputs the multiple market data to the communication module.
8. The financial market data processing method according to claim 7, characterized in that, The process of using an optimization algorithm to select preferred data also includes the following steps: S41: Confirm business requirements and associate the corresponding market data plugin; S42: The communication module establishes the circular buffer for each associated market data plugin, and the market data plugin updates the tail pointer through atomic operations to write the standard market data into the circular buffer; S43: The market data selection module obtains the standard market data in the circular buffer through the polling head pointer; and S44: The market data optimization module uses an optimization algorithm to dynamically filter the standard market data received from different market data plugins, and the optimization algorithm includes the ant colony algorithm.
9. The financial market data processing method according to claim 8, characterized in that, The optimization algorithm is an ant colony algorithm, and the selection of the optimization data also includes the following steps: S441: Abstract the standard market data from different market data plugins into different independent paths, and initialize the initial pheromone concentration for each path; S442: Calculate the data latency and data stability index for each path, and convert them into pheromone increments; S443: Establish a pheromone-time simulation function, calculate the probability distribution of the magnitude of the second derivative of the simulation function at the next time point, and predict the optimal path; and S444: Select the corresponding standard market data as the preferred data for the optimal path.
10. The financial market data processing method according to claim 7, characterized in that, The process of generating multi-level market data using a matching algorithm also includes the following steps: S51: Classify the preferred data and establish a communication channel for each type of preferred data; S52: Each communication channel output corresponds to a specific thread. The preferred data is input one by one into the corresponding communication channel and assigned to the corresponding thread for independent matching. S53: In a single thread, each piece of preferred data is further categorized according to the delegate type, and the preferred data is packaged accordingly; and S54: Merge the preferred data of the same order type in different threads and output the multi-level market data.