A distributed multi-channel market information processing method and system

Abstract

The application discloses a distributed multi-path market situation processing method and system, relates to the field of data processing, and comprises the following steps: receiving multi-path market situation data; selecting one path with the minimum time delay as optimal market situation data; adding basic information of the optimal market situation data; acquiring supplementary information corresponding to a security code by querying a security database according to the basic information; generating security static data according to the supplementary information and the optimal market situation data containing the basic information; generating corresponding K-line data and derivative data according to the optimal market situation data containing the basic information; subscribing to the optimal market situation data containing the basic information, the K-line data and the derivative data from a message queue service by using a streaming subscription through a market situation pushing service; and reducing the processing time delay of market situation data by receiving multi-path market situation data through multiple access points, selecting one path with the minimum time delay as optimal market situation data and the like.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and system for processing distributed multi-channel market information. Background Art

[0002] With the rapid development of financial markets and the increasing popularity of quantitative trading, traditional market information systems are no longer able to meet the real-time, stability, and data diversity requirements of quantitative strategy trading systems. Quantitative trading systems must process massive amounts of historical market data for strategy backtesting, while also providing real-time market data to support real-time strategy calculations and order placement. Furthermore, quantitative trading systems also require the calculation of various aggregated and derived indicators based on market data, placing higher demands on the market information system's data processing capabilities and responsiveness.

[0003] Most existing market data systems utilize a centralized architecture, centralizing market data from various exchanges and data providers to a single market data server for processing. This architecture has numerous drawbacks: First, a single market data server struggles to handle the high concurrency of quantitative trading systems, easily becoming a performance bottleneck. Second, centralized architectures lack flexible data integration and scalability, making it difficult to quickly introduce new market data sources and data types. Third, a single point of failure in the market data server can paralyze the entire trading system, posing significant systemic risks.

[0004] Furthermore, existing market data processing and distribution systems also suffer from shortcomings. Different market data sources provide data in varying formats, lacking unified data specifications and standardized processing, resulting in poor data readability and usability. The aggregated data generated by market data systems, such as K-line charts and time-sharing data, as well as various derivative indicators, often fail to meet the personalized needs of quantitative trading systems. Market data transmission delays are significant, making it difficult to support the real-time requirements of high-frequency trading strategies. Summary of the Invention

[0005] In response to the problem of long processing delay of multi-channel market data in the existing technology, the present application provides a distributed multi-channel market data processing method and system, which receives multi-channel market data through multiple access points, selects the channel with the smallest delay as the optimal market data, etc., thereby reducing the processing delay of market data.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] One aspect of the present application provides a distributed multi-channel market data processing method, including: S1, receiving multi-channel market data through multiple access points; S2, selecting a channel of market data with the smallest delay as the optimal market data, and accessing the selected optimal market data to a message queue; obtaining the optimal market data containing the basic information by adding basic information of the optimal market data; S3, obtaining supplementary information corresponding to the securities code by querying a securities database based on the basic information; generating securities static data based on the supplementary information and the optimal market data containing the basic information; S4, generating corresponding K-line data and derivative data based on the optimal market data containing the basic information through data fitting; S5, using streaming subscription through a market push service to subscribe to the optimal market data, K-line data and derivative data containing the basic information from a message queue service; S6, receiving client requests through an interface gateway service, and forwarding the client requests to a query service.

[0008] The Market Data Push Service is a service component specifically designed to push real-time market data to clients. It interacts with the message queue service to obtain the latest market data and pushes it to subscribed clients as quickly as possible, enabling real-time market data updates. Market data push services typically utilize high-performance, low-latency communication protocols, such as WebSocket or TCP persistent connections, to ensure timely and stable data transmission. Furthermore, the market data push service can provide customized data push solutions based on client needs, such as subscriptions by security code or market type.

[0009] Streaming subscription is a data transmission and consumption model characterized by data being transmitted in a continuous, real-time stream. Data consumers (subscribers) can dynamically subscribe to and unsubscribe from the stream, consuming data in real time based on their processing capabilities. Compared to traditional data subscription models (such as scheduled polling), streaming subscription offers greater real-time data transmission and flexibility in data consumption, meeting the requirements of rapidly updating and dynamically changing market data. Common streaming subscription models include publish-subscribe and push.

[0010] A message queue service (Message Queue Service) is a middleware service used to enable asynchronous communication between different software components. By passing messages (such as market data) from message producers to message consumers, it decouples different components and improves system scalability and fault tolerance. Common message queue services include Apache RabbitMQ and ActiveMQ. In the aforementioned distributed multi-channel market processing method, the message queue service acts as a bridge connecting market data producers (such as the market data access module) and market data consumers (such as the market push service), enabling efficient transmission and sharing of market data between different modules. The message queue service also provides advanced features such as message persistence, message filtering, and message partitioning, further enhancing system reliability and performance.

[0011] Furthermore, basic information includes security code, security name and exchange code; supplementary information includes security type and trading status; and derivative data reflects security price trends and trading activity.

[0012] Furthermore, S2 selects the market data with the smallest delay as the optimal market data, and connects the selected optimal market data to the message queue, including: S21, storing the received multiple market data in the order of arrival time into the specified topic of the RabbitMQ stream processing engine, and associating a timestamp for each market data, which indicates the arrival time of the market data; S22, using the time window mechanism of the RabbitMQ stream processing engine, grouping the market data in the specified time window according to the securities code, and obtaining the market data stream corresponding to each securities code; S23, using the double-ended queue data structure to set a fixed-size sliding window according to the market data stream corresponding to each securities code, and storing the fixed-size sliding window in the sliding window. timestamp of the market data; S24, when new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window. If so, add the timestamp of the new market data to the end of the sliding window and delete the timestamp at the head of the sliding window; S25, obtain the market data corresponding to the minimum timestamp in the sliding window, and use it as the market data with the minimum delay for the corresponding security code in the current sliding window, and use it as the optimal market data; S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the specified topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the security code, and the value is the optimal market data.

[0013] In RabbitMQ, a topic is a concept similar to a database table or file system directory, used to store and manage a group of related message data. Each topic can be partitioned to enable parallel data processing and load balancing. Different types of message data can be published to different topics to achieve logical isolation and orderly data processing. In the above market processing method, by publishing market data for different securities codes to different topics, efficient storage and management of market data can be achieved, facilitating subsequent data processing and analysis.

[0014] The time window mechanism is a common data processing mechanism in stream processing, used to aggregate, calculate, or analyze data within a specific time range. By dividing the continuous data stream into different windows according to the time dimension, the data within each window can be processed as a whole, thus achieving time-sensitive and real-time data processing. Common time window types include tumbling windows, sliding windows, and session windows. In the above market processing method, by utilizing RabbitMQ's time window mechanism, market data within a specified time range can be grouped by security code, resulting in a market data stream corresponding to each security code, providing the data foundation for subsequent optimal market selection.

[0015] In RabbitMQ, the Producer API is a programming interface used to publish message data to RabbitMQ topics. Through the Producer API, applications can serialize message data into byte arrays, specify message metadata (such as topic, partition, key, etc.), and then send the message to the RabbitMQ cluster. RabbitMQ provides Producer APIs in multiple programming languages, such as Java, Python, and Go, to facilitate message publishing in different application scenarios and development environments. In the above market processing method, by utilizing RabbitMQ's Producer API, the selected optimal market data can be published to a designated RabbitMQ topic in the form of key-value pairs for subsequent data consumption and processing.

[0016] Furthermore, S3 generates securities static data based on the supplementary information and the optimal market data containing the basic information, including: S31, extracting the securities code from the optimal market data containing the basic information as a query condition, and querying the supplementary information corresponding to the securities code in the securities database through a predefined database query interface; S32, according to predefined data encapsulation rules, associating the supplementary information with the optimal market data containing the basic information to generate securities static data; parsing the securities static data into the following Go language data structure, where the data structure includes Code for the securities code, Name for the securities name, Exchange for the exchange code, Type for the securities type, and Status for the transaction status.

[0017] S33, uses the Protobuf data serialization format to serialize the securities static data. The serialization steps are as follows: generate the corresponding Protobuf message definition file (.proto) based on the Go language structure of the securities static data; use the Protobuf compiler to compile the .proto file into a Go language code file; reference the generated code in the Go code; create a Protobuf message object corresponding to the securities static data and fill in the fields; call the serialization method of the Protobuf message object to generate binary serialized data; publish the serialized securities static data to the specified topic of the RabbitMQ stream processing engine, using the securities code as the routing key (Routing Key) and the serialized binary data as the message body; S34, synchronize the securities static data in the RabbitMQ stream processing engine to the distributed columnar storage database, using the securities code as the distinguishing key and the various fields of the securities static data as columns; S35, when the supplementary information in the securities database changes, synchronize the change information to the RabbitMQ stream processing engine through the Binlog-based database change mechanism.

[0018] Furthermore, S4 generates corresponding K-line data and derivative data based on the optimal market data containing basic information through data fitting, including: S41, obtaining the optimal market data containing basic information from the designated topic of the RabbitMQ stream processing engine, and grouping and sorting the optimal market data according to the securities code and timestamp to obtain the time series market data corresponding to each securities code; S42, performing K-line synthesis on the time series market data corresponding to each securities code to generate corresponding K-line data; S43, calculating the technical indicators of the time series market data corresponding to each securities code to generate corresponding derivative data; S44, associating the generated K-line data and derivative data according to the securities code and timestamp, and publishing the associated data to the designated topic of the RabbitMQ stream processing engine; S45, synchronizing the K-line data and derivative data in the RabbitMQ stream processing engine to the distributed time series database, with the securities code as the partition key, the timestamp as the sort key, and the various fields of the K-line data and derivative data as measurement indicators.

[0019] Further, S42, K-line synthesis is performed on the time series market data corresponding to each securities code to generate corresponding K-line data, including: setting the time window size of the K-line synthesis according to the preset K-line period, and dividing the time series market data according to the time window size to obtain a continuous time window sequence; calculating the indicators of each time window sequence to generate the K-line data of the corresponding time window; the indicators include opening price, closing price, highest price, lowest price and trading volume; using the 3σ principle to denoise the K-line data to obtain denoised K-line data; based on the denoised K-line data, by calculating the price change rate and trading volume change rate between adjacent K-line data points, a trend feature sequence of the K-line data is obtained; based on the trend feature sequence, run-length encoding is used to compress the data to obtain the final K-line data.

[0020] Furthermore, the opening price is the price of the first market data in the time window; the closing price is the price of the last market data in the time window; the highest price is the maximum value of all market data prices in the time window; the lowest price is the minimum value of all market data prices in the time window; and the trading volume is the sum of the trading volumes of all market data in the time window.

[0021] Further, S43, calculate the technical indicators of the time series market data corresponding to each securities code and generate corresponding derivative data, including: calculating the moving average indicator based on the K-line data, the moving average indicator includes 、 、 、 and ;in, The calculation formula is: ,in, represents the closing price of the i-th day, and n represents the number of periods of the moving average; the exponential smoothing moving average indicator is calculated based on the time series market data and K-line data. The exponential smoothing moving average indicator includes and ;in, Calculated by the following recursive formula: ,in, , n represents the period of the exponential moving average, Indicates the closing price of the current trading day. Indicates the value of the n-day exponential moving average of the previous trading day; and Calculate the MACD indicator, which includes DIF, DEA, and MACD; wherein DIF, DEA, and MACD are calculated using the following formula: , , ,in, The 9-day exponential moving average of DIF is represented. Volume indicators are calculated based on time series market data. The volume indicators include VOL and AMT. VOL and AMT are calculated using the following formulas: , ,in, Indicates the transaction volume of the i-th market data, represents the price of the i-th market data; the volume ratio indicator VR is calculated based on VOL; the VR is calculated by the following formula: ,in, Indicates the trading volume of the current trading day. The 5-day simple moving average of trading volume; the price change rate indicator is calculated based on time series market data, and the price change rate indicator includes and ;in, and Calculated by the following formula: , ,in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Indicates the closing price 5 trading days ago; based on the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator ; Calculated by the following formula: , , where n represents the number of calculation cycles, Indicates the trading volume of the current trading day. Indicates the trading volume n trading days ago; the calculated 、 , MACD, VOL, AMT, VR, and As derivative data, it is associated with the corresponding securities code and timestamp, and the associated derivative data is arranged in the order of securities code and timestamp to generate the final derivative data result.

[0022] Furthermore, it also includes: S6, receiving the client request through the interface gateway service, and forwarding the client request to the query service; the query service queries the corresponding securities static data from the securities database according to the client request, and obtains the corresponding K-line data and derivative data from the market push service; the securities static data, K-line data and derivative data are combined and packaged to obtain the packaged client data, and the packaged client data is returned to the interface gateway service; the interface gateway service sets a response time threshold for the client request. For the client request that exceeds the response time threshold, the interface gateway service terminates the processing of the corresponding client request and returns a request timeout prompt message to the client.

[0023] Distributed service fault handling based on interface monitoring and dynamic expansion includes: S61, using the distributed tracking system to track and monitor the request call links of each service through the interface monitoring service, collecting service request volume, response time, error rate and other indicator data, and using the log aggregation platform to collect the service operation log, extract the error information and exception stack in the log; S62, the interface monitoring service uses the stream processing engine to perform real-time calculation and rule matching on the indicator data collected in step S61. The stream processing engine is based on memory computing technology, processes the indicator data in the form of event stream in memory, and judges the service through predefined aggregation functions and anomaly detection algorithms. Whether the running status meets the preset normal status threshold, and the calculation results are output to the downstream system in the form of a stream; S63, if the running status of the service does not meet the normal status threshold, the interface monitoring service uses the service orchestration engine to automatically generate a fault handling workflow according to the preset fault handling strategy. The fault handling workflow includes a series of fault diagnosis, recovery operations and compensation operations. The workflow engine coordinates the calls and executions between different services to achieve self-healing and recovery of the service; at the same time, the interface monitoring service uses the intelligent alarm system to perform semantic analysis and correlation analysis on the fault handling information, automatically generates fault reports and handling suggestions, and sends them to the operation and maintenance personnel through the multi-channel alarm platform.

[0024] S64: The interface monitoring service uses a time series database to perform persistent storage and aggregate calculations on the indicator data collected in step S61. The time series database uses columnar storage and compression technology to support efficient data writing and querying. Through predefined aggregation functions and window functions, it performs sliding window calculations and multi-dimensional analysis on the indicator data to determine whether the service load status exceeds the preset load threshold, and displays the calculation results in the form of a time series on the monitoring screen.

[0025] S65: If the load status of the service exceeds the load threshold, the interface monitoring service uses an adaptive expansion algorithm to dynamically calculate the optimal number of service instances based on the preset expansion strategy. The adaptive expansion algorithm is based on reinforcement learning technology and continuously optimizes the expansion strategy and parameters through feedback from historical load data and expansion effects to achieve intelligent elastic scaling of the service. At the same time, the interface monitoring service uses shadow deployment technology to verify and preheat the expanded service instances without affecting online business, ensuring smooth expansion and stable operation of the service.

[0026] S66, the dynamic expansion mechanism creates a distributed copy instance of the service through the container orchestration platform, and registers the newly created service instance to the service registration center. At the same time, it updates the routing rules of the interface gateway service; S67, the interface gateway service uses a load balancing algorithm to dynamically distribute client requests to different service instances based on the updated routing rules and the service instance information provided by the service registration center. The load balancing algorithm includes polling, random, minimum number of connections, weighted, etc.; S68, after the expanded service instance processes the request, it returns the processing result to the interface gateway service. The interface gateway service integrates the processing result and returns it to the client. At the same time, it reports the performance indicators of the request processing to the interface monitoring service; the interface monitoring service dynamically adjusts the number of service instances according to the performance indicators and the load status of the service to achieve elastic scaling of the service.

[0027] Another aspect of the present application further provides a distributed multi-channel market information processing system for executing a distributed multi-channel market information processing method of the present application.

[0028] Compared with the existing technology, the advantages of this application are:

[0029] By receiving multiple data streams from multiple access points, this solution leverages the RabbitMQ stream processing engine's time window mechanism and sliding window algorithm to compare the latency of data from different access points in real time and select the stream with the lowest latency as the optimal data stream. Compared to existing technologies, this solution avoids invalid processing of data with significant latency, significantly reducing overall data processing latency.

[0030] After selecting the optimal market data, this solution uses data augmentation to extract basic and supplementary information related to the market data from the securities database, generating complete static securities data. Simultaneously, a data fitting algorithm is used to generate corresponding K-line data and derivative data based on the optimal market data. Compared with existing technologies, this solution fully taps into the potential value of market data and maximizes its utilization through data correlation and derivative indicator calculation.

[0031] For static securities data, this solution utilizes distributed columnar storage, enabling efficient storage and fast querying of massive amounts of data. For K-line and derivative data, this solution employs a distributed time-series database, supporting highly compressed data storage and fast range queries. Furthermore, leveraging the streaming subscription mechanism of the RabbitMQ message queue, this solution enables real-time push of market data, K-line data, and derivative data. Compared to existing technologies, this solution meets the real-time requirements of market data while ensuring efficient data storage and querying.

[0032] This solution introduces an interface gateway service and query service to provide unified access and routing for client requests. Setting a request timeout threshold prevents system crashes caused by request accumulation. Furthermore, the interface monitoring service provides real-time access to the operational status of each service. Service load thresholds are set, and combined with fault handling and dynamic scaling mechanisms, service instances are automatically created based on load, ensuring high system availability. Furthermore, this solution utilizes a microservices architecture and containerized deployment, enabling elastic service expansion and rapid deployment, demonstrating excellent scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present application will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0034] Figure 1 This is an exemplary flow chart of a method for processing distributed multi-channel market information according to some embodiments of the present application. DETAILED DESCRIPTION

[0035] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] like Figure 1As shown, a distributed multi-channel market data processing method includes: receiving multi-channel market data through multiple access points; selecting a channel with the smallest time delay as the optimal market data, and accessing the selected optimal market data to the message queue; obtaining the optimal market data containing the basic information by adding the basic information of the optimal market data; querying the securities database according to the basic information to obtain the supplementary information corresponding to the securities code; generating the securities static data according to the supplementary information and the optimal market data containing the basic information; generating the corresponding K-line data and derivative data through data fitting according to the optimal market data containing the basic information; subscribing the optimal market data containing the basic information, K-line data and derivative data from the message queue service through the streaming subscription of the market push service; receiving the client request through the interface gateway service, and forwarding the client request to the query service.

[0037] Specifically, S1, multi-channel market data is received through multiple access points; the system sets multiple market data access points, such as Shanghai Stock Exchange, Shenzhen Stock Exchange, Hong Kong Stock Exchange, etc., each access point is connected to the market publishing system of the exchange through a dedicated line or VPN, and real-time Level-1, Level-2 and other market data published by each exchange is received. Taking the above Shanghai Level-1 market as an example, real-time market data is received through TCP protocol, the data format is binary, and it contains securities code, latest price, trading volume, buy one price, buy one volume, sell one price, sell one volume and other fields.

[0038] S2, selecting a channel with the smallest time delay as the optimal market data, and accessing the selected optimal market data to the message queue, including: S21, storing the received multi-channel market data in the specified topic of the RabbitMQ stream processing engine in the order of arrival time, and associating a timestamp with each market data, the timestamp indicating the arrival time of the market data. In the specific implementation, the source connector (SourceConnector) of RabbitMQ Connect can be used to write real-time market data of different exchanges into different topics (Topic) of RabbitMQ cluster. Each topic corresponds to a market data stream of an exchange, and the data format can adopt structured formats such as JSON and protobuf, and contains securities code, price, trading volume and other fields. When writing to RabbitMQ, a custom partitioner (Partitioner) can be used to allocate market data of the same securities code to the same partition (Partition) to ensure the orderliness of the data. At the same time, a timestamp field can be added to each market data to record the arrival time of the data, the timestamp can be the time of market publishing or the time of market receiving, and the time accuracy is recommended to reach millisecond level.

[0039] S22 utilizes the RabbitMQ stream processing engine's time window mechanism to group market data within a specified time window by security code, generating a corresponding market data stream for each security code. In specific implementations, the time window APIs (such as Time Windows and Session Windows) provided by the RabbitMQ Streams DSL can be used to group and aggregate market data within a certain time range by security code. The length of the time window can be set based on the market data update frequency and real-time requirements, typically between 100 milliseconds and 1 second. After grouping and aggregation, each security code has a corresponding KStream or KTable, representing the market data stream or status table for that security.

[0040] S23, according to the market data stream corresponding to each securities code, a fixed-size sliding window is set using a double-ended queue data structure, and the sliding window stores the timestamp of the market data. In the RabbitMQ Streams application, a Key Value Store is defined using the Stores API to store the timestamp queue corresponding to each securities code. The Key of the Key Value Store is the securities code, and the Value is a double-ended queue of timestamps. In the setup method of the RabbitMQ Streams application, the Key Value Store is initialized and an empty double-ended queue is created for each securities code. In the transform method of the RabbitMQ Streams application, each piece of market data is processed, its timestamp is extracted, and the double-ended queue of the corresponding securities code is updated. In the transform method of the RabbitMQ Streams application, based on the timestamp of the current market data, the minimum timestamp in the double-ended queue is queried as the basis for selecting the optimal market data. The RabbitMQ Streams application can maintain a fixed-size sliding window for each securities code, and the timestamps of the most recent N pieces of market data are stored in the window. The size of the sliding window can be adjusted according to the update frequency and real-time requirements of the market data. The larger the window, the longer the time span, and the higher the delay of the selected optimal market data; the smaller the window, the shorter the time span, and the lower the delay of the selected optimal market data, but the jitter and error of the data may also be greater. In this application, the double-ended queue supports insertion and deletion operations at the head and tail of the queue, with a time complexity of O(1), without the need to move other elements, and is very efficient. The double-ended queue only stores timestamps, not complete market data, and has a small memory footprint, which can support the concurrent processing of a large number of securities codes. Using a double-ended queue to implement a sliding window can significantly reduce the delay in selecting market data, improve the timeliness of optimal data, and reduce memory and CPU overhead.

[0041] S24: When new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window. If so, add the timestamp of the new market data to the end of the sliding window and delete the timestamp at the head of the sliding window. In specific implementation, you can process the market data stream for each securities code in a RabbitMQ Streams application, using the transform or process method to perform custom processing on each piece of data. In the processing function, first obtain the timestamp of the data and compare it with the maximum timestamp in the sliding window. If the timestamp of the new data is greater than the maximum timestamp, add the new timestamp to the end of the sliding window and delete the timestamp at the head of the sliding window.

[0042] S25: Obtain the market data corresponding to the minimum timestamp in the sliding window. This data is used as the market data with the minimum latency for the corresponding security code within the current sliding window, and is then used as the optimal market data. In a specific implementation, the market data stream for each security code can be processed in a RabbitMQ Streams application, using the transform or process method to perform custom processing on each data item. In the processing function, the minimum timestamp in the sliding window is obtained. The corresponding market data is then queried from a state store (such as a Key Value Store) based on the minimum timestamp, and the query result is output as the optimal market data.

[0043] S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the specified topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the securities code, and the value is the optimal market data. In specific implementation, the optimal market data of each securities code can be output to the specified RabbitMQ topic using the to method in the RabbitMQ Streams application. The output data format can adopt a structured format such as JSON, protobuf, etc., with the securities code as the key of the message and the optimal market data as the value of the message. Subsequent consumer applications can partition and consume and process the optimal market data according to the securities code.

[0044] In this application, RabbitMQ Connect is responsible for real-time data collection and publishing, RabbitMQ Streams is responsible for real-time data calculation and aggregation, and double-ended queues and sliding windows are responsible for data caching and latency control. Compared with the original market data, optimal market data can significantly reduce data transmission and storage overhead, reducing network bandwidth and disk space usage. At the same time, through sliding windows and timestamp comparison, the data with the lowest latency can be dynamically selected from multiple market data, improving the real-time and accuracy of the data. In addition, RabbitMQ Streams provides highly scalable and fault-tolerant stream processing capabilities, which can easily handle the concurrent processing of massive market data and support fast state recovery and replay.

[0045] The optimal market data contains the following basic information: Securities code: A unique identifier for a stock or fund, in the form of "600000.SH." Securities name: The abbreviation for the stock or fund, such as "Shanghai Pudong Development Bank." Exchange code: Indicates the exchange to which the security belongs, in the form of "SH," "SZ," etc. By querying the securities database, you can obtain supplementary information about the security corresponding to the optimal market data, such as: Security type: Stock, fund, bond, etc. Trading status: Normal trading, suspended, delisted, etc. Based on the optimal market data, you can further calculate derivative data that reflects the security's price trend and trading activity, such as: Price change: The percentage change of the current price relative to the previous closing price. Amplitude: The ratio of the difference between the highest and lowest prices of the day to the previous closing price. Turnover rate: The ratio of trading volume to outstanding shares. Multi-point access is used to obtain redundant, multi-source market data. The stream processing engine's time window mechanism and sliding window algorithm are used to dynamically select the market data with the lowest latency as the optimal data. Combined with message queues, low-latency and highly reliable market data distribution is achieved. At the same time, the content of market data is enriched through data enhancement and derivative indicator calculation.

[0046] S3, based on the supplementary information and the optimal market data containing basic information, generates securities static data, including: S31, extracts the securities code from the optimal market data to query the supplementary information: The optimal market data usually contains basic information such as the securities code, price, and trading volume, and can be parsed using formats such as JSON and Protobuf. After parsing, the securities code field can be extracted using regular expressions, string matching, and other methods. After extracting the securities code, you can construct an SQL statement or call the ORM framework to query the securities database through a predefined database query interface (such as a DB connection pool). The query interface generally returns supplementary information data corresponding to the securities code, such as the securities name, exchange, and securities type.

[0047] S32, static data generation by associating supplemental information and basic information: the queried supplemental information and the original basic information are field-spliced and format-converted to generate static data of securities. The association can be performed in memory, such as using a HashMap with the security code as the Key and the static data as the Value. For the convenience of subsequent processing, the static data is generally parsed into a structure / class of a specific programming language, and Go language is used here.

[0048] S33, Protobuf serialization of static data and publishing: first, define an equivalent Protobuf message according to the Go structure of the static data to generate a.proto file. Then, use the protoc tool to compile the.proto file into a corresponding Go source code file, mainly including message structure definition and serialization method. In the code, create a Protobuf message object and assign the field values of the static data to the message object. Call the serialization method of the Protobuf message to serialize the object into binary data. Publish the security code as the routing key and the serialized binary data as the message body to the specified topic of RabbitMQ. RabbitMQ Go client library such as amqp can be used for publishing.

[0049] S34, static data storage: subscribe to the static data topic in RabbitMQ to obtain the newly published message. Call the Protobuf deserialization method to parse the binary message body into a static data structure. Then write the static data into a columnar database such as HBase, Cassandra, etc. Write the security code as the primary key and each field as a separate column, which can be assisted by an ORM framework.

[0050] S35, incremental data synchronization: configure the Binlog of the securities database to write the changed data to the Binlog file in real time. Develop a Binlog parsing program to read the changed content of the Binlog file in real time. Parse the changed data into a static data structure and publish it to a specific topic of RabbitMQ. The downstream subscribes to the changed data from the topic and updates the static information in the database.

[0051] S4, according to the optimal market data containing basic information, generate corresponding K-line data and derivative data through data fitting, including: S41, obtain the optimal market data containing basic information from the specified topic of RabbitMQ stream processing engine, and group and sort the optimal market data according to the security code and timestamp to obtain the time series market data corresponding to each security code. For example, using KStream and KGrouped Stream API of RabbitMQ Streams, read the optimal market data from the topic named "optimal", group according to the security code, and sort according to the timestamp to obtain the time series market data corresponding to each security code. By grouping and sorting by security code, the scattered market data can be converted into continuous time series, facilitating subsequent K-line synthesis and index calculation.

[0052] S42, K-line synthesis is performed on the time series market data corresponding to each security code to generate corresponding K-line data; according to the preset K-line period, set the time window size of K-line synthesis, and divide the time series market data according to the time window size to obtain continuous time window sequence; calculate the index of each time window sequence to generate K-line data of the corresponding time window; the index includes opening price, closing price, highest price, lowest price and trading volume; the opening price is the price of the first market data in the time window; the closing price is the price of the last market data in the time window; the highest price is the maximum value of all market data prices in the time window; the lowest price is the minimum value of all market data prices in the time window; the trading volume is the sum of all market data trading volumes in the time window. For example, assuming that the preset K-line period is 1 minute, the time series market data is divided according to 1 minute time window, and each time window contains multiple market data. Then, the market data in each time window is calculated to generate corresponding K-line data. By setting a fixed size time window, market data of different frequencies can be converted into unified K-line data, facilitating subsequent data analysis and visual display.

[0053] The K-line data is denoised by adopting 3σ principle to obtain denoised K-line data. For example, for the market data in each time window, calculate the mean μ and standard deviation σ of its price, and eliminate the outliers beyond the range of [μ-3σ, μ+3σ], recalculate the K-line index, and obtain the denoised K-line data. Denoising can eliminate the violent fluctuations and error data in the market data, improve the quality and reliability of the K-line data.

[0054] According to the de-noised K-line data, the trend feature sequence of the K-line data is obtained by calculating the price change rate and the trading volume change rate between adjacent K-line data points. For example, for two adjacent K-line data points k1 and k2, their price change rate and trading volume change rate are respectively:

[0055] , wherein, represents the closing price of the K-line data point , represents the closing price of the K-line data point . represents the trading volume of the K-line data point , represents the trading volume of the K-line data point .

[0056] The price change rate and the trading volume change rate of each K-line data point are arranged in time sequence to obtain the trend feature sequence of the K-line data, which reflects the change trend of the security price and the transaction activity. According to the trend feature sequence, data compression is performed by using run-length encoding to obtain the final K-line data. Run-length encoding is a run-length encoding algorithm, which can compress continuous same value data points into the form of (value, length), greatly reducing the storage space of data. For example, for the trend feature sequence [0.01, 0.01, 0.02, -0.01, -0.01], it can be compressed into the form of [ (0.01, 2), (0.02, 1), (-0.01, 2) ]. Using run-length encoding can reduce the storage and transmission overhead of data while preserving the characteristics of data, improving the scalability of the system.

[0057] The present application can generate K-line data in real time from a large amount of market data, and improve the efficiency and quality of data processing through data de-noising, feature extraction and compression encoding and other optimization means. This streaming computing architecture can fully utilize the message caching and horizontal expansion capability of RabbitMQ to realize low-latency generation and real-time pushing of K-line data. At the same time, through multi-level caching and indexing of K-line data, high-concurrency query of the client can be supported, avoiding direct access to the underlying time series database, and significantly reducing the query delay.

[0058] S43, calculating the time series market data technical indicators corresponding to each security code, generating the corresponding derivative data, calculating the moving average line indicators according to the K-line data, including , , , and . In For example, the calculation of the last n trading days' closing price of each security code is read from the RabbitMQ K-line data topic. According to the formula Calculate The value of Represents the closing price on day i. The calculation result is output to the RabbitMQ derived data topic in the form of (symbol, timestamp, value), such as ("600000.SH", "2022-03-01 09:31:00", 10.25).

[0059] Calculate the exponential moving average indicator based on time series market data and K-line data, including and .by For example, the closing price of the most recent trading day for each stock code is read from the RabbitMQ market data topic. , read the previous trading day's data from the derived data topic value , according to the recursive formula Calculate the current trading day value, where , n takes values ​​of 12 and 26. The calculation results are output to the Derivative Data topic in the same format as MA(n).

[0060] according to and Calculates MACD indicators, including DIF, DEA, and MACD. Reads the latest data corresponding to each security code from the derivative data topic. and The values ​​of DIF, DEA and MACD are calculated according to the following formula: , , ,in, Represents the 9-day exponential moving average of DIF. The calculation result is output to the derived data topic in the form of (symbol, timestamp, dif, dea, macd).

[0061] Calculate volume metrics, including VOL and AMT, based on time series market data. Read all market data for the most recent trading day for each security symbol from the market data topic and calculate VOL and AMT using the following formula: , ,in, Indicates the transaction volume of the i-th market data, Represents the price of the i-th market data. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vol, amt).

[0062] Calculate the volume ratio indicator VR based on VOL. Read the VOL values ​​of the last five trading days corresponding to each security code from the derivative data topic, calculate the simple moving average of the trading volume of these five days MA (VOL, 5), and then use the formula Calculate the VR value of the current trading day, where Indicates the trading volume of the current trading day. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vr).

[0063] Calculate the price change rate indicator based on time series market data, including and Read the closing price of the last six trading days for each security code from the market data topic and calculate it according to the following formula and Value: , ,in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Represents the closing price five trading days ago. The calculation result is output to the derived data topic in the form of (symbol, timestamp, change1, change5).

[0064] According to the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator .by For example, the VOL value of the last five trading days and the closing price of the last six trading days corresponding to each security code are read from the derivative data topic, and the following formula is used to calculate and Value: ; ;in, Indicates the trading volume of the current trading day. Indicates the trading volume 5 trading days ago. The calculation of is detailed in the above formula. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vppi5).

[0065] The calculated 、 , MACD, VOL, AMT, VR, and Derivative data such as securities codes and timestamps are associated with the corresponding securities codes and timestamps, and arranged in the order of securities codes and timestamps to generate the final derivative data results. This application uses the state storage and sliding window functions of RabbitMQ Streams to achieve incremental calculation and real-time update of derivative data. Compared with traditional batch processing methods, this solution can significantly reduce computing latency and improve the real-time nature of data. At the same time, by dividing different derivative indicators into different RabbitMQ partitions, parallel computing can be achieved, further improving processing performance.

[0066] S44: Associate the generated K-line data and derived data based on the security code and timestamp, and publish the associated data to the designated topic of the RabbitMQ stream processing engine. Specifically, the latest data corresponding to each security code is read from the RabbitMQ K-line data topic and derived data topic, respectively, and concatenated based on the timestamp to obtain result data containing the complete K-line and derived data fields. The result data is then published to the RabbitMQ result data topic in protobuf format, using the security code as the message key to ensure orderly distribution and consumption of data for the same security code.

[0067] By associating K-line data with derived data in RabbitMQ, we can reduce the complexity of data acquisition and computation for downstream data consumers, improving data availability. The use of the protobuf format enables efficient data serialization and deserialization, reducing data transmission and storage overhead. The orderly distribution of result data helps ensure data timeliness and consistency, supporting real-time computing and display.

[0068] S45 synchronizes the candlestick data and derived data from the RabbitMQ stream processing engine to a distributed time series database, using the stock code as the partition key, the timestamp as the sort key, and the various fields of the candlestick data and derived data as metrics. Specifically, using the Sink Connector provided by a time series database (such as InfluxDB or Prometheus), data from the RabbitMQ result data topic is written to the time series database in real time. In the database, using the stock code as the tag, the timestamp as the time, and various fields of the candlestick data and derived data (such as opening price, closing price, trading volume, MA, MACD, etc.) as fields, a suitable data model and index structure are constructed to support efficient storage and query of massive amounts of time series data.

[0069] By storing data into a time series database, the high-throughput write, aggregation calculation, multi-dimensional query, and other characteristics of the time series database can be fully utilized to achieve persistent storage and flexible analysis of data. The time series database usually adopts columnar storage and compression technology, which can significantly reduce the storage cost and query delay of data. At the same time, the time series database provides rich query languages and visualization tools, which can support complex data analysis and monitoring alarm scenarios, and provide comprehensive data support for quantitative trading.

[0070] S5, receiving a client request through an interface gateway service and forwarding the client request to a query service; the client sends a RESTful style data query request to the interface gateway service through HTTP / HTTPS protocol, and the request parameters include security code, time range, data type (such as K-line data, derivative data) and the like. The interface gateway service uses a high-performance web server (such as Nginx, Envoy) to process client requests, supports concurrent processing and load balancing of requests. Through preset routing rules and service discovery mechanisms (such as Consul, Eureka), the interface gateway service dynamically routes the request to the query service cluster in the back end.

[0071] The query service adopts a micro-service architecture and is composed of multiple independent service instances, each instance being responsible for processing data queries for a part of security codes. The query service consumes static data of securities in real time from a result data topic of RabbitMQ, and caches the data in local memory or a distributed cache (such as Redis) to improve the query speed of data. For K-line data and derivative data, the query service communicates with the market data push service through gRPC protocol to obtain data for specified security codes and time ranges in real time. The market data push service is built based on RabbitMQ Streams and can support real-time data push and query.

[0072] The query service combines and encapsulates the static data of securities, K-line data and derivative data according to a predefined data format to generate response data in JSON or Protocol Buffers format that meets the client request. The query service returns the encapsulated response data to the interface gateway service, which compresses and encrypts (such as gzip, SSL) the response data and transmits it to the client.

[0073] The interface gateway service sets a response time threshold (such as 500 milliseconds) for client requests, and for requests exceeding the threshold, the interface gateway service actively terminates the processing of the request and returns a request timeout prompt to the client, avoiding long-term occupation of server resources and long-term waiting of the client.

[0074] The interface gateway service of the present application is the traffic entrance of the system, which can perform unified authentication, authorization, flow limiting, and fuse processing on the request, thereby improving the security and stability of the system. The query service adopts a micro-service architecture and a cache mechanism, which can support large-scale data query and real-time response. The market push service utilizes the message cache and stream processing capability of RabbitMQ to realize real-time push and on-demand acquisition of data. Efficient binary protocol is used for data transmission between services, reducing the serialization and deserialization overhead of data. At the same time, by setting a request timeout mechanism, the system can prevent infinite waiting of requests and resource exhaustion, thereby improving the availability of the system.

[0075] S6, the embodiment also includes distributed service fault handling based on interface monitoring and dynamic expansion, S61, the interface monitoring service uses a distributed tracking system to track and monitor the request call link of each service, collects the request volume, response time, error rate and other index data of the service, and uses a log aggregation platform to collect the running log of the service, and extracts the error information and abnormal stack in the log. For example, Zipkin is used as a distributed tracking system to sample the request of each service, record the start time, end time, service name, Span ID, Trace ID and other information of the request, and send the tracking data to the Zipkin server for storage and analysis. ELK (Elasticsearch, Logstash, Kibana) is used as a log aggregation platform, the running log of each service is collected through Logstash, and the log data is transmitted to Elasticsearch for indexing and storage, and the visualization query and analysis of log data is realized through Kibana.

[0076] S62, the interface monitoring service uses a stream processing engine to perform real-time calculation and rule matching on the index data collected in step S61, the stream processing engine is based on in-memory computing technology, and the index data is processed in the form of event stream in memory, through predefined aggregation functions and anomaly detection algorithms, the running state of the service is determined whether it meets the preset normal state threshold, and the calculation result is output to the downstream system in the form of stream. For example, Flink is used as a stream processing engine, the index data collected by Zipkin and ELK is sent to Flink in the form of RabbitMQ message, the CEP (Complex Event Processing) library of Flink is used to define anomaly detection rules, such as request response time exceeding 500ms, request error rate exceeding 1%, etc., the events matched to the rules are output to the alarm system or storage system.

[0077] In step S63, if the service's operating status does not meet the normal status threshold, the interface monitoring service automatically generates a fault handling workflow using the service orchestration engine based on a preset fault handling strategy. This workflow includes a series of fault diagnosis, recovery, and compensation operations. The workflow engine coordinates the invocation and execution of different services to achieve self-healing and recovery of the service. Simultaneously, the interface monitoring service uses an intelligent alerting system to perform semantic and correlation analysis on fault handling information, automatically generating a fault report and handling suggestions, which are then sent to operations and maintenance personnel via a multi-channel alerting platform. For example, using Airflow as the service orchestration engine, a fault handling workflow DAG (Directed Acyclic Graph) is defined, including fault diagnosis nodes, service restart nodes, version rollback nodes, and data recovery nodes. Upon receiving an exception event output by Flink, the workflow is automatically triggered. Using natural language processing and machine learning algorithms, semantic and correlation analysis is performed on logs, metrics, and events during the fault handling process, automatically generating information such as the fault cause, impact scope, and recovery plan. This information is then notified to operations and maintenance personnel via multiple channels, such as email, SMS, WeChat, and phone calls.

[0078] S64, the interface monitoring service uses a time series database to perform persistent storage and aggregate calculations on the indicator data collected in step S61. The time series database uses column storage and compression technology to support efficient data writing and querying. Through predefined aggregation functions and window functions, it performs sliding window calculations and multi-dimensional analysis on the indicator data to determine whether the load status of the service exceeds the preset load threshold, and displays the calculation results in the form of a time series on the monitoring screen. For example, using InfluxDB as a time series database, the indicator data collected by Zipkin and ELK is written to InfluxDB, and aggregate calculations are performed through InfluxQL query statements, such as calculating the average response time per minute, the number of requests per hour, etc. The query results of InfluxDB are displayed through Grafana, and the load status and change trends of the service are presented in the form of intuitive charts.

[0079] S65, if the load state of the service exceeds the load threshold, the interface monitoring service adopts an adaptive expansion algorithm to dynamically calculate the optimal number of service instances according to the preset expansion strategy. The adaptive expansion algorithm is based on reinforcement learning technology, and through the feedback of historical load data and expansion effect, the expansion strategy and parameters are continuously optimized to realize intelligent elastic scaling of the service. At the same time, the interface monitoring service adopts shadow deployment technology to verify and warm up the expanded service instances without affecting online business, ensuring smooth expansion and stable operation of the service. For example, using Deep Q-learning algorithm as the adaptive expansion algorithm, using CPU utilization, memory utilization, request volume and other indicators as the state, and using the number of expanded or scaled instances as the action, the optimal expansion strategy is learned through interaction with the environment to obtain the reward value. Before expanding new service instances, a small amount of real traffic is introduced into the new instances using shadow deployment technology to verify the performance, functionality, compatibility, etc. of the new instances, and if the verification is passed, the new instances are added to the production environment to replace or supplement the original service instances.

[0080] S66, the dynamic expansion mechanism creates distributed copy instances of the service through the container orchestration platform, and registers the newly created service instances in the service registry center. At the same time, the routing rules of the interface gateway service are updated. For example, using Kubernetes as the container orchestration platform, the replicas attribute of the service Deployment is modified, or the number of service Pods is automatically increased or decreased through HPA (Horizontal Pod Autoscaler). Using Eureka or Consul as the service registry center, the information of the newly created service instances is registered in the registry center for other service discovery and invocation. Using Zuul or Kong as the interface gateway service, the routing rules are modified to distribute request traffic to new service instances according to certain strategies.

[0081] S67, the interface gateway service dynamically distributes client requests to different service instances according to the updated routing rules combined with the service instance information provided by the service registry center, using load balancing algorithms including round robin, random, minimum connection number, weighted, etc. For example, using Ribbon as the client load balancer, the service instance list in the service registry center is obtained through integration with Eureka, and an instance is selected for request forwarding according to different load balancing strategies. Common strategies include simple round robin, weighted round robin, random, minimum connection number, etc. The appropriate strategy can be selected according to the characteristics and needs of the service, such as using the minimum connection number strategy for CPU-sensitive services and using the weighted round robin strategy for IO-sensitive services.

[0082] S68, the service instance after scaling handles the request, returns the processing result to the interface gateway service, the interface gateway service integrates and returns the processing result to the client, and at the same time, reports the performance index of request processing to the interface monitoring service; the interface monitoring service dynamically adjusts the number of service instances according to the performance index and the load state of the service, and realizes the elastic scaling of the service. For example, the scaled service instance returns the request processing result to the Zuul gateway through the HTTP or RPC protocol, the Zuul gateway aggregates, filters, converts and other operations on the processing results of multiple instances, and returns the HTTP response to the client. At the same time, the Zuul gateway asynchronously sends the processing time, response status code, exception information and other performance indexes of each request to Zipkin or ELK, which are aggregated and calculated and analyzed by the interface monitoring service. When the interface monitoring service finds that the load index of the service returns to the normal level and is stable for a period of time, the adaptive expansion algorithm is triggered, the optimal number of service instances is calculated, and the Kubernetes orchestration platform is notified to scale down the number of service pods, release the excess computing resources, and realize the elastic contraction of the service.

Claims

1. A method for processing distributed multi-channel market information, characterized in that: include: S1, receiving multiple market data through multiple access points; S2: Select the market data with the shortest latency as the optimal market data and add the selected optimal market data to the message queue; By adding the basic information of the optimal market data, the optimal market data containing the basic information is obtained; S3, based on the basic information, obtains supplementary information corresponding to the security code by querying the security database; Generate static securities data based on supplementary information and optimal market data containing basic information; S4, based on the optimal market data containing basic information, generates corresponding K-line data and derivative data through data fitting; S5, using streaming subscription through the market push service, subscribes to the best market data including basic information, K-line data, and derivative data from the message queue service; S6, receives the client request through the interface gateway service and forwards the client request to the query service, including: S61: The interface monitoring service uses a distributed tracking system to track and monitor the request call links of each service. S62: The interface monitoring service uses a stream processing engine to perform real-time calculations and rule matching on the indicator data collected in step S61. The stream processing engine, based on in-memory computing technology, processes the indicator data in the form of an event stream in memory, and uses predefined aggregation functions and anomaly detection algorithms to determine whether the operating status of the service meets the preset normal status threshold, and outputs the calculation results to the downstream system in the form of a stream. S63: If the operating status of the service does not meet the normal status threshold, the interface monitoring service uses the service orchestration engine to automatically generate a fault handling workflow based on the preset fault handling strategy. S64: The interface monitoring service uses a time series database to persistently store and aggregate the indicator data collected in step S61. The time series database uses columnar storage and compression technology. It uses predefined aggregation functions and window functions to perform sliding window calculations and multi-dimensional analysis on the indicator data to determine whether the service load status exceeds a preset load threshold. S65: If the load status of the service exceeds the load threshold, the interface monitoring service uses an adaptive expansion algorithm to dynamically calculate the optimal number of service instances based on the preset expansion strategy. The adaptive expansion algorithm is based on the DeepQ-learning algorithm and realizes intelligent elastic scaling of the service through feedback from historical load data and expansion effects. At the same time, the interface monitoring service uses shadow deployment technology to verify and preheat the expanded service instances.

2. The method for processing distributed multi-channel market information according to claim 1, characterized in that: Basic information includes security code, security name and exchange code; Supplementary information includes security type and trading status; Derivative data reflects securities price trends and trading activity.

3. The method for processing distributed multi-channel market information according to claim 2, characterized in that: S2: Select the market data with the shortest latency as the optimal market data, including: S21, storing the received multiple market data in the designated topic of the RabbitMQ stream processing engine in the order of arrival time, and associating a timestamp with each market data, where the timestamp indicates the arrival time of the market data; S22, using the time window mechanism of the RabbitMQ stream processing engine, grouping the market data within the specified time window by security code to obtain the market data stream corresponding to each security code; S23, using a double-ended queue data structure to set a fixed-size sliding window based on the market data stream corresponding to each securities code, wherein the sliding window stores the timestamp of the market data; S24, when new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window. If so, add the timestamp of the new market data to the end of the sliding window and delete the timestamp at the head of the sliding window. S25, obtaining the market data corresponding to the minimum timestamp in the sliding window as the market data with the minimum delay for the corresponding security code in the current sliding window, and as the optimal market data; S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the designated topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the stock code, and the value is the optimal market data.

4. The method for processing distributed multi-channel market information according to claim 2, characterized in that: S3 generates static securities data based on supplementary information and optimal market data containing basic information, including: S31, extracting a security code from the optimal market data containing basic information as a query condition, and querying the securities database for supplementary information corresponding to the security code through a predefined database query interface; S32, according to predefined data encapsulation rules, associating the supplementary information with the optimal market data containing the basic information to generate securities static data; parsing the securities static data into a GO language data structure, wherein the GO language data structure includes the securities code, securities name, exchange code, securities type, and transaction status; S33, serializes securities static data using protobuf data serialization format; S34, publish the serialized securities static data to the designated topic of the RabbitMQ stream processing engine, using the securities code as the routing key and the serialized binary data as the message body; S35: Synchronize the static securities data in the RabbitMQ stream processing engine to the distributed columnar storage data, using the securities code as the distinguishing key and the various fields of the securities static data as columns; When the supplementary information in the securities database changes, the change information is synchronized to the RabbitMQ stream processing engine through the Binlog-based database change mechanism.

5. The distributed multi-channel market information processing method according to claim 4, characterized in that: S4, generates corresponding K-line data and derivative data, including: S41, obtaining optimal market data containing basic information from a designated topic of the RabbitMQ stream processing engine, and grouping and sorting the optimal market data according to security code and timestamp to obtain time series market data corresponding to each security code; S42, performing K-line synthesis on the time series market data corresponding to each security code to generate corresponding K-line data; S43, calculating the technical indicators of the time series market data corresponding to each security code and generating corresponding derivative data; S44, associating the generated K-line data and derivative data according to the securities code and timestamp, and publishing the associated data to a designated topic of the RabbitMQ stream processing engine; S45 synchronizes the K-line data and derived data in the RabbitMQ stream processing engine to the distributed time series database, using the security code as the partition key, the timestamp as the sort key, and the various fields of the K-line data and derived data as measurement indicators.

6. The method for processing distributed multi-channel market information according to claim 5, characterized in that: S42, performing K-line synthesis on the time series market data corresponding to each security code to generate corresponding K-line data, including: According to the preset K-line cycle, set the time window size of the K-line synthesis, and divide the time series market data according to the time window size to obtain a continuous time window sequence; Calculate the indicators of each time window sequence and generate K-line data for the corresponding time window; the indicators include opening price, closing price, highest price, lowest price and trading volume; The 3σ principle is used to denoise the K-line data to obtain the denoised K-line data; Based on the denoised K-line data, the trend characteristic sequence of the K-line data is obtained by calculating the price change rate and trading volume change rate between adjacent K-line data points; According to the trend feature sequence, run-length encoding is used to compress the data to obtain the final K-line data.

7. The distributed multi-channel market information processing method according to claim 5, characterized in that: The opening price is the price of the first market data in the time window; The closing price is the price of the last market data in the time window; The highest price is the maximum value of all market data prices within the time window; The lowest price is the minimum value of all market data prices within the time window; Trading volume is the sum of all market data trading volumes within the time window.

8. The method for processing distributed multi-channel market information according to claim 5, characterized in that: S43, calculate the technical indicators of the time series market data corresponding to each securities code and generate corresponding derivative data, including: The moving average indicator is calculated based on the K-line data. The moving average indicator includes 、 、 、 and ;in, The calculation formula is: in, represents the closing price of the i-th day, and n represents the number of periods of the moving average; The exponential moving average indicator is calculated based on time series market data and K-line data. The exponential moving average indicator includes and ;in, Calculated by the following recursive formula: in, , n represents the period of the exponential moving average, Indicates the closing price of the current trading day. Indicates the value of the n-day exponential moving average of the previous trading day; according to and Calculate the MACD indicator, which includes DIF, DEA, and MACD; wherein DIF, DEA, and MACD are calculated using the following formula: in, Represents the 9-day exponential moving average of DIF; Calculate the trading volume indicators based on time series market data. The trading volume indicators include VOL and AMT. VOL and AMT are calculated using the following formulas: in, Indicates the transaction volume of the i-th market data, Indicates the price of the i-th market data; The volume ratio indicator VR is calculated based on VOL; the VR is calculated using the following formula: in, Indicates the trading volume of the current trading day. The 5-day simple moving average of volume; The price change rate indicator is calculated based on the time series market data. The price change rate indicator includes and ;in, and Calculated by the following formula: in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Indicates the closing price 5 trading days ago; According to the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator ; Calculated by the following formula: Where n represents the number of calculation cycles, Indicates the trading volume of the current trading day. Indicates the trading volume n trading days ago; The calculated 、 , MACD, VOL, AMT, VR, and As derivative data, it is associated with the corresponding securities code and timestamp, and the associated derivative data is arranged in the order of securities code and timestamp to generate the final derivative data result.

9. A distributed multi-channel market information processing system, characterized in that: include: At least one processing unit; used to execute instructions to implement the distributed multi-channel market information processing method according to any one of claims 1 to 8.

Images